Re: java.io.IOException: All datanodes DatanodeInfoWithStorage

2016-10-17 Thread chiranjeevi vasupilli
Thank You Priyanka, we are opening more number files in our app. can u please let me know , how to find the current limit of no of open files in cluster. so that we will update. On Mon, Oct 17, 2016 at 11:13 AM, Priyanka Gugale wrote: > Looks like you are reaching the limit set for max open f

Re: java.io.IOException: All datanodes DatanodeInfoWithStorage

2016-10-17 Thread Amol Kekre
Chiranjeevi, You will need to set the limit on each server in the cluster -> http://man7.org/linux/man-pages/man3/ulimit.3.html Run ulimit on each server in the cluster to set the value. Once you decide what the number should be, do remember to (a) set it using ulimit, & (b) put this command in a s

Re: balanced of Stream Codec

2016-10-17 Thread Vlad Rozov
Using different hash function will help only in case data is equally distributed across categories. In many cases data is skewed and some categories occur more frequently than others. In such case generic hash function will not help. Can you try to sample data and see if the data is equally dis