Re: How to restrict ephemral ports used by Yarn App Master

Susheel Kumar Gadalay skgadalay@... writes: Hi, I have a question. How do I selectively open port range for Hadoop Yarn App Master on a cluster. I have seen the jira issue in

Re: In Yarn how to increase the number of concurrent applications for a queue

How many nodes do you have in your cluster? Also, could you share the CapacityScheduler initialization logs for each queue, such as: 2014-08-14 15:14:23,835 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: unfunded: capacity=0.5,

Re: conf.get(dfs.data.dir) return null when hdfs-site.xml doesn't set it explicitly

Hi Demai, When you read a property from the conf object, it will only have a value if the conf object contains that property. In your case, you created the conf object as new Configuration() -- adds core-default and core-site.xml. Then you added site.xmls (hdfs-site.xml and core-site.xml) from

Re: conf.get(dfs.data.dir) return null when hdfs-site.xml doesn't set it explicitly

hi, Bhooshan, thanks for your kind response. I run the code on one of the data node of my cluster, with only one hadoop daemon running. I believe my java client code connect to the cluster correctly as I am able to retrieve fileStatus, and list files under a particular hdfs path, and similar

Re: conf.get(dfs.data.dir) return null when hdfs-site.xml doesn't set it explicitly

Hi Demai, conf = new Configuration() will create a new Configuration object and only add the properties from core-default.xml and core-site.xml in the conf object. This is basically a new configuration object, not the same that the daemons in the hadoop cluster use. I think what you are

Re: conf.get(dfs.data.dir) return null when hdfs-site.xml doesn't set it explicitly

Bhooshan, Many thanks. I appreciate the help. I will also try out Cloudera mailing list/community Demai On Mon, Sep 8, 2014 at 4:58 PM, Bhooshan Mogal bhooshan.mo...@gmail.com wrote: Hi Demai, conf = new Configuration() will create a new Configuration object and only add the properties

Re: In Yarn how to increase the number of concurrent applications for a queue

It turned out that it's not a configuration issue, some worker thread which submits job to Yarn was blocked, see below thread dump pool-1-thread-160 id=194 idx=0x30c tid=886 prio=5 alive, blocked, native_blocked -- Blocked trying to get lock:

Re: conf.get(dfs.data.dir) return null when hdfs-site.xml doesn't set it explicitly

One doubt on building Configuration object. I have a Hadoop remote client and Hadoop cluster. When a client submitted a MR job, the Configuration object is built from Hadoop cluster node xml files, basically the resource manager node core-site.xml and mapred-site.xml and yarn-site.xml. Am I