I am using Hortonworks HDP 2.0. I know that this setting is in I guess in Yarn-site.xml. Just not sure how to pass this configuration to PigServer.
On 15 October 2014 16:58, Serega Sheypak <serega.shey...@gmail.com> wrote: > Hi, you need to pass several properties because of NN HA. > What distro do you use? > see here: > > http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html#Configuration_overview > Configuration details > > <property> > <name>dfs.ha.namenodes.mycluster</name> > <value>nn1,nn2</value> > </property> > > <property> > <name>dfs.namenode.rpc-address.mycluster.nn1</name> > <value>machine1.example.com:8020</value> > </property> > <property> > <name>dfs.namenode.rpc-address.mycluster.nn2</name> > <value>machine2.example.com:8020</value> > </property> > > > You sohuld pass these three properties to your "client" (pig). Then > you can refer your nameservice as hdfs://mycluster/path/to/input > > > 2014-10-15 18:16 GMT+04:00 Jakub Stransky <stransky...@gmail.com>: > > > Thanks! Just giving a try but I am not able to figure out the hostname of > > dhfs because we run cluster in HA mode. Is there a way how to pass hadoop > > config? Because othervise I don't see any other option how that would be > > resolved. > > > > On 15 October 2014 14:50, Serega Sheypak <serega.shey...@gmail.com> > wrote: > > > > > Have a look at > > > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html > > > We use it in our integration test utility since default pig-unit > doesn't > > > fit our needs. > > > You can register script and run it. > > > > > > 2014-10-15 16:45 GMT+04:00 Jakub Stransky <stransky...@gmail.com>: > > > > > > > That's probably not the option as I need to wrap it in oozie java > > action > > > > and not sure that pig will be there. May be I am completely mistaken > > > > > > > > On 15 October 2014 14:01, praveenesh kumar <praveen...@gmail.com> > > wrote: > > > > > > > > > You can write a wrapper (like a shell/ruby/python script) which can > > do > > > > the > > > > > parameter generation and then pass those parameters using -p > > parameter > > > in > > > > > your pig calling statement. Wouldn't that work ? > > > > > > > > > > > > > > > > > > > > On Wed, Oct 15, 2014 at 12:16 PM, Jakub Stransky < > > > stransky...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello experienced users, > > > > > > > > > > > > I have a pig script where I need dynamically change input > > parameters > > > > e.g. > > > > > > input paths and for every single combination I need to submit a > pig > > > > > > script. Esentially I am walking around missing features of > loops > > or > > > > for > > > > > > cycles. The only solution which came to my mind is put the whole > > > > dynamic > > > > > > logic in java activity just need to fine a way how to submit a > pig > > > > script > > > > > > from java. > > > > > > > > > > > > Could someone please point me to right direction? > > > > > > > > > > > > Many thanks > > > > > > Jakub > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Jakub Stransky > > > > cz.linkedin.com/in/jakubstransky > > > > > > > > > > > > > > > -- > > Jakub Stransky > > cz.linkedin.com/in/jakubstransky > > > -- Jakub Stransky cz.linkedin.com/in/jakubstransky
