Yuliya, define dynamic? No updating a tarball to update config yes. On the fly config changes no. I agree it could be better, my plan was to serialize the configuration and pass it to the NMs, similar to how the hadoop 1 and storm frameworks do. Pulling the config from the rm worked OK at the time though. I think it's worth revisiting my original plan now. Having a configuri option may be OK, but let's make it an optional that defaults to the current situation until we agree its a good idea and that we're doing it correctly. I'm curious what your trying to set and how you think having a static yarn-site in the tarball would help. Are you sure we can't pass it the way we pass the spindles option? On Sep 23, 2015 1:14 AM, "yuliya Feldman" <[email protected]> wrote:
> Hello Darrin, > I am all for keeping configuration URL in place, I am just not sure that > getting configuration from RM is much better then getting it statically > from tar ball, unless you tell me that configuration for RM is also set > dynamically. > My suggestion would be to keep config URL configurable - it could be > config from RM, some xml from hdfs/maprfs, other sources. This way we can > truly make it more dynamic. > Thanks,Yuliya > From: Darin Johnson <[email protected]> > To: yuliya Feldman <[email protected]> > Cc: Dev <[email protected]> > Sent: Tuesday, September 22, 2015 8:13 PM > Subject: Re: Myraid binary distro - couple of questions > > Yuliya, > > In regards to 1. I believe "chown ${frameworkUser} ." was needed when I set > this up because the capsule plugin expanded some jars into the ./capsule > directory. Now the we're using the nodemanager directly that may not be an > issue. > > As for 2) it's common in frameworks for schedulers to pass configurations > to the executors and I consider it a feature (maybe one that should be > cleaned up). I also view config changes as something that happens more > often than binary changes, so I actually leave the configs outside the > tarball and specifty an additional uri for the configs so I'm not > constantly making new tarballs. I'd hate to see it go away, especially > since I'm cleaning up my script that wgets the hadoop.tgz and does all the > surgery for you to do a super quick deploy. > > I'm also not sure what it gets you as the config would essentially be > static in the tarball. > > Open to a compromise of a config boolean. > > Thanks for looking at remote disto. > > > > > On Tue, Sep 22, 2015 at 9:49 PM, yuliya Feldman <[email protected]> > wrote: > > > Hello Darrin, > > > > I have couple of questions regarding binary distro: > > > > 1. Just wonder if we still need to do "chown" on distro directory after > > untarring? It is working fine for me w/o changing permissions. > > 2. configuration you are getting from RM (config URL) to put into > > yarn-site.xml for NMs - why is this important? > > I assume you would use the same tar.gz to deploy RM and NMs (unless > > not true), so configuration would be the same. Unless I am missing > > something. > > My problem is that in Mapr Hadoop distro we use property to identify > > current physical host for shuffle purposes (Mapr is using maprfs for > > shuffle and not local disk), so I can not blindly copy configuration from > > RM to NM. > > > > Thanks, > > Yuliya > > > > >
