Hello Darrin,
I am all for keeping configuration URL in place, I am just not sure that
getting configuration from RM is much better then getting it statically from
tar ball, unless you tell me that configuration for RM is also set dynamically.
My suggestion would be to keep config URL configurable - it could be config
from RM, some xml from hdfs/maprfs, other sources. This way we can truly make
it more dynamic.
Thanks,Yuliya
From: Darin Johnson <[email protected]>
To: yuliya Feldman <[email protected]>
Cc: Dev <[email protected]>
Sent: Tuesday, September 22, 2015 8:13 PM
Subject: Re: Myraid binary distro - couple of questions
Yuliya,
In regards to 1. I believe "chown ${frameworkUser} ." was needed when I set
this up because the capsule plugin expanded some jars into the ./capsule
directory. Now the we're using the nodemanager directly that may not be an
issue.
As for 2) it's common in frameworks for schedulers to pass configurations
to the executors and I consider it a feature (maybe one that should be
cleaned up). I also view config changes as something that happens more
often than binary changes, so I actually leave the configs outside the
tarball and specifty an additional uri for the configs so I'm not
constantly making new tarballs. I'd hate to see it go away, especially
since I'm cleaning up my script that wgets the hadoop.tgz and does all the
surgery for you to do a super quick deploy.
I'm also not sure what it gets you as the config would essentially be
static in the tarball.
Open to a compromise of a config boolean.
Thanks for looking at remote disto.
On Tue, Sep 22, 2015 at 9:49 PM, yuliya Feldman <[email protected]> wrote:
> Hello Darrin,
>
> I have couple of questions regarding binary distro:
>
> 1. Just wonder if we still need to do "chown" on distro directory after
> untarring? It is working fine for me w/o changing permissions.
> 2. configuration you are getting from RM (config URL) to put into
> yarn-site.xml for NMs - why is this important?
> I assume you would use the same tar.gz to deploy RM and NMs (unless
> not true), so configuration would be the same. Unless I am missing
> something.
> My problem is that in Mapr Hadoop distro we use property to identify
> current physical host for shuffle purposes (Mapr is using maprfs for
> shuffle and not local disk), so I can not blindly copy configuration from
> RM to NM.
>
> Thanks,
> Yuliya
>
