On Sat, Aug 30, 2014 at 1:18 PM, Jon Maron <[email protected]> wrote:
> > On Aug 30, 2014, at 2:55 PM, Sumit Mohanty <[email protected]> > wrote: > > > Should we also move slider-client.tar.gz and application package into the > > application instance specific folder in HDFS? This along with the changes > > due to SLIDER-330 will likely make the application bits completely self > > contained. > > Maybe I’ve been buried in security related work for too long so… > > 1) slider-client.tar.gz? Do you mean slider-agent.tar.gz or is there a > client installer? > Yes, slider-agent.tar.gz. And no, we haven't secretly added another tarball :-) > 2) what is the “application instance specific folder”? > hdfs://user/<username>/slider or the like? > Subfolders within hdfs:///user/<username>/.slider/cluster/<cluster name>. Some AM artifacts are already there under tmp. 3) Do we really want to group Slider/AM artifacts and various application > artifacts together? Perhaps under a shared parent directory rather than > the same directory? > > We can define a folder structure that keeps various artifacts separate. There are two classes of artifacts, "application definition", and "Slider infrastructure". The later includes slider-agent, yarn libraries, AppMaster libraries, etc. * At least the slider-agent.tar.gz could move in as part of the Slider infra. There are two choices here and second one is just an optimization ** Store all Slider infra in a sub-folder within hdfs:///user/<username>/.slider/cluster/<cluster name> ** Create a common area in HDFS and refer to the infrastructure bits from there (a common world readable location) * The application definition zip file can also be moved in to say hdfs:///user/<username>/.slider/cluster/<cluster name>/definition The advantages will be that the application will be self contained and upgrades, to package or slider, can happen independently for each application instance. Again it may be a disadvantage if one wants to upgrade all instances together but that is really a tooling issues. Are there any TEZ/MR best practices that we can refer to? > > > -Sumit > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
