Hi John, * Is there an FAQ for non-MR YARN development?
Have you seen this documentation page yet? If not, I recommend reading it before you get started. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html * Is there an FAQ for configuring/building/running Hadoop from source, preferably in Eclipse? When you check out a copy of the Hadoop source, you'll see a file named BUILDING.txt in the root. This file contains the comprehensive build instructions, including generation of Eclipse project files. https://github.com/apache/hadoop-common/blob/trunk/BUILDING.txt * What is the recommended configuration/environment for development of a YARN app? I would like to use Eclipse under Windows if that even makes any sense. I don't think there are any particular recommendations. I expect Windows to work if you're using the current trunk codebase, given the recent work on trunk for Windows compatibility. * Is there an ApplicationMaster example that can be used as a starting point? DistributedShell is a simple YARN application that just executes an arbitrary shell command on multiple nodes and gathers the output. The code for its Application Master is here: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * Is it possible to install objects or services that run as peers of the NodeManager as opposed to tasks? Are there any recommended per-node patterns as opposed to per-task patterns? I'm not sure if this directly answers it, but there is a notion of "visibility" for localized resources (resources copied locally to each node). In the documentation link I gave, there is a code sample related to this near the bottom. You can choose to allow sharing of localized resources among only the containers for the same application, across all applications owned by the same user, or across all applications/users running on the node. For your remaining questions about which version to select and compatibility, I'll need to defer to others on the list. There has been a lot of work recently around freezing the interfaces. I haven't been deeply involved with that work, so I don't feel qualified to answer. Hope this helps, Chris Nauroth Hortonworks http://hortonworks.com/ On Thu, May 23, 2013 at 3:45 PM, John Lilley <[email protected]>wrote: > I am getting started with development of a custom ApplicationMaster and I > didn't think that the user@ list was quite the right place for it. > Apologies if this list isn't the right place either. Some of my questions > are really newbie, like: > > * Is there an FAQ for non-MR YARN development? > > * Is there an FAQ for configuring/building/running Hadoop from > source, preferably in Eclipse? > > * What is the recommended configuration/environment for > development of a YARN app? I would like to use Eclipse under Windows if > that even makes any sense. > > * Would you start with a Hadoop release or build from version > control? > > * Is it possible to code for compatibility between 2.0 and 0.23? > > * Is there an ApplicationMaster example that can be used as a > starting point? > I also have some more in-depth questions: > > * When a MapReduce task creates its output files and makes them > available over HTTP, is it the NodeManager that serves them up? If my YARN > task wants to do something similar, how does it tell the NodeManager? How > are the files removed later? > > * Is it possible to install objects or services that run as peers > of the NodeManager as opposed to tasks? Are there any recommended per-node > patterns as opposed to per-task patterns? > > Thanks > John > >
