hi John, I just got some content onto the wiki that wasn't there before. Right now, I'm working on organizing the content better... your comments will help a LOT.
Thanks, Ruth On Wed, Aug 19, 2015 at 11:56 AM, John Omernik <[email protected]> wrote: > Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing > skills and suggestions on how to help. Happy to help in the way that you'd > find easiest. > > > John > > On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris <[email protected]> wrote: > > > hi John, > > > > Thank you for your feedback. I'm the assigned technical writer for the > > Myriad project. I'll also be working on updating the information. > > > > I'm also ok with what Adam indicated: directly editing or adding a John's > > page. But please be aware that I'll also be working on the content. Last > > week I did some cleanup work on the original files in GitHub and then > > brought them into the Wiki, although, I still have some more work in > terms > > of organizing and identifying holes. > > > > If you like, I can create a "John's comments" page and then work on > > incorporating some of the obvious things that you mentioned. > > > > Thanks, Ruth > > > > Ruth Harris > > Sr. Tech. Writer > > [email protected] > > > > On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <[email protected]> wrote: > > > > > Thanks Adam, I signed up with "mandoskippy". > > > > > > I am honored to help in this capacity, for updating etc, do we go > through > > > some kind of review? Is it better to ask questions on the dev list then > > > update when consensus occurs? How about when I'd like to post a page > and > > > then have someone review the work? If I have a comment on the page, is > > that > > > public or can I just send to author? Just curious on any guidelines I > > > should be following in that regard. > > > > > > John > > > > > > > > > > > > On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <[email protected]> > > wrote: > > > > > > > John, thanks a ton for your valuable feedback! We're glad to have > your > > > > perspective as a user of the project, and I'm ready+willing to give > you > > > > edit access to the wiki if you want to update it with your learnings, > > > > elaborate anything that's unclear, or add a new "John's tips" page. > > Just > > > > sign up for a wiki account, send me your accountId, and I'll grant > you > > > edit > > > > access. > > > > (I'll let others answer your specific questions) > > > > > > > > On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <[email protected]> > > wrote: > > > > > > > > > Today, I will be playing the role of the fool/jester trying to get > > > Myriad > > > > > running. Basically, since getting Myriad running with Santosh > quite a > > > > while > > > > > ago, and now trying again with new versions of Hadoop, MapR, and > > > Myriad, > > > > I > > > > > wanted to hit up the wiki ( > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) > and > > > > > outline > > > > > points that as a non-dev living the code, are unclear to someone > > trying > > > > to > > > > > utilize myriad or understand it's operation. > > > > > > > > > > Obviously, some of my points can be answered with "look here in the > > > code" > > > > > or look at this page, but I will try to outline my thought > processes > > > as I > > > > > reviewed the current docs. Sometimes the way I approached the > > problem > > > > led > > > > > me down a path of to a certain page, missing the answer in a > > different > > > > > page, and thus some cross linking could be helpful. > > > > > > > > > > Please do not let my points be taken as anything other than a > desire > > to > > > > > improve how accessible Myriad is to the community, this is not a > > > critique > > > > > of the hard work everyone has done on the project. I also > understand > > > > that > > > > > given the work load and other issues, that fixing these issues in > > > > > documentation may not be a priority. I am listing them out here, > so > > > that > > > > > those folks who are SMEs on various points may be able to quickly > add > > > > stuff > > > > > and we'll organize it later. > > > > > > > > > > > > > > > *Remote Distribution: * > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution > > > > > > > > > > This whole section could use some work from a standpoint of what > runs > > > > where > > > > > and where that component gets its files. For example, I think it > > would > > > > > help people to understand that the whole tarball created in step 6 > > has > > > > all > > > > > the files for node managers and resource managers. Basically, > > > everything > > > > > runs from there. Here is a small example I am currently working > with: > > > > > > > > > > > > > > > Starting Myriad: > > > > > Option 1: Use Marathon (provide example json, here is mine) > > > > > { > > > > > "cmd": "env && export > > > > > > > > > > > > > > > > > > > > YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos > > > > > && hadoop-2.7.0/bin/yarn resourcemanager", > > > > > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"], > > > > > "cpus": 1.0, > > > > > "mem": 1024, > > > > > "id": "myriad", > > > > > "instances": 1, > > > > > "user": "mapr" > > > > > } > > > > > > > > > > In this case, Marathon grabs the hadoop tarball and pulls it down, > > this > > > > > tarball also has the Myriad yml file. When it executes the resource > > > > > manager, it is brought up in Myriad and ready to run node managers > by > > > > > pulling the tarball to the slave nodes and executing the > nodemanager. > > > (I > > > > > would imagine the work with history server etc would also use this > > > > > tarball?). > > > > > > > > > > From here it will us NMInstances to launch a node manager. (Note, > > this > > > > is > > > > > different from when I originally set things up... before, I could > run > > > the > > > > > resource manager/myriad without a nodemanager, now it seems it's > > > required > > > > > based on the config in the src... could we expound on this in the > > docs > > > > > somewhere?) > > > > > > > > > > > > > > > Option 2: ???? (Are there other ways to launch the resource > manager?) > > > > > > > > > > Step 6: So something that is unclear to me is the handling of the > > > > > hadoop/yarn config files. In Step 6 on this page, there is "sudo > rm > > > > > hadoop- > > > > > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually > > > > ignored > > > > > this step. For me, if I remove these xml files, then there is no > > place > > > > to > > > > > get my files... I think? Since I am running the RM and NM from the > > same > > > > > tarball, and Myriad config is here, and my goal is to not have > > anything > > > > > installed on a node, where would I set yarn settings? This could > be > > > much > > > > > clearer to me, and probably others. > > > > > > > > > > Step 2: Should we just be copying the Myriad files to > > > > > /share/hadoop/yarn/lib folder? Do we worry about potentially > > overwrites > > > > of > > > > > jars or version conflicts? > > > > > > > > > > *Configuring Cgroups* > > > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups > > > > > At some point a little bit more about why one would want CGroups > and > > > > issues > > > > > that could occur with them. While many folks using Mesos/Myriad may > > > > > understand this, others may not, and it's a good way to help people > > > think > > > > > positively about our project if we help educate them along the way. > > > > > > > > > > Minor point on enabling CGroups. This is confusing given my > questions > > > in > > > > > remote distribution. in this it says I need to edit my > yarn-site.xml, > > > but > > > > > in remote distribution it says delete my hadoop xml files. We need > to > > > > > address this conflict cause it can be confusing for a user coming > > > onboard > > > > > > > > > > Nitpick: Enabling cgroups for mess-slave - should be - Enabling > > cgroups > > > > for > > > > > mesos-slave > > > > > > > > > > *Myriad Configuration Properties*: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties > > > > > Based on the conversation on list with Yuliya, > > > > > "Currently, this file is built into Myriad Scheduler jar. So, if > you > > > need > > > > > to modify some of the properties in this file, modify them before > > > > building > > > > > Myriad Scheduler." > > > > > isn't accurate any more, and we should address that. > > > > > > > > > > The configuration file in the wiki is an old one, the nmInstances > > isn't > > > > in > > > > > it, (and see my question about that above). > > > > > > > > > > Frameworks and usernames. I think the users that the framework > runs > > > as, > > > > > the actual node and resource managers, etc is confusing to a user > (I > > am > > > > > very confused!) When I first got Myriad up I set my user under the > > > > > executor to be mapr, and then it appeared to work with > impersonation > > > from > > > > > queries etc. Now, I am trying the remote distribution and I have > > users > > > > set > > > > > in the config, potentially a user in my marathon json, and I am > > getting > > > > > errors on permissions of files when a node manager tries to start > (a > > > > > separate issue I will post later). Basically, this is complex, and > a > > > page > > > > > describing out what needs to run where with which permissions and > how > > > > that > > > > > interacts will be huge for people looking to put this into play. > > > > > > > > > > *Example Yarn Site:* > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml > > > > > > > > > > This is helpful, but where does it go? Remember, the remote > > > distribution > > > > > had us delete the yarn-site in the hadoop etc folder. > > > > > > > > > > *Myriad Webapp * > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp > > > > > > > > > > This should be fleshed out a bit more. Also, it's in the > > > > > /myriad-scheduler/src/main/resources/webapp based on my git clone, > > but > > > in > > > > > the wiki that's not listed. I had to dig for it. > > > > > > > > > > Some questions here: could the webapp be built during the myriad > > > > building > > > > > process? Could it be then be packaged as tarball for execution > either > > > > > manually via marathon or automatically in a container on mesos? I > > > > > understand this is a fresh piece of the puzzle, I am just thinking > > > about > > > > > and verbalizing the "where" on this for the future > > > > > > > > > > > > > > > > > > > > Those are the items that come to mind thus far. I hope the tone of > > my > > > > > email is correct, this is a great project, and I want others to try > > it > > > > as I > > > > > have. > > > > > > > > > > John Omernik > > > > > > > > > > > > > > > > > > > > -- > > Ruth Harris > > Sr. Technical Writer, MapR > > > -- Ruth Harris Sr. Technical Writer, MapR
