john, the details of where to send it on in the pdf that must be filled out and signed. the fastest way is to send the ICLA to [email protected] <mailto:[email protected]>
ken > On Aug 19, 2015, at 1:56 PM, John Omernik <[email protected]> wrote: > > Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing > skills and suggestions on how to help. Happy to help in the way that you'd > find easiest. > > > John > > On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris <[email protected]> wrote: > >> hi John, >> >> Thank you for your feedback. I'm the assigned technical writer for the >> Myriad project. I'll also be working on updating the information. >> >> I'm also ok with what Adam indicated: directly editing or adding a John's >> page. But please be aware that I'll also be working on the content. Last >> week I did some cleanup work on the original files in GitHub and then >> brought them into the Wiki, although, I still have some more work in terms >> of organizing and identifying holes. >> >> If you like, I can create a "John's comments" page and then work on >> incorporating some of the obvious things that you mentioned. >> >> Thanks, Ruth >> >> Ruth Harris >> Sr. Tech. Writer >> [email protected] >> >> On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <[email protected]> wrote: >> >>> Thanks Adam, I signed up with "mandoskippy". >>> >>> I am honored to help in this capacity, for updating etc, do we go through >>> some kind of review? Is it better to ask questions on the dev list then >>> update when consensus occurs? How about when I'd like to post a page and >>> then have someone review the work? If I have a comment on the page, is >> that >>> public or can I just send to author? Just curious on any guidelines I >>> should be following in that regard. >>> >>> John >>> >>> >>> >>> On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <[email protected]> >> wrote: >>> >>>> John, thanks a ton for your valuable feedback! We're glad to have your >>>> perspective as a user of the project, and I'm ready+willing to give you >>>> edit access to the wiki if you want to update it with your learnings, >>>> elaborate anything that's unclear, or add a new "John's tips" page. >> Just >>>> sign up for a wiki account, send me your accountId, and I'll grant you >>> edit >>>> access. >>>> (I'll let others answer your specific questions) >>>> >>>> On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <[email protected]> >> wrote: >>>> >>>>> Today, I will be playing the role of the fool/jester trying to get >>> Myriad >>>>> running. Basically, since getting Myriad running with Santosh quite a >>>> while >>>>> ago, and now trying again with new versions of Hadoop, MapR, and >>> Myriad, >>>> I >>>>> wanted to hit up the wiki ( >>>>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and >>>>> outline >>>>> points that as a non-dev living the code, are unclear to someone >> trying >>>> to >>>>> utilize myriad or understand it's operation. >>>>> >>>>> Obviously, some of my points can be answered with "look here in the >>> code" >>>>> or look at this page, but I will try to outline my thought processes >>> as I >>>>> reviewed the current docs. Sometimes the way I approached the >> problem >>>> led >>>>> me down a path of to a certain page, missing the answer in a >> different >>>>> page, and thus some cross linking could be helpful. >>>>> >>>>> Please do not let my points be taken as anything other than a desire >> to >>>>> improve how accessible Myriad is to the community, this is not a >>> critique >>>>> of the hard work everyone has done on the project. I also understand >>>> that >>>>> given the work load and other issues, that fixing these issues in >>>>> documentation may not be a priority. I am listing them out here, so >>> that >>>>> those folks who are SMEs on various points may be able to quickly add >>>> stuff >>>>> and we'll organize it later. >>>>> >>>>> >>>>> *Remote Distribution: * >>>>> >>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution >>>>> >>>>> This whole section could use some work from a standpoint of what runs >>>> where >>>>> and where that component gets its files. For example, I think it >> would >>>>> help people to understand that the whole tarball created in step 6 >> has >>>> all >>>>> the files for node managers and resource managers. Basically, >>> everything >>>>> runs from there. Here is a small example I am currently working with: >>>>> >>>>> >>>>> Starting Myriad: >>>>> Option 1: Use Marathon (provide example json, here is mine) >>>>> { >>>>> "cmd": "env && export >>>>> >>>>> >>>> >>> >> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos >>>>> && hadoop-2.7.0/bin/yarn resourcemanager", >>>>> "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"], >>>>> "cpus": 1.0, >>>>> "mem": 1024, >>>>> "id": "myriad", >>>>> "instances": 1, >>>>> "user": "mapr" >>>>> } >>>>> >>>>> In this case, Marathon grabs the hadoop tarball and pulls it down, >> this >>>>> tarball also has the Myriad yml file. When it executes the resource >>>>> manager, it is brought up in Myriad and ready to run node managers by >>>>> pulling the tarball to the slave nodes and executing the nodemanager. >>> (I >>>>> would imagine the work with history server etc would also use this >>>>> tarball?). >>>>> >>>>> From here it will us NMInstances to launch a node manager. (Note, >> this >>>> is >>>>> different from when I originally set things up... before, I could run >>> the >>>>> resource manager/myriad without a nodemanager, now it seems it's >>> required >>>>> based on the config in the src... could we expound on this in the >> docs >>>>> somewhere?) >>>>> >>>>> >>>>> Option 2: ???? (Are there other ways to launch the resource manager?) >>>>> >>>>> Step 6: So something that is unclear to me is the handling of the >>>>> hadoop/yarn config files. In Step 6 on this page, there is "sudo rm >>>>> hadoop- >>>>> 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually >>>> ignored >>>>> this step. For me, if I remove these xml files, then there is no >> place >>>> to >>>>> get my files... I think? Since I am running the RM and NM from the >> same >>>>> tarball, and Myriad config is here, and my goal is to not have >> anything >>>>> installed on a node, where would I set yarn settings? This could be >>> much >>>>> clearer to me, and probably others. >>>>> >>>>> Step 2: Should we just be copying the Myriad files to >>>>> /share/hadoop/yarn/lib folder? Do we worry about potentially >> overwrites >>>> of >>>>> jars or version conflicts? >>>>> >>>>> *Configuring Cgroups* >>>>> >> https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups >>>>> At some point a little bit more about why one would want CGroups and >>>> issues >>>>> that could occur with them. While many folks using Mesos/Myriad may >>>>> understand this, others may not, and it's a good way to help people >>> think >>>>> positively about our project if we help educate them along the way. >>>>> >>>>> Minor point on enabling CGroups. This is confusing given my questions >>> in >>>>> remote distribution. in this it says I need to edit my yarn-site.xml, >>> but >>>>> in remote distribution it says delete my hadoop xml files. We need to >>>>> address this conflict cause it can be confusing for a user coming >>> onboard >>>>> >>>>> Nitpick: Enabling cgroups for mess-slave - should be - Enabling >> cgroups >>>> for >>>>> mesos-slave >>>>> >>>>> *Myriad Configuration Properties*: >>>>> >>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties >>>>> Based on the conversation on list with Yuliya, >>>>> "Currently, this file is built into Myriad Scheduler jar. So, if you >>> need >>>>> to modify some of the properties in this file, modify them before >>>> building >>>>> Myriad Scheduler." >>>>> isn't accurate any more, and we should address that. >>>>> >>>>> The configuration file in the wiki is an old one, the nmInstances >> isn't >>>> in >>>>> it, (and see my question about that above). >>>>> >>>>> Frameworks and usernames. I think the users that the framework runs >>> as, >>>>> the actual node and resource managers, etc is confusing to a user (I >> am >>>>> very confused!) When I first got Myriad up I set my user under the >>>>> executor to be mapr, and then it appeared to work with impersonation >>> from >>>>> queries etc. Now, I am trying the remote distribution and I have >> users >>>> set >>>>> in the config, potentially a user in my marathon json, and I am >> getting >>>>> errors on permissions of files when a node manager tries to start (a >>>>> separate issue I will post later). Basically, this is complex, and a >>> page >>>>> describing out what needs to run where with which permissions and how >>>> that >>>>> interacts will be huge for people looking to put this into play. >>>>> >>>>> *Example Yarn Site:* >>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml >>>>> >>>>> This is helpful, but where does it go? Remember, the remote >>> distribution >>>>> had us delete the yarn-site in the hadoop etc folder. >>>>> >>>>> *Myriad Webapp * >>>>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp >>>>> >>>>> This should be fleshed out a bit more. Also, it's in the >>>>> /myriad-scheduler/src/main/resources/webapp based on my git clone, >> but >>> in >>>>> the wiki that's not listed. I had to dig for it. >>>>> >>>>> Some questions here: could the webapp be built during the myriad >>>> building >>>>> process? Could it be then be packaged as tarball for execution either >>>>> manually via marathon or automatically in a container on mesos? I >>>>> understand this is a fresh piece of the puzzle, I am just thinking >>> about >>>>> and verbalizing the "where" on this for the future >>>>> >>>>> >>>>> >>>>> Those are the items that come to mind thus far. I hope the tone of >> my >>>>> email is correct, this is a great project, and I want others to try >> it >>>> as I >>>>> have. >>>>> >>>>> John Omernik >>>>> >>>> >>> >> >> >> >> -- >> Ruth Harris >> Sr. Technical Writer, MapR >>
