Thanks, Yaniv. 1. I was able to succeed partially on the setup front. I am able to run mesos inside a vagrant box and submit the job inside the box using host 127.0.0.1. I believe that the right way to use the vagrant box is just for mesos and not for submitting the jobs. I would like to submit the job from the host machine, which I am pretty sure must be possible. There's some network configuration which I am missing.
a. Either I have a job which doesn't move as an Active framework (always stick on Inactive framework) or b. I get a "Transport endpoint is not connected". I am going to spend some more time on this and will bother you all. Meanwhile, I am also trying to setup mesos on my host machine so that I could still use the localhost to submit the jobs. Was able to do it on Ubuntu but not on Mac :-( 2. I believe that I have some basic understanding of mesos in the short time I spent and I am able to traverse through the logs through the UI as well as the console. Looking forward to the YARN push. Thanks !! 3. Dev setup - I'll get back to you shortly. 4. External context : Yes, more than happy to be of help. If there's some design work previously done for this, please let me know. Else, I'll come up with a draft for a review. Cheers, Arun On Sun, Dec 31, 2017 at 8:29 PM Yaniv Rodenski <ya...@shinto.io> wrote: > Hi Arun, > > Happy new year! > > As for your questions: > 1. this seems to be a new issue with the vagrant image. Java used to be > installed with the image used in the vagrant script, really annoying but > I'll fix as per your suggestion and push it asap. BTW I myself had issues > running Mesos on MacOS but it worked fine on both CentOS and Ubuntu so that > might be a source of issues. > 2. The first thing I would look into is the executor logs, the easiest way > to get those is via the Mesos UI (by default port 5050 of the vagrant > master) also a really good tool for debugging Amaterasu jobs on Mesos, as a > side note, YARN is really around the corner, so you will get the same logs > via the YARN ResourceManager UI. > 3. Not sure, I haven't tried that but I'm happy to have a look and help you > try, can you push that to a branch on your own fork? I'll try to have a > look > 4. Supporting multiple repositories is planned for version 0.3.0 > > https://issues.apache.org/jira/projects/AMATERASU/issues/AMATERASU-9?filter=allopenissues > and also support for binaries from external repositories > > https://issues.apache.org/jira/projects/AMATERASU/issues/AMATERASU-8?filter=allopenissues > which seems to be what you are after. If you are interested in helping out > with any of them it would be much appreciated :) > One thing that you should note is that AmaContext is created outside of the > Spark job and injected by Amaterasu, so you can just access it in your job, > For pyspark jobs, it will be ama_context and same for SparkSQL. > > Hope that helps, > Yaniv > > On Sun, Dec 31, 2017 at 4:35 PM, Arun Manivannan <a...@arunma.com> wrote: > > > Hi, > > > > Very Good morning and Wish you all a wonderful New Year ahead. > > > > Sorry to bother you on New Year's eve but I really appreciate any hints. > > > > *1. Setting up and running the basic setup (on MacOS High Sierra) > > (Solution found): * > > > > I remember having done this successfully before but it was strange this > > time. > > > > a. Cloned amaterasu and amaterasu-vagrant repos. > > b. Built amaterasu using ./gradlew buildHomeDir test (and later tried > with > > buildDistribution) > > c. Have a vagrant box up and running (have modified the location of the > > sync folder to point to amaterasu's build/amaterasu directory) > > d. Have installed mesos locally using brew and set the > > MESOS_NATIVE_JAVA_LIBRARY to point to /usr/local/lib/libmesos.dylib > > e. Did a > > > > ama-start.sh --repo=" > https://github.com/shintoio/amaterasu-job-sample.git"; --branch="master" > --env="test" --report="code" > > > > > > I found from mesos that zookeeper wasn't running and from zookeeper logs > > that java wasn't installed. I manually installed java (sudo yum install > > java-1.7.0-openjdk-devel) and then started zookeeper (service > > zookeeper-server start). Mesos came up. > > > > *Question 1 : Is it okay if I add the java installation as part of the > > provision.sh or did I miss anything earlier in order to bump into this > > issue?* > > > > 2. Upon submitting the job, I saw that the job didn't run fine. > > > > > > *Client logs (./ama-start.sh)* > > > > I1231 13:24:01.548068 193015808 sched.cpp:232] Version: 1.3.0 > > I1231 13:24:01.554518 226975744 sched.cpp:336] New master detected at > > master@192.168.33.11:5050 > > I1231 13:24:01.554733 226975744 sched.cpp:352] No credentials provided. > > Attempting to register without authentication > > I1231 13:24:01.558101 226975744 sched.cpp:759] Framework registered with > > 70566146-bf07-4515-aa0c-ed7fd597efe3-0019 > > ===> moving to err action null > > 2017-12-31 13:24:08.884:INFO:oejs.ServerConnector:Thread-22: Stopped > > ServerConnector@424e1977{HTTP/1.1}{0.0.0.0:8000} > > 2017-12-31 13:24:08.886:INFO:oejsh.ContextHandler:Thread-22: Stopped > > o.e.j.s.ServletContextHandler@7bedc48a{/,file:/Users/arun/ > > IdeaProjects/amaterasu/build/amaterasu/dist/,UNAVAILABLE} > > 2017-12-31 13:24:08.886:INFO:oejsh.ContextHandler:Thread-22: Stopped > > o.e.j.s.h.ContextHandler@4802796d{/,null,UNAVAILABLE} > > I1231 13:24:08.887260 229122048 sched.cpp:2021] Asked to stop the driver > > I1231 13:24:08.887711 229122048 sched.cpp:1203] Stopping framework > > 70566146-bf07-4515-aa0c-ed7fd597efe3-0019 > > > > *(MESOS LOGS as ATTACHMENT MesosMaster-INFO.log)* > > > > *Question 2 : Hints please* > > > > *2. Dev setup* > > I would like to debug the program to understand how the flow of code > > within Amaterasu. I have attempted the following and that gives the same > > result as above. Created a main program that invokes the job launcher > with > > the following parameters (this class is in test directory primarily to > > bring in the "provided" libraries. Not sure if that makes sense). > > > > *Program arguments : * > > -Djava.library.path=/usr/lib > > *Environment variables : (not sure why System.setProperty doesn't picked > > up)* > > AMA_NODE = MacBook-Pro.local > > MESOS_NATIVE_JAVA_LIBRARY = /usr/local/lib/libmesos.dylib > > > > object JobLauncherDebug extends App { > > JobLauncher.main(Array( > > "--home", "/Users/arun/IdeaProjects/amaterasu/build/amaterasu", > > "--repo", "https://github.com/shintoio/amaterasu-job-sample.git";, > > "--branch", "master", > > "--env", "test", > > "--report", "code" > > )) > > } > > > > *Question 3 : Would this be a good idea to go about debugging into the > > code?* > > > > *3. Pipelining two Spark jobs* > > > > My ultimate goal would be to run the following use-case using Amaterasu. > > This is very similar to what I do at work. > > > > a. Create a bunch of Hive tables (and hdfs directories) using a Spark job > > (that will be used as a deployment script - one time setup but no harm in > > running it again since it has the "if not exists" clause) ( > > https://github.com/arunma/ama_schemapreparer) > > b. Run another Spark job that populates the data (this job runs on > regular > > intervals throughout the day) ( > https://github.com/arunma/ama_datapopulator) > > > > c. Run a different Spark job that reconciles the populated data. > > > > I am yet to create the "job" project for this one which I intend to do > > once I have the default testcase running. > > > > *Question 4 :* > > A couple of hurdles that I believe I would have is that Amaterasu, at the > > moment, > > > > a. Expects the Spark jobs to be in the same repository. > > b. The file that instantiates the Spark session, context etc has to be > > explicitly given as a ".scala" file (we then use the IMain interpreter to > > inject the AmaContext?) > > > > Now, with two repositories in play and only the binaries/repository name > > given for the repo, would it be a good idea to achieve the AmaContext > > insertion using a compiler plugin? I am pretty sure this has been > > discussed before and it would be great if you could share your views on > > this. I can come up with a POC PR of sorts if I get some ideas. > > > > Best Regards, > > Arun > > > > > > > -- > Yaniv Rodenski >
