Apologies for multiple mails, the previous email was sent accidentally. I didn't add a problem description, before hitting send.
When I query aurora for all the jobs using getJobs, I find the aurora error as given below and the response I get using my client. -- κρισhναν On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]> wrote: > Thanks Jake! > > That worked like a charm, & I was wondering why does install from source > doesn't work! > > Aurora Log: > Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109 > [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for > HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} > > Thrift client: > ./mrfantastic_service.out > Connecting to aurora.... > <nil> EOF > > Thrift client sourcec: > func thriftAuroraJobs() { > var protocolFactory thrift.TProtocolFactory > var transportFactory thrift.TTransportFactory > var transport thrift.TTransport > var client *api.ReadOnlySchedulerClient > var err error > > protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() > //transportFactory = > thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) > transportFactory = thrift.NewTTransportFactory() > fmt.Println("Connecting to aurora....") > transport, err = thrift.NewTSocket("54.210.234.190:8081") > if err != nil { > fmt.Println("Error opening socket:", err) > os.Exit(1) > //return err > } > if transport == nil { > os.Exit(1) > } > transport = transportFactory.GetTransport(transport) > if transport == nil { > os.Exit(1) > } > err = transport.Open() > if err != nil { > os.Exit(1) > } > defer transport.Close() > client = api.NewReadOnlySchedulerClientFactory(transport, > protocolFactory) > fmt.Println(client.GetJobs("")) > } > > > My hunch is that the thrift port isn't 8081, as the server/aurora is > looking for HTTP data on the socket. > Is there a config that needs to be set for thrift API to be initialized? > > > > > -- > κρισhναν > > On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]> > wrote: > >> if you just need the compiler you can install the thrift-compiler package >> with one of the following, otherwise you can run ./bootstrap.sh && >> ./configure && cd compiler/cpp && make to just build the compiler. >> >> -Jake >> >> >> >> deb packaging (tested with ubuntu trusty) >> >> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >> > gpg --export --armor 66B778F9 | sudo apt-key add - >> > /etc/apt/sources.list.d/thrift.list >> >> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >> >> >> or for centos/rhel (tested with centos 7.2) >> >> > /etc/yum.repos.d/thrift.repo >> >> [thrift] >> name=Apache Thrift rpm repo >> baseurl=http://www.apache.org/dist/thrift/rpm/ >> enabled=1 >> gpgcheck=0 >> >> >> >> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]> >> wrote: >> >>> Jake/Chris, >>> Thanks for the info. >>> When I try to install thrift v0.9.3 from source, I get an error as >>> follows while running `make check`: >>> ... >>> ... >>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time >>> elapsed: 0.062 sec >>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol >>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time >>> elapsed: 0.046 sec >>> >>> BUILD FAILED >>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>> >>> Total time: 17 seconds >>> make[3]: *** [check-local] Error 1 >>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>> make[2]: *** [check-am] Error 2 >>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>> make[1]: *** [check-recursive] Error 1 >>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>> make: *** [check-recursive] Error 1 >>> >>> >>> >>> >>> -- >>> κρισhναν >>> >>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <[email protected]> >>> wrote: >>> >>>> I've used the latest thrift to generate go code, and then manually >>>> created executor config which works and is able to launch jobs. >>>> >>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]> >>>> wrote: >>>> >>>>> Hi Krish >>>>> We are using Thrift with go for all our api calls to Aurora, would >>>>> recommend you use Thrift 0.9.3 to interact with the api. >>>>> >>>>> happy to help answer any questions you might have >>>>> >>>>> -Jake >>>>> >>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks, Bill. >>>>>> >>>>>> Well I have started my foray into the the thrift API today. And I >>>>>> think I am stuck with some thrift configs. >>>>>> >>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with >>>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any >>>>>> changelog or compatibility statement on the thrift project site. >>>>>> >>>>>> >>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos >>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source. >>>>>> However, when I try to generate golang code, I think I hit a compiler >>>>>> bug: >>>>>> krish@krish:/tmp >>>>>> > thrift --gen go api.thrift >>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal >>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>> >>>>>> I can modify the golang code by hand, but I would like to play it >>>>>> safe and use the working compiler from the debian repos. >>>>>> >>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test >>>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a >>>>>> similar error and gone past it? >>>>>> I have already done a `go get >>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Regarding documentation - Maxim is correct that there isn't much in >>>>>>> the way of independent/holistic docs for the thrift API. There is, >>>>>>> however, scant javadoc-style documentation within the IDL spec itself: >>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>> >>>>>>> If you are looking to use the thrift API directly, the most >>>>>>> difficult API method will be defining the ExecutorConfig.data value when >>>>>>> calling createJob. Please don't hesitate to ask for assistance if you >>>>>>> get >>>>>>> to that point! >>>>>>> >>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> 1. All APIs require thrift inputs of the structs specified, and >>>>>>>>> return thrift values only in Response.result field. >>>>>>>> >>>>>>>> Correct. There is also 'details' field that may have additional >>>>>>>> messages (of error or informational nature) >>>>>>>> >>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>> understand Thrift API better? >>>>>>>> >>>>>>>> The thrift API is largely undocumented. There is an effort to bring >>>>>>>> up a fully supported REST API that will presumably get documented and >>>>>>>> become much easier to use. It's mostly in flux now. >>>>>>>> >>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>> >>>>>>>> This is the API to use when you a brand new service or adhoc >>>>>>>> (batch) job created. The JobDescription is populated from the .aurora >>>>>>>> config. You may want to trace "aurora job create" client command >>>>>>>> implementation to see how it happens. >>>>>>>> >>>>>>>> 4. What is the Lock object? I see that some APIs require locking >>>>>>>>> and some don't. For example, createJob needs a Lock object as >>>>>>>>> parameter, & >>>>>>>>> I am assuming that it is required so that one does not create >>>>>>>>> multiple jobs >>>>>>>>> with the same JobKey. >>>>>>>> >>>>>>>> Ignore this object as it's an echo of the old client updater. It's >>>>>>>> now deprecated and will be removed soon. You can pass NULL for now. >>>>>>>> >>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>> >>>>>>>> Another echo of the client updater but this time it's got a second >>>>>>>> life. Check out its new signature and comments in the api.thrift. It's >>>>>>>> essentially a "scale-out" API that can add instances to the existing >>>>>>>> job >>>>>>>> without changing the underlying task assumptions. >>>>>>>> >>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>> >>>>>>>> It's actually 'getPendingReason' and is currently used exclusively >>>>>>>> by the UI to get the reason for a task PENDING state. >>>>>>>> >>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>> >>>>>>>> This is to set role-level quota. Currently only required for tasks >>>>>>>> with 'production=True'. Search through our docs for more details. >>>>>>>> >>>>>>>> 8. killTasks to kill all running instances of a job in the cluster. >>>>>>>> >>>>>>>> It's quite versatile and can be used to kill some or all instances >>>>>>>> of the job. >>>>>>>> >>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>> >>>>>>>> Your observations are correct. This is the main API to change your >>>>>>>> service job in any way (including adding, removing or modifying >>>>>>>> instances). >>>>>>>> >>>>>>>> An aurora scheduling question is if I start a job with 5 instances, >>>>>>>>> and there are resources available to run only 4 of them, does the >>>>>>>>> entire >>>>>>>>> job block, or only the 5th instance of the job blocks? >>>>>>>> >>>>>>>> Scheduler will try to schedule as many instances as it can. Those >>>>>>>> that will not find resources will remain in PENDING state until more >>>>>>>> resources are available. In your particular example only the 5th will >>>>>>>> remain PENDING. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> I was going through the Aurora Thrift API to determine how to add >>>>>>>>> new jobs. >>>>>>>>> I am using aurora v0.12 released last month and have upgraded to >>>>>>>>> mesos v0.25 accordingly. >>>>>>>>> >>>>>>>>> Below is a summary of my (very limited) understanding of some >>>>>>>>> APIs, & would help it if someone can point out flaws in my >>>>>>>>> understanding: >>>>>>>>> >>>>>>>>> 1. All APIs require thrift inputs of the structs specified, >>>>>>>>> and return thrift values only in Response.result field. >>>>>>>>> >>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>> understand Thrift API better? >>>>>>>>> >>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora >>>>>>>>> files for job configuration. >>>>>>>>> >>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>> locking and some don't. For example, createJob needs a Lock object >>>>>>>>> as >>>>>>>>> parameter, & I am assuming that it is required so that one does >>>>>>>>> not create >>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>> >>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>> By the naming convention, it seems this is used to increase >>>>>>>>> the number of instances of a job. It will not result in stopping >>>>>>>>> of current >>>>>>>>> instances of the job. >>>>>>>>> >>>>>>>>> My second explanation for this API: Since it uses a set of >>>>>>>>> instanceIds, this is used for adding already running job in slaves >>>>>>>>> to the >>>>>>>>> internal data structures of Aurora to track the job. >>>>>>>>> >>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>> Return the reason (in string) about why the job is PENDING. >>>>>>>>> For example: insufficient CPU. >>>>>>>>> >>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>> >>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>> cluster. >>>>>>>>> >>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>> Used for updating jobs with the new TaskConfig specified. Can >>>>>>>>> be used if resource requirement changes. For example: If I wanted >>>>>>>>> aurora to >>>>>>>>> update the version of container used for a job using >>>>>>>>> TaskConfig.Container >>>>>>>>> attribute. >>>>>>>>> >>>>>>>>> >>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>> instances, and there are resources available to run only 4 of them, >>>>>>>>> does >>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> -- >>>>>>>>> κρισhναν >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> >
