Thanks Bill. I checked the HTTP request. The sent packet data is: GET /api HTTP/1.1 Host: 54.209.127.223:8081 User-Agent: Go-http-client/1.1 Accept-Encoding: gzip
I can attach the complete capture if that is needed. Link to my code on github: https://github.com/krish7919/aurora_thrift_api -- κρισhναν On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <[email protected]> wrote: > Looks to me like it's failing before the request is even deserialized - in > the thrift layer (the exception is thrown here > <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132> > - EOF). Can you capture the HTTP request that was sent by the client? > That might provide more clues. > > Also, if you're able to put up a git repo with your client code, i can > poke at it. > > On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected]> wrote: > >> Hi Bill, >> Tried digging more about aurora thrift API this weekend. >> The thrift generated code is a good reference point. >> >> You are right; the '/' path just gives me the list of URLs that one gets >> on the browser when I query it using my thrift client. >> Any pointers based on the data below will be very helpful to find out why >> is aurora bailing out when processing the thrift request, or is it some >> client side error that is causing it. >> >> >> My thrift client trying to query for getJobs: >> >> .... >> var protocolFactory thrift.TProtocolFactory >> var transport thrift.TTransport >> var client *api.ReadOnlySchedulerClient >> var err error >> transport, err = thrift.NewTHttpClient("http://54.209.17.221:8081/api >> ") >> defer transport.Close() >> protocolFactory = thrift.NewTJSONProtocolFactory() >> client = api.NewReadOnlySchedulerClientFactory(transport, >> protocolFactory) >> err = transport.Open() >> if err != nil { >> fmt.Println("Error opening socket: ", err) >> os.Exit(1) >> } >> defer transport.Close() >> fmt.Println(client.GetJobs("")) >> .... >> >> >> I did a wireshark analysis of outgoing packets to aurora, & I do get a >> response packet from aurora, and my thrift client bails out with an error >> (runtime error: invalid memory address or nil pointer dereference). The >> data in the packet sent by server is: >> >> <html> >> <head> >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> >> <title>Error 500 </title> >> </head> >> <body> >> <h2>HTTP ERROR: 500</h2> >> <p>Problem accessing /api. Reason: >> <pre> javax.servlet.ServletException: >> org.apache.thrift.transport.TTransportException</pre></p> >> <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// >> 9.3.6.v20151106</a><hr/> >> </body> >> </html> >> >> >> >> >> Stacktrace from the server/aurora console logs: >> >> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235 >> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException: >> org.apache.thrift.transport.TTransportException >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.server.TServlet.doPost(TServlet.java:86) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.server.TServlet.doGet(TServlet.java:96) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >> ~[javax.servlet-api-3.1.0.jar:3.1.0] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> ~[javax.servlet-api-3.1.0.jar:3.1.0] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72) >> ~[aurora-0.12.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >> ~[aurora-0.12.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71) >> ~[aurora-0.12.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >> ~[aurora-0.12.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >> ~[guice-servlet-3.0.jar:na] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) >> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) >> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) >> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) >> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) >> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >> ... >> ... >> ... >> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by: >> org.apache.thrift.transport.TTransportException: null >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >> org.apache.thrift.server.TServlet.doPost(TServlet.java:83) >> ~[libthrift-0.9.1.jar:0.9.1] >> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted >> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235 >> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - - >> [19/Mar/2016:18:12:23 + >> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389 >> >> >> >> >> -- >> κρισhναν >> >> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected]> wrote: >> >>> 8081 is indeed the default thrift port. If you can capture a raw HTTP >>> request, we could diagnose this better. My first hunch is that the thrift >>> client expects the API to be mounted at /, when in fact we mount it at /api. >>> >>> Jake can probably tell you exactly what's wrong based on his code, but >>> in the meantime you might find it helpful to compare against how we set up >>> the JS and python clients: >>> >>> >>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188 >>> >>> >>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115 >>> >>> >>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected]> >>> wrote: >>> >>>> Apologies for multiple mails, the previous email was sent accidentally. >>>> I didn't add a problem description, before hitting send. >>>> >>>> When I query aurora for all the jobs using getJobs, I find the aurora >>>> error as given below and the response I get using my client. >>>> >>>> -- >>>> κρισhναν >>>> >>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]> >>>> wrote: >>>> >>>>> Thanks Jake! >>>>> >>>>> That worked like a charm, & I was wondering why does install from >>>>> source doesn't work! >>>>> >>>>> Aurora Log: >>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 >>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for >>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} >>>>> >>>>> Thrift client: >>>>> ./mrfantastic_service.out >>>>> Connecting to aurora.... >>>>> <nil> EOF >>>>> >>>>> Thrift client sourcec: >>>>> func thriftAuroraJobs() { >>>>> var protocolFactory thrift.TProtocolFactory >>>>> var transportFactory thrift.TTransportFactory >>>>> var transport thrift.TTransport >>>>> var client *api.ReadOnlySchedulerClient >>>>> var err error >>>>> >>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() >>>>> //transportFactory = >>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) >>>>> transportFactory = thrift.NewTTransportFactory() >>>>> fmt.Println("Connecting to aurora....") >>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081") >>>>> if err != nil { >>>>> fmt.Println("Error opening socket:", err) >>>>> os.Exit(1) >>>>> //return err >>>>> } >>>>> if transport == nil { >>>>> os.Exit(1) >>>>> } >>>>> transport = transportFactory.GetTransport(transport) >>>>> if transport == nil { >>>>> os.Exit(1) >>>>> } >>>>> err = transport.Open() >>>>> if err != nil { >>>>> os.Exit(1) >>>>> } >>>>> defer transport.Close() >>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>> protocolFactory) >>>>> fmt.Println(client.GetJobs("")) >>>>> } >>>>> >>>>> >>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is >>>>> looking for HTTP data on the socket. >>>>> Is there a config that needs to be set for thrift API to be >>>>> initialized? >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> κρισhναν >>>>> >>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]> >>>>> wrote: >>>>> >>>>>> if you just need the compiler you can install the thrift-compiler >>>>>> package with one of the following, otherwise you can run ./bootstrap.sh >>>>>> && >>>>>> ./configure && cd compiler/cpp && make to just build the compiler. >>>>>> >>>>>> -Jake >>>>>> >>>>>> >>>>>> >>>>>> deb packaging (tested with ubuntu trusty) >>>>>> >>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add - >>>>>> > /etc/apt/sources.list.d/thrift.list >>>>>> >>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >>>>>> >>>>>> >>>>>> or for centos/rhel (tested with centos 7.2) >>>>>> >>>>>> > /etc/yum.repos.d/thrift.repo >>>>>> >>>>>> [thrift] >>>>>> name=Apache Thrift rpm repo >>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/ >>>>>> enabled=1 >>>>>> gpgcheck=0 >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Jake/Chris, >>>>>>> Thanks for the info. >>>>>>> When I try to install thrift v0.9.3 from source, I get an error as >>>>>>> follows while running `make check`: >>>>>>> ... >>>>>>> ... >>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time >>>>>>> elapsed: 0.062 sec >>>>>>> [junit] Running >>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol >>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time >>>>>>> elapsed: 0.046 sec >>>>>>> >>>>>>> BUILD FAILED >>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>>>>>> >>>>>>> Total time: 17 seconds >>>>>>> make[3]: *** [check-local] Error 1 >>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>> make[2]: *** [check-am] Error 2 >>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>> make[1]: *** [check-recursive] Error 1 >>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>>>>>> make: *** [check-recursive] Error 1 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> κρισhναν >>>>>>> >>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I've used the latest thrift to generate go code, and then manually >>>>>>>> created executor config which works and is able to launch jobs. >>>>>>>> >>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Krish >>>>>>>>> We are using Thrift with go for all our api calls to Aurora, would >>>>>>>>> recommend you use Thrift 0.9.3 to interact with the api. >>>>>>>>> >>>>>>>>> happy to help answer any questions you might have >>>>>>>>> >>>>>>>>> -Jake >>>>>>>>> >>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks, Bill. >>>>>>>>>> >>>>>>>>>> Well I have started my foray into the the thrift API today. And I >>>>>>>>>> think I am stuck with some thrift configs. >>>>>>>>>> >>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk >>>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find >>>>>>>>>> any >>>>>>>>>> changelog or compatibility statement on the thrift project site. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian >>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from >>>>>>>>>> source. >>>>>>>>>> However, when I try to generate golang code, I think I hit a >>>>>>>>>> compiler bug: >>>>>>>>>> krish@krish:/tmp >>>>>>>>>> > thrift --gen go api.thrift >>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal >>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>>>>>> >>>>>>>>>> I can modify the golang code by hand, but I would like to play it >>>>>>>>>> safe and use the working compiler from the debian repos. >>>>>>>>>> >>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a >>>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone >>>>>>>>>> faced a >>>>>>>>>> similar error and gone past it? >>>>>>>>>> I have already done a `go get >>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> κρισhναν >>>>>>>>>> >>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't much >>>>>>>>>>> in the way of independent/holistic docs for the thrift API. There >>>>>>>>>>> is, >>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec >>>>>>>>>>> itself: >>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>>>>>> >>>>>>>>>>> If you are looking to use the thrift API directly, the most >>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value >>>>>>>>>>> when >>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if >>>>>>>>>>> you get >>>>>>>>>>> to that point! >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and >>>>>>>>>>>>> return thrift values only in Response.result field. >>>>>>>>>>>> >>>>>>>>>>>> Correct. There is also 'details' field that may have additional >>>>>>>>>>>> messages (of error or informational nature) >>>>>>>>>>>> >>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>> >>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to >>>>>>>>>>>> bring up a fully supported REST API that will presumably get >>>>>>>>>>>> documented and >>>>>>>>>>>> become much easier to use. It's mostly in flux now. >>>>>>>>>>>> >>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>> >>>>>>>>>>>> This is the API to use when you a brand new service or adhoc >>>>>>>>>>>> (batch) job created. The JobDescription is populated from the >>>>>>>>>>>> .aurora >>>>>>>>>>>> config. You may want to trace "aurora job create" client command >>>>>>>>>>>> implementation to see how it happens. >>>>>>>>>>>> >>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>> object as >>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does >>>>>>>>>>>>> not create >>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>> >>>>>>>>>>>> Ignore this object as it's an echo of the old client updater. >>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL >>>>>>>>>>>> for now. >>>>>>>>>>>> >>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>> >>>>>>>>>>>> Another echo of the client updater but this time it's got a >>>>>>>>>>>> second life. Check out its new signature and comments in the >>>>>>>>>>>> api.thrift. >>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the >>>>>>>>>>>> existing >>>>>>>>>>>> job without changing the underlying task assumptions. >>>>>>>>>>>> >>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>> >>>>>>>>>>>> It's actually 'getPendingReason' and is currently used >>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state. >>>>>>>>>>>> >>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>> >>>>>>>>>>>> This is to set role-level quota. Currently only required for >>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more >>>>>>>>>>>> details. >>>>>>>>>>>> >>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>>> cluster. >>>>>>>>>>>> >>>>>>>>>>>> It's quite versatile and can be used to kill some or all >>>>>>>>>>>> instances of the job. >>>>>>>>>>>> >>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>>>>> >>>>>>>>>>>> Your observations are correct. This is the main API to change >>>>>>>>>>>> your service job in any way (including adding, removing or >>>>>>>>>>>> modifying >>>>>>>>>>>> instances). >>>>>>>>>>>> >>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>> them, does >>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>>> >>>>>>>>>>>> Scheduler will try to schedule as many instances as it can. >>>>>>>>>>>> Those that will not find resources will remain in PENDING state >>>>>>>>>>>> until more >>>>>>>>>>>> resources are available. In your particular example only the 5th >>>>>>>>>>>> will >>>>>>>>>>>> remain PENDING. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to >>>>>>>>>>>>> add new jobs. >>>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded >>>>>>>>>>>>> to mesos v0.25 accordingly. >>>>>>>>>>>>> >>>>>>>>>>>>> Below is a summary of my (very limited) understanding of some >>>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my >>>>>>>>>>>>> understanding: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs >>>>>>>>>>>>> specified, and return thrift values only in Response.result >>>>>>>>>>>>> field. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>>> >>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>> This is basically the API to replace the Aurora >>>>>>>>>>>>> DSL/.aurora files for job configuration. >>>>>>>>>>>>> >>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>> object as >>>>>>>>>>>>> parameter, & I am assuming that it is required so that one >>>>>>>>>>>>> does not create >>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>>> >>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>> By the naming convention, it seems this is used to >>>>>>>>>>>>> increase the number of instances of a job. It will not result >>>>>>>>>>>>> in stopping >>>>>>>>>>>>> of current instances of the job. >>>>>>>>>>>>> >>>>>>>>>>>>> My second explanation for this API: Since it uses a set of >>>>>>>>>>>>> instanceIds, this is used for adding already running job in >>>>>>>>>>>>> slaves to the >>>>>>>>>>>>> internal data structures of Aurora to track the job. >>>>>>>>>>>>> >>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>> Return the reason (in string) about why the job is >>>>>>>>>>>>> PENDING. For example: insufficient CPU. >>>>>>>>>>>>> >>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>> >>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>>> cluster. >>>>>>>>>>>>> >>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string >>>>>>>>>>>>> message): >>>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified. >>>>>>>>>>>>> Can be used if resource requirement changes. For example: If I >>>>>>>>>>>>> wanted >>>>>>>>>>>>> aurora to update the version of container used for a job using >>>>>>>>>>>>> TaskConfig.Container attribute. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>> them, does >>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
