Looks to me like it's failing before the request is even deserialized - in the thrift layer (the exception is thrown here <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132> - EOF). Can you capture the HTTP request that was sent by the client? That might provide more clues.
Also, if you're able to put up a git repo with your client code, i can poke at it. On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected]> wrote: > Hi Bill, > Tried digging more about aurora thrift API this weekend. > The thrift generated code is a good reference point. > > You are right; the '/' path just gives me the list of URLs that one gets > on the browser when I query it using my thrift client. > Any pointers based on the data below will be very helpful to find out why > is aurora bailing out when processing the thrift request, or is it some > client side error that is causing it. > > > My thrift client trying to query for getJobs: > > .... > var protocolFactory thrift.TProtocolFactory > var transport thrift.TTransport > var client *api.ReadOnlySchedulerClient > var err error > transport, err = thrift.NewTHttpClient("http://54.209.17.221:8081/api > ") > defer transport.Close() > protocolFactory = thrift.NewTJSONProtocolFactory() > client = api.NewReadOnlySchedulerClientFactory(transport, > protocolFactory) > err = transport.Open() > if err != nil { > fmt.Println("Error opening socket: ", err) > os.Exit(1) > } > defer transport.Close() > fmt.Println(client.GetJobs("")) > .... > > > I did a wireshark analysis of outgoing packets to aurora, & I do get a > response packet from aurora, and my thrift client bails out with an error > (runtime error: invalid memory address or nil pointer dereference). The > data in the packet sent by server is: > > <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> > <title>Error 500 </title> > </head> > <body> > <h2>HTTP ERROR: 500</h2> > <p>Problem accessing /api. Reason: > <pre> javax.servlet.ServletException: > org.apache.thrift.transport.TTransportException</pre></p> > <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// > 9.3.6.v20151106</a><hr/> > </body> > </html> > > > > > Stacktrace from the server/aurora console logs: > > Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235 > [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException: > org.apache.thrift.transport.TTransportException > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.server.TServlet.doPost(TServlet.java:86) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.server.TServlet.doGet(TServlet.java:96) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > ~[javax.servlet-api-3.1.0.jar:3.1.0] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > ~[javax.servlet-api-3.1.0.jar:3.1.0] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72) > ~[aurora-0.12.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) > ~[aurora-0.12.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71) > ~[aurora-0.12.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) > ~[aurora-0.12.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > ~[guice-servlet-3.0.jar:na] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) > ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) > [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) > [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) > [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) > [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] > ... > ... > ... > Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by: > org.apache.thrift.transport.TTransportException: null > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: at > org.apache.thrift.server.TServlet.doPost(TServlet.java:83) > ~[libthrift-0.9.1.jar:0.9.1] > Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted > Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235 > [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - - > [19/Mar/2016:18:12:23 + > 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389 > > > > > -- > κρισhναν > > On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected]> wrote: > >> 8081 is indeed the default thrift port. If you can capture a raw HTTP >> request, we could diagnose this better. My first hunch is that the thrift >> client expects the API to be mounted at /, when in fact we mount it at /api. >> >> Jake can probably tell you exactly what's wrong based on his code, but in >> the meantime you might find it helpful to compare against how we set up the >> JS and python clients: >> >> >> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188 >> >> >> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115 >> >> >> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected]> wrote: >> >>> Apologies for multiple mails, the previous email was sent accidentally. >>> I didn't add a problem description, before hitting send. >>> >>> When I query aurora for all the jobs using getJobs, I find the aurora >>> error as given below and the response I get using my client. >>> >>> -- >>> κρισhναν >>> >>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]> >>> wrote: >>> >>>> Thanks Jake! >>>> >>>> That worked like a charm, & I was wondering why does install from >>>> source doesn't work! >>>> >>>> Aurora Log: >>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109 >>>> [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for >>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} >>>> >>>> Thrift client: >>>> ./mrfantastic_service.out >>>> Connecting to aurora.... >>>> <nil> EOF >>>> >>>> Thrift client sourcec: >>>> func thriftAuroraJobs() { >>>> var protocolFactory thrift.TProtocolFactory >>>> var transportFactory thrift.TTransportFactory >>>> var transport thrift.TTransport >>>> var client *api.ReadOnlySchedulerClient >>>> var err error >>>> >>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() >>>> //transportFactory = >>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) >>>> transportFactory = thrift.NewTTransportFactory() >>>> fmt.Println("Connecting to aurora....") >>>> transport, err = thrift.NewTSocket("54.210.234.190:8081") >>>> if err != nil { >>>> fmt.Println("Error opening socket:", err) >>>> os.Exit(1) >>>> //return err >>>> } >>>> if transport == nil { >>>> os.Exit(1) >>>> } >>>> transport = transportFactory.GetTransport(transport) >>>> if transport == nil { >>>> os.Exit(1) >>>> } >>>> err = transport.Open() >>>> if err != nil { >>>> os.Exit(1) >>>> } >>>> defer transport.Close() >>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>> protocolFactory) >>>> fmt.Println(client.GetJobs("")) >>>> } >>>> >>>> >>>> My hunch is that the thrift port isn't 8081, as the server/aurora is >>>> looking for HTTP data on the socket. >>>> Is there a config that needs to be set for thrift API to be initialized? >>>> >>>> >>>> >>>> >>>> -- >>>> κρισhναν >>>> >>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]> >>>> wrote: >>>> >>>>> if you just need the compiler you can install the thrift-compiler >>>>> package with one of the following, otherwise you can run ./bootstrap.sh && >>>>> ./configure && cd compiler/cpp && make to just build the compiler. >>>>> >>>>> -Jake >>>>> >>>>> >>>>> >>>>> deb packaging (tested with ubuntu trusty) >>>>> >>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >>>>> > gpg --export --armor 66B778F9 | sudo apt-key add - >>>>> > /etc/apt/sources.list.d/thrift.list >>>>> >>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >>>>> >>>>> >>>>> or for centos/rhel (tested with centos 7.2) >>>>> >>>>> > /etc/yum.repos.d/thrift.repo >>>>> >>>>> [thrift] >>>>> name=Apache Thrift rpm repo >>>>> baseurl=http://www.apache.org/dist/thrift/rpm/ >>>>> enabled=1 >>>>> gpgcheck=0 >>>>> >>>>> >>>>> >>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]> >>>>> wrote: >>>>> >>>>>> Jake/Chris, >>>>>> Thanks for the info. >>>>>> When I try to install thrift v0.9.3 from source, I get an error as >>>>>> follows while running `make check`: >>>>>> ... >>>>>> ... >>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time >>>>>> elapsed: 0.062 sec >>>>>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol >>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time >>>>>> elapsed: 0.046 sec >>>>>> >>>>>> BUILD FAILED >>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>>>>> >>>>>> Total time: 17 seconds >>>>>> make[3]: *** [check-local] Error 1 >>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>> make[2]: *** [check-am] Error 2 >>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>> make[1]: *** [check-recursive] Error 1 >>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>>>>> make: *** [check-recursive] Error 1 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I've used the latest thrift to generate go code, and then manually >>>>>>> created executor config which works and is able to launch jobs. >>>>>>> >>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Krish >>>>>>>> We are using Thrift with go for all our api calls to Aurora, would >>>>>>>> recommend you use Thrift 0.9.3 to interact with the api. >>>>>>>> >>>>>>>> happy to help answer any questions you might have >>>>>>>> >>>>>>>> -Jake >>>>>>>> >>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks, Bill. >>>>>>>>> >>>>>>>>> Well I have started my foray into the the thrift API today. And I >>>>>>>>> think I am stuck with some thrift configs. >>>>>>>>> >>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk >>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't find >>>>>>>>> any >>>>>>>>> changelog or compatibility statement on the thrift project site. >>>>>>>>> >>>>>>>>> >>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos >>>>>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source. >>>>>>>>> However, when I try to generate golang code, I think I hit a compiler >>>>>>>>> bug: >>>>>>>>> krish@krish:/tmp >>>>>>>>> > thrift --gen go api.thrift >>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal >>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>>>>> >>>>>>>>> I can modify the golang code by hand, but I would like to play it >>>>>>>>> safe and use the working compiler from the debian repos. >>>>>>>>> >>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a >>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone >>>>>>>>> faced a >>>>>>>>> similar error and gone past it? >>>>>>>>> I have already done a `go get >>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> κρισhναν >>>>>>>>> >>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Regarding documentation - Maxim is correct that there isn't much >>>>>>>>>> in the way of independent/holistic docs for the thrift API. There >>>>>>>>>> is, >>>>>>>>>> however, scant javadoc-style documentation within the IDL spec >>>>>>>>>> itself: >>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>>>>> >>>>>>>>>> If you are looking to use the thrift API directly, the most >>>>>>>>>> difficult API method will be defining the ExecutorConfig.data value >>>>>>>>>> when >>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if >>>>>>>>>> you get >>>>>>>>>> to that point! >>>>>>>>>> >>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and >>>>>>>>>>>> return thrift values only in Response.result field. >>>>>>>>>>> >>>>>>>>>>> Correct. There is also 'details' field that may have additional >>>>>>>>>>> messages (of error or informational nature) >>>>>>>>>>> >>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>> >>>>>>>>>>> The thrift API is largely undocumented. There is an effort to >>>>>>>>>>> bring up a fully supported REST API that will presumably get >>>>>>>>>>> documented and >>>>>>>>>>> become much easier to use. It's mostly in flux now. >>>>>>>>>>> >>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>> >>>>>>>>>>> This is the API to use when you a brand new service or adhoc >>>>>>>>>>> (batch) job created. The JobDescription is populated from the >>>>>>>>>>> .aurora >>>>>>>>>>> config. You may want to trace "aurora job create" client command >>>>>>>>>>> implementation to see how it happens. >>>>>>>>>>> >>>>>>>>>>> 4. What is the Lock object? I see that some APIs require locking >>>>>>>>>>>> and some don't. For example, createJob needs a Lock object as >>>>>>>>>>>> parameter, & >>>>>>>>>>>> I am assuming that it is required so that one does not create >>>>>>>>>>>> multiple jobs >>>>>>>>>>>> with the same JobKey. >>>>>>>>>>> >>>>>>>>>>> Ignore this object as it's an echo of the old client updater. >>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL for >>>>>>>>>>> now. >>>>>>>>>>> >>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>> >>>>>>>>>>> Another echo of the client updater but this time it's got a >>>>>>>>>>> second life. Check out its new signature and comments in the >>>>>>>>>>> api.thrift. >>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the >>>>>>>>>>> existing >>>>>>>>>>> job without changing the underlying task assumptions. >>>>>>>>>>> >>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>> >>>>>>>>>>> It's actually 'getPendingReason' and is currently used >>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state. >>>>>>>>>>> >>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>> >>>>>>>>>>> This is to set role-level quota. Currently only required for >>>>>>>>>>> tasks with 'production=True'. Search through our docs for more >>>>>>>>>>> details. >>>>>>>>>>> >>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>> cluster. >>>>>>>>>>> >>>>>>>>>>> It's quite versatile and can be used to kill some or all >>>>>>>>>>> instances of the job. >>>>>>>>>>> >>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>>>> >>>>>>>>>>> Your observations are correct. This is the main API to change >>>>>>>>>>> your service job in any way (including adding, removing or modifying >>>>>>>>>>> instances). >>>>>>>>>>> >>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>> them, does >>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>> >>>>>>>>>>> Scheduler will try to schedule as many instances as it can. >>>>>>>>>>> Those that will not find resources will remain in PENDING state >>>>>>>>>>> until more >>>>>>>>>>> resources are available. In your particular example only the 5th >>>>>>>>>>> will >>>>>>>>>>> remain PENDING. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to >>>>>>>>>>>> add new jobs. >>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded >>>>>>>>>>>> to mesos v0.25 accordingly. >>>>>>>>>>>> >>>>>>>>>>>> Below is a summary of my (very limited) understanding of some >>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my >>>>>>>>>>>> understanding: >>>>>>>>>>>> >>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, >>>>>>>>>>>> and return thrift values only in Response.result field. >>>>>>>>>>>> >>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>> >>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora >>>>>>>>>>>> files for job configuration. >>>>>>>>>>>> >>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>> object as >>>>>>>>>>>> parameter, & I am assuming that it is required so that one does >>>>>>>>>>>> not create >>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>> >>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>> By the naming convention, it seems this is used to increase >>>>>>>>>>>> the number of instances of a job. It will not result in >>>>>>>>>>>> stopping of current >>>>>>>>>>>> instances of the job. >>>>>>>>>>>> >>>>>>>>>>>> My second explanation for this API: Since it uses a set of >>>>>>>>>>>> instanceIds, this is used for adding already running job in >>>>>>>>>>>> slaves to the >>>>>>>>>>>> internal data structures of Aurora to track the job. >>>>>>>>>>>> >>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>> Return the reason (in string) about why the job is PENDING. >>>>>>>>>>>> For example: insufficient CPU. >>>>>>>>>>>> >>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>> >>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>> cluster. >>>>>>>>>>>> >>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified. >>>>>>>>>>>> Can be used if resource requirement changes. For example: If I >>>>>>>>>>>> wanted >>>>>>>>>>>> aurora to update the version of container used for a job using >>>>>>>>>>>> TaskConfig.Container attribute. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>> them, does >>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> κρισhναν >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> >
