Figured it out. I was first tipped off that the request being made is a GET, while i've normally seen HTTP thrift clients making POST requests. I haven't yet drilled into the rationale behind the alternative client in the thrift library, but this fixed up your code on my trials:
- transport, err = thrift.NewTHttpClient(endpoint) + transport, err = thrift.NewTHttpPostClient(endpoint) On Sun, Mar 20, 2016 at 10:58 PM, Krish <[email protected]> wrote: > Thanks Bill. > I checked the HTTP request. The sent packet data is: > GET /api HTTP/1.1 > Host: 54.209.127.223:8081 > User-Agent: Go-http-client/1.1 > Accept-Encoding: gzip > > I can attach the complete capture if that is needed. > > Link to my code on github: > https://github.com/krish7919/aurora_thrift_api > > > > > > -- > κρισhναν > > On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <[email protected]> wrote: > >> Looks to me like it's failing before the request is even deserialized - >> in the thrift layer (the exception is thrown here >> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132> >> - EOF). Can you capture the HTTP request that was sent by the client? >> That might provide more clues. >> >> Also, if you're able to put up a git repo with your client code, i can >> poke at it. >> >> On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected]> >> wrote: >> >>> Hi Bill, >>> Tried digging more about aurora thrift API this weekend. >>> The thrift generated code is a good reference point. >>> >>> You are right; the '/' path just gives me the list of URLs that one gets >>> on the browser when I query it using my thrift client. >>> Any pointers based on the data below will be very helpful to find out >>> why is aurora bailing out when processing the thrift request, or is it some >>> client side error that is causing it. >>> >>> >>> My thrift client trying to query for getJobs: >>> >>> .... >>> var protocolFactory thrift.TProtocolFactory >>> var transport thrift.TTransport >>> var client *api.ReadOnlySchedulerClient >>> var err error >>> transport, err = thrift.NewTHttpClient(" >>> http://54.209.17.221:8081/api";) >>> defer transport.Close() >>> protocolFactory = thrift.NewTJSONProtocolFactory() >>> client = api.NewReadOnlySchedulerClientFactory(transport, >>> protocolFactory) >>> err = transport.Open() >>> if err != nil { >>> fmt.Println("Error opening socket: ", err) >>> os.Exit(1) >>> } >>> defer transport.Close() >>> fmt.Println(client.GetJobs("")) >>> .... >>> >>> >>> I did a wireshark analysis of outgoing packets to aurora, & I do get a >>> response packet from aurora, and my thrift client bails out with an error >>> (runtime error: invalid memory address or nil pointer dereference). The >>> data in the packet sent by server is: >>> >>> <html> >>> <head> >>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> >>> <title>Error 500 </title> >>> </head> >>> <body> >>> <h2>HTTP ERROR: 500</h2> >>> <p>Problem accessing /api. Reason: >>> <pre> javax.servlet.ServletException: >>> org.apache.thrift.transport.TTransportException</pre></p> >>> <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// >>> 9.3.6.v20151106</a><hr/> >>> </body> >>> </html> >>> >>> >>> >>> >>> Stacktrace from the server/aurora console logs: >>> >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235 >>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException: >>> org.apache.thrift.transport.TTransportException >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72) >>> ~[aurora-0.12.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>> ~[aurora-0.12.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71) >>> ~[aurora-0.12.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>> ~[aurora-0.12.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>> ~[guice-servlet-3.0.jar:na] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) >>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) >>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) >>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) >>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) >>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>> ... >>> ... >>> ... >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by: >>> org.apache.thrift.transport.TTransportException: null >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83) >>> ~[libthrift-0.9.1.jar:0.9.1] >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted >>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235 >>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - - >>> [19/Mar/2016:18:12:23 + >>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389 >>> >>> >>> >>> >>> -- >>> κρισhναν >>> >>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected]> wrote: >>> >>>> 8081 is indeed the default thrift port. If you can capture a raw HTTP >>>> request, we could diagnose this better. My first hunch is that the thrift >>>> client expects the API to be mounted at /, when in fact we mount it at >>>> /api. >>>> >>>> Jake can probably tell you exactly what's wrong based on his code, but >>>> in the meantime you might find it helpful to compare against how we set up >>>> the JS and python clients: >>>> >>>> >>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188 >>>> >>>> >>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115 >>>> >>>> >>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected]> >>>> wrote: >>>> >>>>> Apologies for multiple mails, the previous email was sent accidentally. >>>>> I didn't add a problem description, before hitting send. >>>>> >>>>> When I query aurora for all the jobs using getJobs, I find the aurora >>>>> error as given below and the response I get using my client. >>>>> >>>>> -- >>>>> κρισhναν >>>>> >>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks Jake! >>>>>> >>>>>> That worked like a charm, & I was wondering why does install from >>>>>> source doesn't work! >>>>>> >>>>>> Aurora Log: >>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 >>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for >>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} >>>>>> >>>>>> Thrift client: >>>>>> ./mrfantastic_service.out >>>>>> Connecting to aurora.... >>>>>> <nil> EOF >>>>>> >>>>>> Thrift client sourcec: >>>>>> func thriftAuroraJobs() { >>>>>> var protocolFactory thrift.TProtocolFactory >>>>>> var transportFactory thrift.TTransportFactory >>>>>> var transport thrift.TTransport >>>>>> var client *api.ReadOnlySchedulerClient >>>>>> var err error >>>>>> >>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() >>>>>> //transportFactory = >>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) >>>>>> transportFactory = thrift.NewTTransportFactory() >>>>>> fmt.Println("Connecting to aurora....") >>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081") >>>>>> if err != nil { >>>>>> fmt.Println("Error opening socket:", err) >>>>>> os.Exit(1) >>>>>> //return err >>>>>> } >>>>>> if transport == nil { >>>>>> os.Exit(1) >>>>>> } >>>>>> transport = transportFactory.GetTransport(transport) >>>>>> if transport == nil { >>>>>> os.Exit(1) >>>>>> } >>>>>> err = transport.Open() >>>>>> if err != nil { >>>>>> os.Exit(1) >>>>>> } >>>>>> defer transport.Close() >>>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>>> protocolFactory) >>>>>> fmt.Println(client.GetJobs("")) >>>>>> } >>>>>> >>>>>> >>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is >>>>>> looking for HTTP data on the socket. >>>>>> Is there a config that needs to be set for thrift API to be >>>>>> initialized? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> if you just need the compiler you can install the thrift-compiler >>>>>>> package with one of the following, otherwise you can run ./bootstrap.sh >>>>>>> && >>>>>>> ./configure && cd compiler/cpp && make to just build the compiler. >>>>>>> >>>>>>> -Jake >>>>>>> >>>>>>> >>>>>>> >>>>>>> deb packaging (tested with ubuntu trusty) >>>>>>> >>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add - >>>>>>> > /etc/apt/sources.list.d/thrift.list >>>>>>> >>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >>>>>>> >>>>>>> >>>>>>> or for centos/rhel (tested with centos 7.2) >>>>>>> >>>>>>> > /etc/yum.repos.d/thrift.repo >>>>>>> >>>>>>> [thrift] >>>>>>> name=Apache Thrift rpm repo >>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/ >>>>>>> enabled=1 >>>>>>> gpgcheck=0 >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Jake/Chris, >>>>>>>> Thanks for the info. >>>>>>>> When I try to install thrift v0.9.3 from source, I get an error as >>>>>>>> follows while running `make check`: >>>>>>>> ... >>>>>>>> ... >>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time >>>>>>>> elapsed: 0.062 sec >>>>>>>> [junit] Running >>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol >>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time >>>>>>>> elapsed: 0.046 sec >>>>>>>> >>>>>>>> BUILD FAILED >>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>>>>>>> >>>>>>>> Total time: 17 seconds >>>>>>>> make[3]: *** [check-local] Error 1 >>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>> make[2]: *** [check-am] Error 2 >>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>> make[1]: *** [check-recursive] Error 1 >>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>>>>>>> make: *** [check-recursive] Error 1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> κρισhναν >>>>>>>> >>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I've used the latest thrift to generate go code, and then manually >>>>>>>>> created executor config which works and is able to launch jobs. >>>>>>>>> >>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Krish >>>>>>>>>> We are using Thrift with go for all our api calls to Aurora, >>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api. >>>>>>>>>> >>>>>>>>>> happy to help answer any questions you might have >>>>>>>>>> >>>>>>>>>> -Jake >>>>>>>>>> >>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Thanks, Bill. >>>>>>>>>>> >>>>>>>>>>> Well I have started my foray into the the thrift API today. And >>>>>>>>>>> I think I am stuck with some thrift configs. >>>>>>>>>>> >>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk >>>>>>>>>>> with aurora using thrift 0.9.1? Are they compatible? I couldn't >>>>>>>>>>> find any >>>>>>>>>>> changelog or compatibility statement on the thrift project site. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian >>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from >>>>>>>>>>> source. >>>>>>>>>>> However, when I try to generate golang code, I think I hit a >>>>>>>>>>> compiler bug: >>>>>>>>>>> krish@krish:/tmp >>>>>>>>>>> > thrift --gen go api.thrift >>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal >>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>>>>>>> >>>>>>>>>>> I can modify the golang code by hand, but I would like to play >>>>>>>>>>> it safe and use the working compiler from the debian repos. >>>>>>>>>>> >>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a >>>>>>>>>>> test golang app, it fails to find "thriftlib/api" package. Anyone >>>>>>>>>>> faced a >>>>>>>>>>> similar error and gone past it? >>>>>>>>>>> I have already done a `go get >>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> κρισhναν >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't >>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. >>>>>>>>>>>> There is, >>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec >>>>>>>>>>>> itself: >>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>>>>>>> >>>>>>>>>>>> If you are looking to use the thrift API directly, the most >>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data >>>>>>>>>>>> value when >>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance if >>>>>>>>>>>> you get >>>>>>>>>>>> to that point! >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, >>>>>>>>>>>>>> and return thrift values only in Response.result field. >>>>>>>>>>>>> >>>>>>>>>>>>> Correct. There is also 'details' field that may have >>>>>>>>>>>>> additional messages (of error or informational nature) >>>>>>>>>>>>> >>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>>> >>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to >>>>>>>>>>>>> bring up a fully supported REST API that will presumably get >>>>>>>>>>>>> documented and >>>>>>>>>>>>> become much easier to use. It's mostly in flux now. >>>>>>>>>>>>> >>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>> >>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc >>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the >>>>>>>>>>>>> .aurora >>>>>>>>>>>>> config. You may want to trace "aurora job create" client command >>>>>>>>>>>>> implementation to see how it happens. >>>>>>>>>>>>> >>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>>> object as >>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does >>>>>>>>>>>>>> not create >>>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>>> >>>>>>>>>>>>> Ignore this object as it's an echo of the old client updater. >>>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL >>>>>>>>>>>>> for now. >>>>>>>>>>>>> >>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>> >>>>>>>>>>>>> Another echo of the client updater but this time it's got a >>>>>>>>>>>>> second life. Check out its new signature and comments in the >>>>>>>>>>>>> api.thrift. >>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the >>>>>>>>>>>>> existing >>>>>>>>>>>>> job without changing the underlying task assumptions. >>>>>>>>>>>>> >>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>> >>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used >>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state. >>>>>>>>>>>>> >>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>> >>>>>>>>>>>>> This is to set role-level quota. Currently only required for >>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more >>>>>>>>>>>>> details. >>>>>>>>>>>>> >>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>>>> cluster. >>>>>>>>>>>>> >>>>>>>>>>>>> It's quite versatile and can be used to kill some or all >>>>>>>>>>>>> instances of the job. >>>>>>>>>>>>> >>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>>>>>> >>>>>>>>>>>>> Your observations are correct. This is the main API to change >>>>>>>>>>>>> your service job in any way (including adding, removing or >>>>>>>>>>>>> modifying >>>>>>>>>>>>> instances). >>>>>>>>>>>>> >>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>> them, does >>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>>>> >>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can. >>>>>>>>>>>>> Those that will not find resources will remain in PENDING state >>>>>>>>>>>>> until more >>>>>>>>>>>>> resources are available. In your particular example only the 5th >>>>>>>>>>>>> will >>>>>>>>>>>>> remain PENDING. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how to >>>>>>>>>>>>>> add new jobs. >>>>>>>>>>>>>> I am using aurora v0.12 released last month and have upgraded >>>>>>>>>>>>>> to mesos v0.25 accordingly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of some >>>>>>>>>>>>>> APIs, & would help it if someone can point out flaws in my >>>>>>>>>>>>>> understanding: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs >>>>>>>>>>>>>> specified, and return thrift values only in Response.result >>>>>>>>>>>>>> field. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to >>>>>>>>>>>>>> help understand Thrift API better? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>>> This is basically the API to replace the Aurora >>>>>>>>>>>>>> DSL/.aurora files for job configuration. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>>> object as >>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one >>>>>>>>>>>>>> does not create >>>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>>> By the naming convention, it seems this is used to >>>>>>>>>>>>>> increase the number of instances of a job. It will not result >>>>>>>>>>>>>> in stopping >>>>>>>>>>>>>> of current instances of the job. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My second explanation for this API: Since it uses a set >>>>>>>>>>>>>> of instanceIds, this is used for adding already running job >>>>>>>>>>>>>> in slaves to >>>>>>>>>>>>>> the internal data structures of Aurora to track the job. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>>> Return the reason (in string) about why the job is >>>>>>>>>>>>>> PENDING. For example: insufficient CPU. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in >>>>>>>>>>>>>> the cluster. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string >>>>>>>>>>>>>> message): >>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig specified. >>>>>>>>>>>>>> Can be used if resource requirement changes. For example: If >>>>>>>>>>>>>> I wanted >>>>>>>>>>>>>> aurora to update the version of container used for a job using >>>>>>>>>>>>>> TaskConfig.Container attribute. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>> them, does >>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job blocks? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
