Great to hear, onwards! On Wednesday, March 23, 2016, Krish <[email protected]> wrote:
> Thanks Bill/wfarner, Jake/jfarrell, Steve/t3hSteve, Joshua/jcohen, Maxim & > all the others (forgive me if I miss your name here), who took time out to > guide me over mail & on IRC over the past few days. > > The last hurdle, till now that is, was this: > The java source has a lot of parameters annotated with @Nullable, which > means we can set this param to null. Accordingly, I was setting them to > `nil` in the golang structs, which made the client break while accessing > nil fields. > I had to modify the generated code a bit (trivial changes), to make it > write only the fields that were present. > > There is still a long way to go to actually call this 'done', however, it > looks good so far; just implemented the addInstances & getPendingReason > APIs in my thrift client, and it works flawlessly. Getting a hang of how > things work. > > For all the folks who might be interested in playing with the thrift API, > please have a look @https://github.com/krish7919/aurora_thrift_api/. > It doesn't have great documentation, but some comments to go with source > code. > I am also going to put up a short writeup on aurora, and how to use the > APIs in the coming few days. > > > > -- > κρισhναν > > On Tue, Mar 22, 2016 at 3:53 PM, Krish <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> Thanks, Bill! Program counter is moving forward! :) >> >> Since, golang net/http does not support redirects, I hacked my code to >> query all aurora cluster members if the request is redirected. The proper >> fix, as Jake Farrell rightly pointed out on IRC, would be to make the go >> code handle redirects by using a properly configured http client. But I can >> delay that for now for some time. >> >> I got the proper getJobs list. I parse the resultant response as: >> //parse the response >> y := resp.Result_.GetJobsResult_.Configs >> fmt.Println("Configs: ", y) >> >> I am sure there is a better way to parse the response in golang, instead >> of getting a map. >> I killed all the mesos slaves & query for the pending reason for a job >> (which was already scheduled), and have a few doubts about it. >> >> Firstly, I thought passing an empty TaskQuery structure is supposed to >> give me all the pending reasons. >> >> Secondly, I see (at least in golang) the Owner field is mandatory. The >> client fails with a null pointer reference if it's not present. >> >> Thirdly, once I add the Owner field, aurora client receives an error from >> the server as below, even though I have not set the Statuses and SlaveHosts >> field: >> >> > ./aurora_client_service --api "http://54.209.127.224:8081/api"; >> ResponseCode: 'INVALID_REQUEST' >> ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})' >> Details: '[ResponseDetail({Message:Statuses or slaveHosts are not >> supported in TaskQuery(owner:Identity(role:, user:), role:, environment:, >> jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[], >> jobKeys:[], offset:0, limit:0)})]' >> Result: '<nil>' >> >> >> My task query struct is as follows: >> >> taskQuery := api.NewTaskQuery() >> //taskQuery.JobName = "my_job" >> //taskQuery.TaskIds = nil >> //taskQuery.Statuses = nil >> //taskQuery.InstanceIds = nil >> //taskQuery.SlaveHosts = nil >> //taskQuery.Environment = "" >> //taskQuery.JobKeys = nil >> //taskQuery.Offset = 0 >> //taskQuery.Limit = 0 >> //taskQuery.Role = "" >> taskQuery.Owner = &api.Identity { >> Role: "", >> User: "", >> } >> resp, err := client.GetPendingReason(taskQuery) >> >> >> >> >> -- >> κρισhναν >> >> On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> Figured it out. I was first tipped off that the request being made is a >>> GET, while i've normally seen HTTP thrift clients making POST requests. I >>> haven't yet drilled into the rationale behind the alternative client in the >>> thrift library, but this fixed up your code on my trials: >>> >>> - transport, err = thrift.NewTHttpClient(endpoint) >>> + transport, err = thrift.NewTHttpPostClient(endpoint) >>> >>> On Sun, Mar 20, 2016 at 10:58 PM, Krish <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> Thanks Bill. >>>> I checked the HTTP request. The sent packet data is: >>>> GET /api HTTP/1.1 >>>> Host: 54.209.127.223:8081 >>>> User-Agent: Go-http-client/1.1 >>>> Accept-Encoding: gzip >>>> >>>> I can attach the complete capture if that is needed. >>>> >>>> Link to my code on github: >>>> https://github.com/krish7919/aurora_thrift_api >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> κρισhναν >>>> >>>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>>>> Looks to me like it's failing before the request is even deserialized >>>>> - in the thrift layer (the exception is thrown here >>>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132> >>>>> - EOF). Can you capture the HTTP request that was sent by the client? >>>>> That might provide more clues. >>>>> >>>>> Also, if you're able to put up a git repo with your client code, i can >>>>> poke at it. >>>>> >>>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>> >>>>>> Hi Bill, >>>>>> Tried digging more about aurora thrift API this weekend. >>>>>> The thrift generated code is a good reference point. >>>>>> >>>>>> You are right; the '/' path just gives me the list of URLs that one >>>>>> gets on the browser when I query it using my thrift client. >>>>>> Any pointers based on the data below will be very helpful to find out >>>>>> why is aurora bailing out when processing the thrift request, or is it >>>>>> some >>>>>> client side error that is causing it. >>>>>> >>>>>> >>>>>> My thrift client trying to query for getJobs: >>>>>> >>>>>> .... >>>>>> var protocolFactory thrift.TProtocolFactory >>>>>> var transport thrift.TTransport >>>>>> var client *api.ReadOnlySchedulerClient >>>>>> var err error >>>>>> transport, err = thrift.NewTHttpClient(" >>>>>> http://54.209.17.221:8081/api";) >>>>>> defer transport.Close() >>>>>> protocolFactory = thrift.NewTJSONProtocolFactory() >>>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>>> protocolFactory) >>>>>> err = transport.Open() >>>>>> if err != nil { >>>>>> fmt.Println("Error opening socket: ", err) >>>>>> os.Exit(1) >>>>>> } >>>>>> defer transport.Close() >>>>>> fmt.Println(client.GetJobs("")) >>>>>> .... >>>>>> >>>>>> >>>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get >>>>>> a response packet from aurora, and my thrift client bails out with an >>>>>> error >>>>>> (runtime error: invalid memory address or nil pointer dereference). The >>>>>> data in the packet sent by server is: >>>>>> >>>>>> <html> >>>>>> <head> >>>>>> <meta http-equiv="Content-Type" >>>>>> content="text/html;charset=ISO-8859-1"/> >>>>>> <title>Error 500 </title> >>>>>> </head> >>>>>> <body> >>>>>> <h2>HTTP ERROR: 500</h2> >>>>>> <p>Problem accessing /api. Reason: >>>>>> <pre> javax.servlet.ServletException: >>>>>> org.apache.thrift.transport.TTransportException</pre></p> >>>>>> <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// >>>>>> 9.3.6.v20151106</a><hr/> >>>>>> </body> >>>>>> </html> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Stacktrace from the server/aurora console logs: >>>>>> >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235 >>>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException: >>>>>> org.apache.thrift.transport.TTransportException >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72) >>>>>> ~[aurora-0.12.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>>>>> ~[aurora-0.12.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71) >>>>>> ~[aurora-0.12.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>>>>> ~[aurora-0.12.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>>>>> ~[guice-servlet-3.0.jar:na] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) >>>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) >>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) >>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) >>>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) >>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>>> ... >>>>>> ... >>>>>> ... >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by: >>>>>> org.apache.thrift.transport.TTransportException: null >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83) >>>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames >>>>>> omitted >>>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235 >>>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - - >>>>>> [19/Mar/2016:18:12:23 + >>>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> κρισhναν >>>>>> >>>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected] >>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>> >>>>>>> 8081 is indeed the default thrift port. If you can capture a raw >>>>>>> HTTP request, we could diagnose this better. My first hunch is that the >>>>>>> thrift client expects the API to be mounted at /, when in fact we mount >>>>>>> it >>>>>>> at /api. >>>>>>> >>>>>>> Jake can probably tell you exactly what's wrong based on his code, >>>>>>> but in the meantime you might find it helpful to compare against how we >>>>>>> set >>>>>>> up the JS and python clients: >>>>>>> >>>>>>> >>>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188 >>>>>>> >>>>>>> >>>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115 >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected] >>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>> >>>>>>>> Apologies for multiple mails, the previous email was sent >>>>>>>> accidentally. >>>>>>>> I didn't add a problem description, before hitting send. >>>>>>>> >>>>>>>> When I query aurora for all the jobs using getJobs, I find the >>>>>>>> aurora error as given below and the response I get using my client. >>>>>>>> >>>>>>>> -- >>>>>>>> κρισhναν >>>>>>>> >>>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected] >>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>> >>>>>>>>> Thanks Jake! >>>>>>>>> >>>>>>>>> That worked like a charm, & I was wondering why does install from >>>>>>>>> source doesn't work! >>>>>>>>> >>>>>>>>> Aurora Log: >>>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 >>>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 >>>>>>>>> for >>>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} >>>>>>>>> >>>>>>>>> Thrift client: >>>>>>>>> ./mrfantastic_service.out >>>>>>>>> Connecting to aurora.... >>>>>>>>> <nil> EOF >>>>>>>>> >>>>>>>>> Thrift client sourcec: >>>>>>>>> func thriftAuroraJobs() { >>>>>>>>> var protocolFactory thrift.TProtocolFactory >>>>>>>>> var transportFactory thrift.TTransportFactory >>>>>>>>> var transport thrift.TTransport >>>>>>>>> var client *api.ReadOnlySchedulerClient >>>>>>>>> var err error >>>>>>>>> >>>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() >>>>>>>>> //transportFactory = >>>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) >>>>>>>>> transportFactory = thrift.NewTTransportFactory() >>>>>>>>> fmt.Println("Connecting to aurora....") >>>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081") >>>>>>>>> if err != nil { >>>>>>>>> fmt.Println("Error opening socket:", err) >>>>>>>>> os.Exit(1) >>>>>>>>> //return err >>>>>>>>> } >>>>>>>>> if transport == nil { >>>>>>>>> os.Exit(1) >>>>>>>>> } >>>>>>>>> transport = transportFactory.GetTransport(transport) >>>>>>>>> if transport == nil { >>>>>>>>> os.Exit(1) >>>>>>>>> } >>>>>>>>> err = transport.Open() >>>>>>>>> if err != nil { >>>>>>>>> os.Exit(1) >>>>>>>>> } >>>>>>>>> defer transport.Close() >>>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>>>>>> protocolFactory) >>>>>>>>> fmt.Println(client.GetJobs("")) >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora >>>>>>>>> is looking for HTTP data on the socket. >>>>>>>>> Is there a config that needs to be set for thrift API to be >>>>>>>>> initialized? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> κρισhναν >>>>>>>>> >>>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell < >>>>>>>>> [email protected] >>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>> >>>>>>>>>> if you just need the compiler you can install the thrift-compiler >>>>>>>>>> package with one of the following, otherwise you can run >>>>>>>>>> ./bootstrap.sh && >>>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler. >>>>>>>>>> >>>>>>>>>> -Jake >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> deb packaging (tested with ubuntu trusty) >>>>>>>>>> >>>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >>>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add - >>>>>>>>>> > /etc/apt/sources.list.d/thrift.list >>>>>>>>>> >>>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> or for centos/rhel (tested with centos 7.2) >>>>>>>>>> >>>>>>>>>> > /etc/yum.repos.d/thrift.repo >>>>>>>>>> >>>>>>>>>> [thrift] >>>>>>>>>> name=Apache Thrift rpm repo >>>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/ >>>>>>>>>> enabled=1 >>>>>>>>>> gpgcheck=0 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish < >>>>>>>>>> [email protected] >>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Jake/Chris, >>>>>>>>>>> Thanks for the info. >>>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error >>>>>>>>>>> as follows while running `make check`: >>>>>>>>>>> ... >>>>>>>>>>> ... >>>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, >>>>>>>>>>> Time elapsed: 0.062 sec >>>>>>>>>>> [junit] Running >>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol >>>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, >>>>>>>>>>> Time elapsed: 0.046 sec >>>>>>>>>>> >>>>>>>>>>> BUILD FAILED >>>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>>>>>>>>>> >>>>>>>>>>> Total time: 17 seconds >>>>>>>>>>> make[3]: *** [check-local] Error 1 >>>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>>>>> make[2]: *** [check-am] Error 2 >>>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>>>>> make[1]: *** [check-recursive] Error 1 >>>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>>>>>>>>>> make: *** [check-recursive] Error 1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> κρισhναν >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister < >>>>>>>>>>> [email protected] >>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I've used the latest thrift to generate go code, and then >>>>>>>>>>>> manually created executor config which works and is able to launch >>>>>>>>>>>> jobs. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, < >>>>>>>>>>>> [email protected] >>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Krish >>>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora, >>>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api. >>>>>>>>>>>>> >>>>>>>>>>>>> happy to help answer any questions you might have >>>>>>>>>>>>> >>>>>>>>>>>>> -Jake >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish < >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Bill. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Well I have started my foray into the the thrift API today. >>>>>>>>>>>>>> And I think I am stuck with some thrift configs. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to >>>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I >>>>>>>>>>>>>> couldn't find >>>>>>>>>>>>>> any changelog or compatibility statement on the thrift project >>>>>>>>>>>>>> site. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian >>>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 >>>>>>>>>>>>>> from source. >>>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a >>>>>>>>>>>>>> compiler bug: >>>>>>>>>>>>>> krish@krish:/tmp >>>>>>>>>>>>>> > thrift --gen go api.thrift >>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite >>>>>>>>>>>>>> literal >>>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can modify the golang code by hand, but I would like to >>>>>>>>>>>>>> play it safe and use the working compiler from the debian repos. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code >>>>>>>>>>>>>> into a test golang app, it fails to find "thriftlib/api" >>>>>>>>>>>>>> package. Anyone >>>>>>>>>>>>>> faced a similar error and gone past it? >>>>>>>>>>>>>> I have already done a `go get >>>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner < >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't >>>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift >>>>>>>>>>>>>>> API. There is, >>>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec >>>>>>>>>>>>>>> itself: >>>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most >>>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data >>>>>>>>>>>>>>> value when >>>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance >>>>>>>>>>>>>>> if you get >>>>>>>>>>>>>>> to that point! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko < >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, >>>>>>>>>>>>>>>>> and return thrift values only in Response.result field. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Correct. There is also 'details' field that may have >>>>>>>>>>>>>>>> additional messages (of error or informational nature) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort >>>>>>>>>>>>>>>> to bring up a fully supported REST API that will presumably >>>>>>>>>>>>>>>> get documented >>>>>>>>>>>>>>>> and become much easier to use. It's mostly in flux now. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is the API to use when you a brand new service or >>>>>>>>>>>>>>>> adhoc (batch) job created. The JobDescription is populated >>>>>>>>>>>>>>>> from the .aurora >>>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>> implementation to see how it happens. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>>>>>> object as >>>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one >>>>>>>>>>>>>>>>> does not create >>>>>>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client >>>>>>>>>>>>>>>> updater. It's now deprecated and will be removed soon. You can >>>>>>>>>>>>>>>> pass NULL >>>>>>>>>>>>>>>> for now. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a >>>>>>>>>>>>>>>> second life. Check out its new signature and comments in the >>>>>>>>>>>>>>>> api.thrift. >>>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to >>>>>>>>>>>>>>>> the existing >>>>>>>>>>>>>>>> job without changing the underlying task assumptions. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used >>>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING >>>>>>>>>>>>>>>> state. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is to set role-level quota. Currently only required >>>>>>>>>>>>>>>> for tasks with 'production=True'. Search through our docs for >>>>>>>>>>>>>>>> more details. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>>>>>>> cluster. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all >>>>>>>>>>>>>>>> instances of the job. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string >>>>>>>>>>>>>>>>> message): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Your observations are correct. This is the main API to >>>>>>>>>>>>>>>> change your service job in any way (including adding, removing >>>>>>>>>>>>>>>> or modifying >>>>>>>>>>>>>>>> instances). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>>>>> them, does >>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job >>>>>>>>>>>>>>>>> blocks? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can. >>>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING >>>>>>>>>>>>>>>> state until more >>>>>>>>>>>>>>>> resources are available. In your particular example only the >>>>>>>>>>>>>>>> 5th will >>>>>>>>>>>>>>>> remain PENDING. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish < >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');> >>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how >>>>>>>>>>>>>>>>> to add new jobs. >>>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have >>>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of >>>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in >>>>>>>>>>>>>>>>> my >>>>>>>>>>>>>>>>> understanding: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs >>>>>>>>>>>>>>>>> specified, and return thrift values only in >>>>>>>>>>>>>>>>> Response.result field. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to >>>>>>>>>>>>>>>>> help understand Thrift API better? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>>>>>> This is basically the API to replace the Aurora >>>>>>>>>>>>>>>>> DSL/.aurora files for job configuration. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs >>>>>>>>>>>>>>>>> require locking and some don't. For example, createJob >>>>>>>>>>>>>>>>> needs a Lock object >>>>>>>>>>>>>>>>> as parameter, & I am assuming that it is required so that >>>>>>>>>>>>>>>>> one does not >>>>>>>>>>>>>>>>> create multiple jobs with the same JobKey. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>>>>>> By the naming convention, it seems this is used to >>>>>>>>>>>>>>>>> increase the number of instances of a job. It will not >>>>>>>>>>>>>>>>> result in stopping >>>>>>>>>>>>>>>>> of current instances of the job. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a >>>>>>>>>>>>>>>>> set of instanceIds, this is used for adding already >>>>>>>>>>>>>>>>> running job in slaves >>>>>>>>>>>>>>>>> to the internal data structures of Aurora to track the job. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>>>>>> Return the reason (in string) about why the job is >>>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in >>>>>>>>>>>>>>>>> the cluster. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string >>>>>>>>>>>>>>>>> message): >>>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig >>>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. >>>>>>>>>>>>>>>>> For example: If I >>>>>>>>>>>>>>>>> wanted aurora to update the version of container used for >>>>>>>>>>>>>>>>> a job using >>>>>>>>>>>>>>>>> TaskConfig.Container attribute. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>>>>> them, does >>>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job >>>>>>>>>>>>>>>>> blocks? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
