Thanks Bill/wfarner, Jake/jfarrell, Steve/t3hSteve, Joshua/jcohen, Maxim & all the others (forgive me if I miss your name here), who took time out to guide me over mail & on IRC over the past few days.
The last hurdle, till now that is, was this: The java source has a lot of parameters annotated with @Nullable, which means we can set this param to null. Accordingly, I was setting them to `nil` in the golang structs, which made the client break while accessing nil fields. I had to modify the generated code a bit (trivial changes), to make it write only the fields that were present. There is still a long way to go to actually call this 'done', however, it looks good so far; just implemented the addInstances & getPendingReason APIs in my thrift client, and it works flawlessly. Getting a hang of how things work. For all the folks who might be interested in playing with the thrift API, please have a look @https://github.com/krish7919/aurora_thrift_api/. It doesn't have great documentation, but some comments to go with source code. I am also going to put up a short writeup on aurora, and how to use the APIs in the coming few days. -- κρισhναν On Tue, Mar 22, 2016 at 3:53 PM, Krish <[email protected]> wrote: > Thanks, Bill! Program counter is moving forward! :) > > Since, golang net/http does not support redirects, I hacked my code to > query all aurora cluster members if the request is redirected. The proper > fix, as Jake Farrell rightly pointed out on IRC, would be to make the go > code handle redirects by using a properly configured http client. But I can > delay that for now for some time. > > I got the proper getJobs list. I parse the resultant response as: > //parse the response > y := resp.Result_.GetJobsResult_.Configs > fmt.Println("Configs: ", y) > > I am sure there is a better way to parse the response in golang, instead > of getting a map. > I killed all the mesos slaves & query for the pending reason for a job > (which was already scheduled), and have a few doubts about it. > > Firstly, I thought passing an empty TaskQuery structure is supposed to > give me all the pending reasons. > > Secondly, I see (at least in golang) the Owner field is mandatory. The > client fails with a null pointer reference if it's not present. > > Thirdly, once I add the Owner field, aurora client receives an error from > the server as below, even though I have not set the Statuses and SlaveHosts > field: > > > ./aurora_client_service --api "http://54.209.127.224:8081/api"; > ResponseCode: 'INVALID_REQUEST' > ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})' > Details: '[ResponseDetail({Message:Statuses or slaveHosts are not > supported in TaskQuery(owner:Identity(role:, user:), role:, environment:, > jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[], > jobKeys:[], offset:0, limit:0)})]' > Result: '<nil>' > > > My task query struct is as follows: > > taskQuery := api.NewTaskQuery() > //taskQuery.JobName = "my_job" > //taskQuery.TaskIds = nil > //taskQuery.Statuses = nil > //taskQuery.InstanceIds = nil > //taskQuery.SlaveHosts = nil > //taskQuery.Environment = "" > //taskQuery.JobKeys = nil > //taskQuery.Offset = 0 > //taskQuery.Limit = 0 > //taskQuery.Role = "" > taskQuery.Owner = &api.Identity { > Role: "", > User: "", > } > resp, err := client.GetPendingReason(taskQuery) > > > > > -- > κρισhναν > > On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <[email protected]> wrote: > >> Figured it out. I was first tipped off that the request being made is a >> GET, while i've normally seen HTTP thrift clients making POST requests. I >> haven't yet drilled into the rationale behind the alternative client in the >> thrift library, but this fixed up your code on my trials: >> >> - transport, err = thrift.NewTHttpClient(endpoint) >> + transport, err = thrift.NewTHttpPostClient(endpoint) >> >> On Sun, Mar 20, 2016 at 10:58 PM, Krish <[email protected]> >> wrote: >> >>> Thanks Bill. >>> I checked the HTTP request. The sent packet data is: >>> GET /api HTTP/1.1 >>> Host: 54.209.127.223:8081 >>> User-Agent: Go-http-client/1.1 >>> Accept-Encoding: gzip >>> >>> I can attach the complete capture if that is needed. >>> >>> Link to my code on github: >>> https://github.com/krish7919/aurora_thrift_api >>> >>> >>> >>> >>> >>> -- >>> κρισhναν >>> >>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <[email protected]> wrote: >>> >>>> Looks to me like it's failing before the request is even deserialized - >>>> in the thrift layer (the exception is thrown here >>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132> >>>> - EOF). Can you capture the HTTP request that was sent by the client? >>>> That might provide more clues. >>>> >>>> Also, if you're able to put up a git repo with your client code, i can >>>> poke at it. >>>> >>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected]> >>>> wrote: >>>> >>>>> Hi Bill, >>>>> Tried digging more about aurora thrift API this weekend. >>>>> The thrift generated code is a good reference point. >>>>> >>>>> You are right; the '/' path just gives me the list of URLs that one >>>>> gets on the browser when I query it using my thrift client. >>>>> Any pointers based on the data below will be very helpful to find out >>>>> why is aurora bailing out when processing the thrift request, or is it >>>>> some >>>>> client side error that is causing it. >>>>> >>>>> >>>>> My thrift client trying to query for getJobs: >>>>> >>>>> .... >>>>> var protocolFactory thrift.TProtocolFactory >>>>> var transport thrift.TTransport >>>>> var client *api.ReadOnlySchedulerClient >>>>> var err error >>>>> transport, err = thrift.NewTHttpClient(" >>>>> http://54.209.17.221:8081/api";) >>>>> defer transport.Close() >>>>> protocolFactory = thrift.NewTJSONProtocolFactory() >>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>> protocolFactory) >>>>> err = transport.Open() >>>>> if err != nil { >>>>> fmt.Println("Error opening socket: ", err) >>>>> os.Exit(1) >>>>> } >>>>> defer transport.Close() >>>>> fmt.Println(client.GetJobs("")) >>>>> .... >>>>> >>>>> >>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get a >>>>> response packet from aurora, and my thrift client bails out with an error >>>>> (runtime error: invalid memory address or nil pointer dereference). The >>>>> data in the packet sent by server is: >>>>> >>>>> <html> >>>>> <head> >>>>> <meta http-equiv="Content-Type" >>>>> content="text/html;charset=ISO-8859-1"/> >>>>> <title>Error 500 </title> >>>>> </head> >>>>> <body> >>>>> <h2>HTTP ERROR: 500</h2> >>>>> <p>Problem accessing /api. Reason: >>>>> <pre> javax.servlet.ServletException: >>>>> org.apache.thrift.transport.TTransportException</pre></p> >>>>> <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// >>>>> 9.3.6.v20151106</a><hr/> >>>>> </body> >>>>> </html> >>>>> >>>>> >>>>> >>>>> >>>>> Stacktrace from the server/aurora console logs: >>>>> >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235 >>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException: >>>>> org.apache.thrift.transport.TTransportException >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72) >>>>> ~[aurora-0.12.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>>>> ~[aurora-0.12.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71) >>>>> ~[aurora-0.12.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >>>>> ~[aurora-0.12.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>>>> ~[guice-servlet-3.0.jar:na] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) >>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) >>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) >>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) >>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) >>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106] >>>>> ... >>>>> ... >>>>> ... >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by: >>>>> org.apache.thrift.transport.TTransportException: null >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at >>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83) >>>>> ~[libthrift-0.9.1.jar:0.9.1] >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames >>>>> omitted >>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235 >>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - - >>>>> [19/Mar/2016:18:12:23 + >>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389 >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> κρισhναν >>>>> >>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected]> >>>>> wrote: >>>>> >>>>>> 8081 is indeed the default thrift port. If you can capture a raw >>>>>> HTTP request, we could diagnose this better. My first hunch is that the >>>>>> thrift client expects the API to be mounted at /, when in fact we mount >>>>>> it >>>>>> at /api. >>>>>> >>>>>> Jake can probably tell you exactly what's wrong based on his code, >>>>>> but in the meantime you might find it helpful to compare against how we >>>>>> set >>>>>> up the JS and python clients: >>>>>> >>>>>> >>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188 >>>>>> >>>>>> >>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115 >>>>>> >>>>>> >>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Apologies for multiple mails, the previous email was sent >>>>>>> accidentally. >>>>>>> I didn't add a problem description, before hitting send. >>>>>>> >>>>>>> When I query aurora for all the jobs using getJobs, I find the >>>>>>> aurora error as given below and the response I get using my client. >>>>>>> >>>>>>> -- >>>>>>> κρισhναν >>>>>>> >>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Jake! >>>>>>>> >>>>>>>> That worked like a charm, & I was wondering why does install from >>>>>>>> source doesn't work! >>>>>>>> >>>>>>>> Aurora Log: >>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 >>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 >>>>>>>> for >>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null} >>>>>>>> >>>>>>>> Thrift client: >>>>>>>> ./mrfantastic_service.out >>>>>>>> Connecting to aurora.... >>>>>>>> <nil> EOF >>>>>>>> >>>>>>>> Thrift client sourcec: >>>>>>>> func thriftAuroraJobs() { >>>>>>>> var protocolFactory thrift.TProtocolFactory >>>>>>>> var transportFactory thrift.TTransportFactory >>>>>>>> var transport thrift.TTransport >>>>>>>> var client *api.ReadOnlySchedulerClient >>>>>>>> var err error >>>>>>>> >>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault() >>>>>>>> //transportFactory = >>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory()) >>>>>>>> transportFactory = thrift.NewTTransportFactory() >>>>>>>> fmt.Println("Connecting to aurora....") >>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081") >>>>>>>> if err != nil { >>>>>>>> fmt.Println("Error opening socket:", err) >>>>>>>> os.Exit(1) >>>>>>>> //return err >>>>>>>> } >>>>>>>> if transport == nil { >>>>>>>> os.Exit(1) >>>>>>>> } >>>>>>>> transport = transportFactory.GetTransport(transport) >>>>>>>> if transport == nil { >>>>>>>> os.Exit(1) >>>>>>>> } >>>>>>>> err = transport.Open() >>>>>>>> if err != nil { >>>>>>>> os.Exit(1) >>>>>>>> } >>>>>>>> defer transport.Close() >>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport, >>>>>>>> protocolFactory) >>>>>>>> fmt.Println(client.GetJobs("")) >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora >>>>>>>> is looking for HTTP data on the socket. >>>>>>>> Is there a config that needs to be set for thrift API to be >>>>>>>> initialized? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> κρισhναν >>>>>>>> >>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> if you just need the compiler you can install the thrift-compiler >>>>>>>>> package with one of the following, otherwise you can run >>>>>>>>> ./bootstrap.sh && >>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler. >>>>>>>>> >>>>>>>>> -Jake >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> deb packaging (tested with ubuntu trusty) >>>>>>>>> >>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import - >>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add - >>>>>>>>> > /etc/apt/sources.list.d/thrift.list >>>>>>>>> >>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main >>>>>>>>> >>>>>>>>> >>>>>>>>> or for centos/rhel (tested with centos 7.2) >>>>>>>>> >>>>>>>>> > /etc/yum.repos.d/thrift.repo >>>>>>>>> >>>>>>>>> [thrift] >>>>>>>>> name=Apache Thrift rpm repo >>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/ >>>>>>>>> enabled=1 >>>>>>>>> gpgcheck=0 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Jake/Chris, >>>>>>>>>> Thanks for the info. >>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error >>>>>>>>>> as follows while running `make check`: >>>>>>>>>> ... >>>>>>>>>> ... >>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil >>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, >>>>>>>>>> Time elapsed: 0.062 sec >>>>>>>>>> [junit] Running >>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol >>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, >>>>>>>>>> Time elapsed: 0.046 sec >>>>>>>>>> >>>>>>>>>> BUILD FAILED >>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test >>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed >>>>>>>>>> >>>>>>>>>> Total time: 17 seconds >>>>>>>>>> make[3]: *** [check-local] Error 1 >>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>>>> make[2]: *** [check-am] Error 2 >>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java' >>>>>>>>>> make[1]: *** [check-recursive] Error 1 >>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib' >>>>>>>>>> make: *** [check-recursive] Error 1 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> κρισhναν >>>>>>>>>> >>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I've used the latest thrift to generate go code, and then >>>>>>>>>>> manually created executor config which works and is able to launch >>>>>>>>>>> jobs. >>>>>>>>>>> >>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Krish >>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora, >>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api. >>>>>>>>>>>> >>>>>>>>>>>> happy to help answer any questions you might have >>>>>>>>>>>> >>>>>>>>>>>> -Jake >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, Bill. >>>>>>>>>>>>> >>>>>>>>>>>>> Well I have started my foray into the the thrift API today. >>>>>>>>>>>>> And I think I am stuck with some thrift configs. >>>>>>>>>>>>> >>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to >>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I >>>>>>>>>>>>> couldn't find >>>>>>>>>>>>> any changelog or compatibility statement on the thrift project >>>>>>>>>>>>> site. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian >>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 >>>>>>>>>>>>> from source. >>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a >>>>>>>>>>>>> compiler bug: >>>>>>>>>>>>> krish@krish:/tmp >>>>>>>>>>>>> > thrift --gen go api.thrift >>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite >>>>>>>>>>>>> literal >>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '=' >>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed. >>>>>>>>>>>>> >>>>>>>>>>>>> I can modify the golang code by hand, but I would like to play >>>>>>>>>>>>> it safe and use the working compiler from the debian repos. >>>>>>>>>>>>> >>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into >>>>>>>>>>>>> a test golang app, it fails to find "thriftlib/api" package. >>>>>>>>>>>>> Anyone faced a >>>>>>>>>>>>> similar error and gone past it? >>>>>>>>>>>>> I have already done a `go get >>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...` >>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't >>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API. >>>>>>>>>>>>>> There is, >>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec >>>>>>>>>>>>>> itself: >>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift >>>>>>>>>>>>>> >>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most >>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data >>>>>>>>>>>>>> value when >>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance >>>>>>>>>>>>>> if you get >>>>>>>>>>>>>> to that point! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified, >>>>>>>>>>>>>>>> and return thrift values only in Response.result field. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Correct. There is also 'details' field that may have >>>>>>>>>>>>>>> additional messages (of error or informational nature) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help >>>>>>>>>>>>>>>> understand Thrift API better? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort >>>>>>>>>>>>>>> to bring up a fully supported REST API that will presumably get >>>>>>>>>>>>>>> documented >>>>>>>>>>>>>>> and become much easier to use. It's mostly in flux now. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc >>>>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the >>>>>>>>>>>>>>> .aurora >>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command >>>>>>>>>>>>>>> implementation to see how it happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require >>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock >>>>>>>>>>>>>>>> object as >>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one >>>>>>>>>>>>>>>> does not create >>>>>>>>>>>>>>>> multiple jobs with the same JobKey. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client >>>>>>>>>>>>>>> updater. It's now deprecated and will be removed soon. You can >>>>>>>>>>>>>>> pass NULL >>>>>>>>>>>>>>> for now. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a >>>>>>>>>>>>>>> second life. Check out its new signature and comments in the >>>>>>>>>>>>>>> api.thrift. >>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to >>>>>>>>>>>>>>> the existing >>>>>>>>>>>>>>> job without changing the underlying task assumptions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used >>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING >>>>>>>>>>>>>>> state. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is to set role-level quota. Currently only required for >>>>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more >>>>>>>>>>>>>>> details. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the >>>>>>>>>>>>>>>> cluster. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all >>>>>>>>>>>>>>> instances of the job. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Your observations are correct. This is the main API to >>>>>>>>>>>>>>> change your service job in any way (including adding, removing >>>>>>>>>>>>>>> or modifying >>>>>>>>>>>>>>> instances). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>>>> them, does >>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job >>>>>>>>>>>>>>>> blocks? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can. >>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state >>>>>>>>>>>>>>> until more >>>>>>>>>>>>>>> resources are available. In your particular example only the >>>>>>>>>>>>>>> 5th will >>>>>>>>>>>>>>> remain PENDING. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how >>>>>>>>>>>>>>>> to add new jobs. >>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have >>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of >>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in my >>>>>>>>>>>>>>>> understanding: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs >>>>>>>>>>>>>>>> specified, and return thrift values only in Response.result >>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to >>>>>>>>>>>>>>>> help understand Thrift API better? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock): >>>>>>>>>>>>>>>> This is basically the API to replace the Aurora >>>>>>>>>>>>>>>> DSL/.aurora files for job configuration. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs >>>>>>>>>>>>>>>> require locking and some don't. For example, createJob >>>>>>>>>>>>>>>> needs a Lock object >>>>>>>>>>>>>>>> as parameter, & I am assuming that it is required so that >>>>>>>>>>>>>>>> one does not >>>>>>>>>>>>>>>> create multiple jobs with the same JobKey. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock): >>>>>>>>>>>>>>>> By the naming convention, it seems this is used to >>>>>>>>>>>>>>>> increase the number of instances of a job. It will not >>>>>>>>>>>>>>>> result in stopping >>>>>>>>>>>>>>>> of current instances of the job. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a set >>>>>>>>>>>>>>>> of instanceIds, this is used for adding already running job >>>>>>>>>>>>>>>> in slaves to >>>>>>>>>>>>>>>> the internal data structures of Aurora to track the job. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery): >>>>>>>>>>>>>>>> Return the reason (in string) about why the job is >>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in >>>>>>>>>>>>>>>> the cluster. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string >>>>>>>>>>>>>>>> message): >>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig >>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. For >>>>>>>>>>>>>>>> example: If I >>>>>>>>>>>>>>>> wanted aurora to update the version of container used for a >>>>>>>>>>>>>>>> job using >>>>>>>>>>>>>>>> TaskConfig.Container attribute. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5 >>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of >>>>>>>>>>>>>>>> them, does >>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job >>>>>>>>>>>>>>>> blocks? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> κρισhναν >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
