Thanks, Bill! Program counter is moving forward! :)
Since, golang net/http does not support redirects, I hacked my code to
query all aurora cluster members if the request is redirected. The proper
fix, as Jake Farrell rightly pointed out on IRC, would be to make the go
code handle redirects by using a properly configured http client. But I can
delay that for now for some time.
I got the proper getJobs list. I parse the resultant response as:
//parse the response
y := resp.Result_.GetJobsResult_.Configs
fmt.Println("Configs: ", y)
I am sure there is a better way to parse the response in golang, instead of
getting a map.
I killed all the mesos slaves & query for the pending reason for a job
(which was already scheduled), and have a few doubts about it.
Firstly, I thought passing an empty TaskQuery structure is supposed to give
me all the pending reasons.
Secondly, I see (at least in golang) the Owner field is mandatory. The
client fails with a null pointer reference if it's not present.
Thirdly, once I add the Owner field, aurora client receives an error from
the server as below, even though I have not set the Statuses and SlaveHosts
field:
> ./aurora_client_service --api "http://54.209.127.224:8081/api";
ResponseCode: 'INVALID_REQUEST'
ServerInfo: 'ServerInfo({ClusterName:aurora-cluster StatsUrlPrefix:})'
Details: '[ResponseDetail({Message:Statuses or slaveHosts are not supported
in TaskQuery(owner:Identity(role:, user:), role:, environment:,
jobName:my_job, taskIds:[], statuses:[], instanceIds:[], slaveHosts:[],
jobKeys:[], offset:0, limit:0)})]'
Result: '<nil>'
My task query struct is as follows:
taskQuery := api.NewTaskQuery()
//taskQuery.JobName = "my_job"
//taskQuery.TaskIds = nil
//taskQuery.Statuses = nil
//taskQuery.InstanceIds = nil
//taskQuery.SlaveHosts = nil
//taskQuery.Environment = ""
//taskQuery.JobKeys = nil
//taskQuery.Offset = 0
//taskQuery.Limit = 0
//taskQuery.Role = ""
taskQuery.Owner = &api.Identity {
Role: "",
User: "",
}
resp, err := client.GetPendingReason(taskQuery)
--
κρισhναν
On Tue, Mar 22, 2016 at 6:26 AM, Bill Farner <[email protected]> wrote:
> Figured it out. I was first tipped off that the request being made is a
> GET, while i've normally seen HTTP thrift clients making POST requests. I
> haven't yet drilled into the rationale behind the alternative client in the
> thrift library, but this fixed up your code on my trials:
>
> - transport, err = thrift.NewTHttpClient(endpoint)
> + transport, err = thrift.NewTHttpPostClient(endpoint)
>
> On Sun, Mar 20, 2016 at 10:58 PM, Krish <[email protected]> wrote:
>
>> Thanks Bill.
>> I checked the HTTP request. The sent packet data is:
>> GET /api HTTP/1.1
>> Host: 54.209.127.223:8081
>> User-Agent: Go-http-client/1.1
>> Accept-Encoding: gzip
>>
>> I can attach the complete capture if that is needed.
>>
>> Link to my code on github:
>> https://github.com/krish7919/aurora_thrift_api
>>
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Sun, Mar 20, 2016 at 2:29 AM, Bill Farner <[email protected]> wrote:
>>
>>> Looks to me like it's failing before the request is even deserialized -
>>> in the thrift layer (the exception is thrown here
>>> <https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L132>
>>> - EOF). Can you capture the HTTP request that was sent by the client?
>>> That might provide more clues.
>>>
>>> Also, if you're able to put up a git repo with your client code, i can
>>> poke at it.
>>>
>>> On Sat, Mar 19, 2016 at 11:26 AM, Krish <[email protected]>
>>> wrote:
>>>
>>>> Hi Bill,
>>>> Tried digging more about aurora thrift API this weekend.
>>>> The thrift generated code is a good reference point.
>>>>
>>>> You are right; the '/' path just gives me the list of URLs that one
>>>> gets on the browser when I query it using my thrift client.
>>>> Any pointers based on the data below will be very helpful to find out
>>>> why is aurora bailing out when processing the thrift request, or is it some
>>>> client side error that is causing it.
>>>>
>>>>
>>>> My thrift client trying to query for getJobs:
>>>>
>>>> ....
>>>> var protocolFactory thrift.TProtocolFactory
>>>> var transport thrift.TTransport
>>>> var client *api.ReadOnlySchedulerClient
>>>> var err error
>>>> transport, err = thrift.NewTHttpClient("
>>>> http://54.209.17.221:8081/api";)
>>>> defer transport.Close()
>>>> protocolFactory = thrift.NewTJSONProtocolFactory()
>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>> protocolFactory)
>>>> err = transport.Open()
>>>> if err != nil {
>>>> fmt.Println("Error opening socket: ", err)
>>>> os.Exit(1)
>>>> }
>>>> defer transport.Close()
>>>> fmt.Println(client.GetJobs(""))
>>>> ....
>>>>
>>>>
>>>> I did a wireshark analysis of outgoing packets to aurora, & I do get a
>>>> response packet from aurora, and my thrift client bails out with an error
>>>> (runtime error: invalid memory address or nil pointer dereference). The
>>>> data in the packet sent by server is:
>>>>
>>>> <html>
>>>> <head>
>>>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>>>> <title>Error 500 </title>
>>>> </head>
>>>> <body>
>>>> <h2>HTTP ERROR: 500</h2>
>>>> <p>Problem accessing /api. Reason:
>>>> <pre> javax.servlet.ServletException:
>>>> org.apache.thrift.transport.TTransportException</pre></p>
>>>> <hr /><a href="http://eclipse.org/jetty";>Powered by Jetty://
>>>> 9.3.6.v20151106</a><hr/>
>>>> </body>
>>>> </html>
>>>>
>>>>
>>>>
>>>>
>>>> Stacktrace from the server/aurora console logs:
>>>>
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: W0319 18:12:23.235
>>>> [qtp1289158047-127, ServletHandler:623] javax.servlet.ServletException:
>>>> org.apache.thrift.transport.TTransportException
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:86)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doGet(TServlet.java:96)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>> ~[javax.servlet-api-3.1.0.jar:3.1.0]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.LeaderRedirectFilter.doFilter(LeaderRedirectFilter.java:72)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:71)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>>>> ~[aurora-0.12.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>> ~[guice-servlet-3.0.jar:na]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>>> ~[jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>>> [jetty-servlet-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [jetty-server-9.3.6.v20151106.jar:9.3.6.v20151106]
>>>> ...
>>>> ...
>>>> ...
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: Caused by:
>>>> org.apache.thrift.transport.TTransportException: null
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: at
>>>> org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>>>> ~[libthrift-0.9.1.jar:0.9.1]
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: ... 51 common frames omitted
>>>> Mar 19 18:12:23 aurora-3 start.bash[21316]: I0319 18:12:23.235
>>>> [qtp1289158047-127, Slf4jRequestLog:60] 10.20.3.241 - -
>>>> [19/Mar/2016:18:12:23 +
>>>> 0000] "GET //54.209.17.221:8081/api HTTP/1.1" 500 389
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> κρισhναν
>>>>
>>>> On Fri, Mar 18, 2016 at 9:58 PM, Bill Farner <[email protected]>
>>>> wrote:
>>>>
>>>>> 8081 is indeed the default thrift port. If you can capture a raw HTTP
>>>>> request, we could diagnose this better. My first hunch is that the thrift
>>>>> client expects the API to be mounted at /, when in fact we mount it at
>>>>> /api.
>>>>>
>>>>> Jake can probably tell you exactly what's wrong based on his code, but
>>>>> in the meantime you might find it helpful to compare against how we set up
>>>>> the JS and python clients:
>>>>>
>>>>>
>>>>> https://github.com/apache/aurora/blob/master/src/main/resources/scheduler/assets/js/services.js#L185-L188
>>>>>
>>>>>
>>>>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/api/scheduler_client.py#L106-L115
>>>>>
>>>>>
>>>>> On Fri, Mar 18, 2016 at 7:50 AM, Krish <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Apologies for multiple mails, the previous email was sent
>>>>>> accidentally.
>>>>>> I didn't add a problem description, before hitting send.
>>>>>>
>>>>>> When I query aurora for all the jobs using getJobs, I find the aurora
>>>>>> error as given below and the response I get using my client.
>>>>>>
>>>>>> --
>>>>>> κρισhναν
>>>>>>
>>>>>> On Fri, Mar 18, 2016 at 8:15 PM, Krish <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Jake!
>>>>>>>
>>>>>>> That worked like a charm, & I was wondering why does install from
>>>>>>> source doesn't work!
>>>>>>>
>>>>>>> Aurora Log:
>>>>>>> Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318
>>>>>>> 14:34:49.109 [qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400
>>>>>>> for
>>>>>>> HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
>>>>>>>
>>>>>>> Thrift client:
>>>>>>> ./mrfantastic_service.out
>>>>>>> Connecting to aurora....
>>>>>>> <nil> EOF
>>>>>>>
>>>>>>> Thrift client sourcec:
>>>>>>> func thriftAuroraJobs() {
>>>>>>> var protocolFactory thrift.TProtocolFactory
>>>>>>> var transportFactory thrift.TTransportFactory
>>>>>>> var transport thrift.TTransport
>>>>>>> var client *api.ReadOnlySchedulerClient
>>>>>>> var err error
>>>>>>>
>>>>>>> protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
>>>>>>> //transportFactory =
>>>>>>> thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
>>>>>>> transportFactory = thrift.NewTTransportFactory()
>>>>>>> fmt.Println("Connecting to aurora....")
>>>>>>> transport, err = thrift.NewTSocket("54.210.234.190:8081")
>>>>>>> if err != nil {
>>>>>>> fmt.Println("Error opening socket:", err)
>>>>>>> os.Exit(1)
>>>>>>> //return err
>>>>>>> }
>>>>>>> if transport == nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> transport = transportFactory.GetTransport(transport)
>>>>>>> if transport == nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> err = transport.Open()
>>>>>>> if err != nil {
>>>>>>> os.Exit(1)
>>>>>>> }
>>>>>>> defer transport.Close()
>>>>>>> client = api.NewReadOnlySchedulerClientFactory(transport,
>>>>>>> protocolFactory)
>>>>>>> fmt.Println(client.GetJobs(""))
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> My hunch is that the thrift port isn't 8081, as the server/aurora is
>>>>>>> looking for HTTP data on the socket.
>>>>>>> Is there a config that needs to be set for thrift API to be
>>>>>>> initialized?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> κρισhναν
>>>>>>>
>>>>>>> On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> if you just need the compiler you can install the thrift-compiler
>>>>>>>> package with one of the following, otherwise you can run
>>>>>>>> ./bootstrap.sh &&
>>>>>>>> ./configure && cd compiler/cpp && make to just build the compiler.
>>>>>>>>
>>>>>>>> -Jake
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> deb packaging (tested with ubuntu trusty)
>>>>>>>>
>>>>>>>> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
>>>>>>>> > gpg --export --armor 66B778F9 | sudo apt-key add -
>>>>>>>> > /etc/apt/sources.list.d/thrift.list
>>>>>>>>
>>>>>>>> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>>>>>>>>
>>>>>>>>
>>>>>>>> or for centos/rhel (tested with centos 7.2)
>>>>>>>>
>>>>>>>> > /etc/yum.repos.d/thrift.repo
>>>>>>>>
>>>>>>>> [thrift]
>>>>>>>> name=Apache Thrift rpm repo
>>>>>>>> baseurl=http://www.apache.org/dist/thrift/rpm/
>>>>>>>> enabled=1
>>>>>>>> gpgcheck=0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Jake/Chris,
>>>>>>>>> Thanks for the info.
>>>>>>>>> When I try to install thrift v0.9.3 from source, I get an error as
>>>>>>>>> follows while running `make check`:
>>>>>>>>> ...
>>>>>>>>> ...
>>>>>>>>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>>>>>>>>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>>>>> elapsed: 0.062 sec
>>>>>>>>> [junit] Running
>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol
>>>>>>>>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>>>>>>>>> elapsed: 0.046 sec
>>>>>>>>>
>>>>>>>>> BUILD FAILED
>>>>>>>>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>>>>>>>>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>>>>>>>>
>>>>>>>>> Total time: 17 seconds
>>>>>>>>> make[3]: *** [check-local] Error 1
>>>>>>>>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>> make[2]: *** [check-am] Error 2
>>>>>>>>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>>>>>>>>> make[1]: *** [check-recursive] Error 1
>>>>>>>>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>>>>>>>>> make: *** [check-recursive] Error 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> κρισhναν
>>>>>>>>>
>>>>>>>>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> I've used the latest thrift to generate go code, and then
>>>>>>>>>> manually created executor config which works and is able to launch
>>>>>>>>>> jobs.
>>>>>>>>>>
>>>>>>>>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Krish
>>>>>>>>>>> We are using Thrift with go for all our api calls to Aurora,
>>>>>>>>>>> would recommend you use Thrift 0.9.3 to interact with the api.
>>>>>>>>>>>
>>>>>>>>>>> happy to help answer any questions you might have
>>>>>>>>>>>
>>>>>>>>>>> -Jake
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Bill.
>>>>>>>>>>>>
>>>>>>>>>>>> Well I have started my foray into the the thrift API today. And
>>>>>>>>>>>> I think I am stuck with some thrift configs.
>>>>>>>>>>>>
>>>>>>>>>>>> Does it matter if I use thrift v0.9.0 on the client side to
>>>>>>>>>>>> talk with aurora using thrift 0.9.1? Are they compatible? I
>>>>>>>>>>>> couldn't find
>>>>>>>>>>>> any changelog or compatibility statement on the thrift project
>>>>>>>>>>>> site.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian
>>>>>>>>>>>> repos have 0.9.0, I had to compile the thrift compiler v0.9.1 from
>>>>>>>>>>>> source.
>>>>>>>>>>>> However, when I try to generate golang code, I think I hit a
>>>>>>>>>>>> compiler bug:
>>>>>>>>>>>> krish@krish:/tmp
>>>>>>>>>>>> > thrift --gen go api.thrift
>>>>>>>>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>>>>>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>>>>>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>>>>>>>>
>>>>>>>>>>>> I can modify the golang code by hand, but I would like to play
>>>>>>>>>>>> it safe and use the working compiler from the debian repos.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, when I use thrift v0.9.0, and try to integrate code into
>>>>>>>>>>>> a test golang app, it fails to find "thriftlib/api" package.
>>>>>>>>>>>> Anyone faced a
>>>>>>>>>>>> similar error and gone past it?
>>>>>>>>>>>> I have already done a `go get
>>>>>>>>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>>>>>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding documentation - Maxim is correct that there isn't
>>>>>>>>>>>>> much in the way of independent/holistic docs for the thrift API.
>>>>>>>>>>>>> There is,
>>>>>>>>>>>>> however, scant javadoc-style documentation within the IDL spec
>>>>>>>>>>>>> itself:
>>>>>>>>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you are looking to use the thrift API directly, the most
>>>>>>>>>>>>> difficult API method will be defining the ExecutorConfig.data
>>>>>>>>>>>>> value when
>>>>>>>>>>>>> calling createJob. Please don't hesitate to ask for assistance
>>>>>>>>>>>>> if you get
>>>>>>>>>>>>> to that point!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs specified,
>>>>>>>>>>>>>>> and return thrift values only in Response.result field.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Correct. There is also 'details' field that may have
>>>>>>>>>>>>>> additional messages (of error or informational nature)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>>>>>>>>> understand Thrift API better?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The thrift API is largely undocumented. There is an effort to
>>>>>>>>>>>>>> bring up a fully supported REST API that will presumably get
>>>>>>>>>>>>>> documented and
>>>>>>>>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is the API to use when you a brand new service or adhoc
>>>>>>>>>>>>>> (batch) job created. The JobDescription is populated from the
>>>>>>>>>>>>>> .aurora
>>>>>>>>>>>>>> config. You may want to trace "aurora job create" client command
>>>>>>>>>>>>>> implementation to see how it happens.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock
>>>>>>>>>>>>>>> object as
>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one does
>>>>>>>>>>>>>>> not create
>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ignore this object as it's an echo of the old client updater.
>>>>>>>>>>>>>> It's now deprecated and will be removed soon. You can pass NULL
>>>>>>>>>>>>>> for now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another echo of the client updater but this time it's got a
>>>>>>>>>>>>>> second life. Check out its new signature and comments in the
>>>>>>>>>>>>>> api.thrift.
>>>>>>>>>>>>>> It's essentially a "scale-out" API that can add instances to the
>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>> job without changing the underlying task assumptions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's actually 'getPendingReason' and is currently used
>>>>>>>>>>>>>> exclusively by the UI to get the reason for a task PENDING state.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is to set role-level quota. Currently only required for
>>>>>>>>>>>>>> tasks with 'production=True'. Search through our docs for more
>>>>>>>>>>>>>> details.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>>>>>>>>> cluster.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's quite versatile and can be used to kill some or all
>>>>>>>>>>>>>> instances of the job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Your observations are correct. This is the main API to change
>>>>>>>>>>>>>> your service job in any way (including adding, removing or
>>>>>>>>>>>>>> modifying
>>>>>>>>>>>>>> instances).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of
>>>>>>>>>>>>>>> them, does
>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job
>>>>>>>>>>>>>>> blocks?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scheduler will try to schedule as many instances as it can.
>>>>>>>>>>>>>> Those that will not find resources will remain in PENDING state
>>>>>>>>>>>>>> until more
>>>>>>>>>>>>>> resources are available. In your particular example only the 5th
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> remain PENDING.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <
>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> I was going through the Aurora Thrift API to determine how
>>>>>>>>>>>>>>> to add new jobs.
>>>>>>>>>>>>>>> I am using aurora v0.12 released last month and have
>>>>>>>>>>>>>>> upgraded to mesos v0.25 accordingly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Below is a summary of my (very limited) understanding of
>>>>>>>>>>>>>>> some APIs, & would help it if someone can point out flaws in my
>>>>>>>>>>>>>>> understanding:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. All APIs require thrift inputs of the structs
>>>>>>>>>>>>>>> specified, and return thrift values only in Response.result
>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. Is there a set of examples in the documentation to
>>>>>>>>>>>>>>> help understand Thrift API better?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>>>>>>>>> This is basically the API to replace the Aurora
>>>>>>>>>>>>>>> DSL/.aurora files for job configuration.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>>>>>>>>> locking and some don't. For example, createJob needs a Lock
>>>>>>>>>>>>>>> object as
>>>>>>>>>>>>>>> parameter, & I am assuming that it is required so that one
>>>>>>>>>>>>>>> does not create
>>>>>>>>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>>>>>>>>> By the naming convention, it seems this is used to
>>>>>>>>>>>>>>> increase the number of instances of a job. It will not
>>>>>>>>>>>>>>> result in stopping
>>>>>>>>>>>>>>> of current instances of the job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My second explanation for this API: Since it uses a set
>>>>>>>>>>>>>>> of instanceIds, this is used for adding already running job
>>>>>>>>>>>>>>> in slaves to
>>>>>>>>>>>>>>> the internal data structures of Aurora to track the job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>>>>>>>>> Return the reason (in string) about why the job is
>>>>>>>>>>>>>>> PENDING. For example: insufficient CPU.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 8. killTasks to kill all running instances of a job in
>>>>>>>>>>>>>>> the cluster.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string
>>>>>>>>>>>>>>> message):
>>>>>>>>>>>>>>> Used for updating jobs with the new TaskConfig
>>>>>>>>>>>>>>> specified. Can be used if resource requirement changes. For
>>>>>>>>>>>>>>> example: If I
>>>>>>>>>>>>>>> wanted aurora to update the version of container used for a
>>>>>>>>>>>>>>> job using
>>>>>>>>>>>>>>> TaskConfig.Container attribute.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An aurora scheduling question is if I start a job with 5
>>>>>>>>>>>>>>> instances, and there are resources available to run only 4 of
>>>>>>>>>>>>>>> them, does
>>>>>>>>>>>>>>> the entire job block, or only the 5th instance of the job
>>>>>>>>>>>>>>> blocks?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> κρισhναν
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
