Thanks Jake!
That worked like a charm, & I was wondering why does install from source
doesn't work!
Aurora Log:
Mar 18 14:34:49 adx-aurora-2 aurora-start.bash[947]: W0318 14:34:49.109
[qtp743672940-126, HttpParser:1286] bad HTTP parsed: 400 for
HttpChannelOverHttp@11212ec3{r=0,c=false,a=IDLE,uri=null}
Thrift client:
./mrfantastic_service.out
Connecting to aurora....
<nil> EOF
Thrift client sourcec:
func thriftAuroraJobs() {
var protocolFactory thrift.TProtocolFactory
var transportFactory thrift.TTransportFactory
var transport thrift.TTransport
var client *api.ReadOnlySchedulerClient
var err error
protocolFactory = thrift.NewTBinaryProtocolFactoryDefault()
//transportFactory =
thrift.NewTFramedTransportFactory(thrift.NewTTransportFactory())
transportFactory = thrift.NewTTransportFactory()
fmt.Println("Connecting to aurora....")
transport, err = thrift.NewTSocket("54.210.234.190:8081")
if err != nil {
fmt.Println("Error opening socket:", err)
os.Exit(1)
//return err
}
if transport == nil {
os.Exit(1)
}
transport = transportFactory.GetTransport(transport)
if transport == nil {
os.Exit(1)
}
err = transport.Open()
if err != nil {
os.Exit(1)
}
defer transport.Close()
client = api.NewReadOnlySchedulerClientFactory(transport,
protocolFactory)
fmt.Println(client.GetJobs(""))
}
My hunch is that the thrift port isn't 8081, as the server/aurora is
looking for HTTP data on the socket.
Is there a config that needs to be set for thrift API to be initialized?
--
κρισhναν
On Thu, Mar 17, 2016 at 10:08 PM, Jake Farrell <[email protected]> wrote:
> if you just need the compiler you can install the thrift-compiler package
> with one of the following, otherwise you can run ./bootstrap.sh &&
> ./configure && cd compiler/cpp && make to just build the compiler.
>
> -Jake
>
>
>
> deb packaging (tested with ubuntu trusty)
>
> > curl -sSL http://apache.org/dist/thrift/KEYS | gpg --import -
> > gpg --export --armor 66B778F9 | sudo apt-key add -
> > /etc/apt/sources.list.d/thrift.list
>
> deb http://www.apache.org/dist/thrift/debian 0.9.3 main
>
>
> or for centos/rhel (tested with centos 7.2)
>
> > /etc/yum.repos.d/thrift.repo
>
> [thrift]
> name=Apache Thrift rpm repo
> baseurl=http://www.apache.org/dist/thrift/rpm/
> enabled=1
> gpgcheck=0
>
>
>
> On Thu, Mar 17, 2016 at 12:32 PM, Krish <[email protected]> wrote:
>
>> Jake/Chris,
>> Thanks for the info.
>> When I try to install thrift v0.9.3 from source, I get an error as
>> follows while running `make check`:
>> ...
>> ...
>> [junit] Running org.apache.thrift.protocol.TestTProtocolUtil
>> [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
>> elapsed: 0.062 sec
>> [junit] Running org.apache.thrift.protocol.TestTSimpleJSONProtocol
>> [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
>> elapsed: 0.046 sec
>>
>> BUILD FAILED
>> /tmp/thrift-0.9.3/lib/java/build.xml:202: Test
>> org.apache.thrift.protocol.TestTSimpleJSONProtocol failed
>>
>> Total time: 17 seconds
>> make[3]: *** [check-local] Error 1
>> make[3]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>> make[2]: *** [check-am] Error 2
>> make[2]: Leaving directory `/tmp/thrift-0.9.3/lib/java'
>> make[1]: *** [check-recursive] Error 1
>> make[1]: Leaving directory `/tmp/thrift-0.9.3/lib'
>> make: *** [check-recursive] Error 1
>>
>>
>>
>>
>> --
>> κρισhναν
>>
>> On Thu, Mar 17, 2016 at 7:32 PM, Chris Bannister <[email protected]>
>> wrote:
>>
>>> I've used the latest thrift to generate go code, and then manually
>>> created executor config which works and is able to launch jobs.
>>>
>>> On Thu, 17 Mar 2016, 1:55 p.m. Jake Farrell, <[email protected]>
>>> wrote:
>>>
>>>> Hi Krish
>>>> We are using Thrift with go for all our api calls to Aurora, would
>>>> recommend you use Thrift 0.9.3 to interact with the api.
>>>>
>>>> happy to help answer any questions you might have
>>>>
>>>> -Jake
>>>>
>>>> On Thu, Mar 17, 2016 at 9:43 AM, Krish <[email protected]>
>>>> wrote:
>>>>
>>>>> Thanks, Bill.
>>>>>
>>>>> Well I have started my foray into the the thrift API today. And I
>>>>> think I am stuck with some thrift configs.
>>>>>
>>>>> Does it matter if I use thrift v0.9.0 on the client side to talk with
>>>>> aurora using thrift 0.9.1? Are they compatible? I couldn't find any
>>>>> changelog or compatibility statement on the thrift project site.
>>>>>
>>>>>
>>>>> Since Aurora v0.12 uses thrift version 0.9.1, and the debian repos
>>>>> have 0.9.0, I had to compile the thrift compiler v0.9.1 from source.
>>>>> However, when I try to generate golang code, I think I hit a compiler bug:
>>>>> krish@krish:/tmp
>>>>> > thrift --gen go api.thrift
>>>>> ./gen-go//api/ttypes.go:2623:6: missing ',' in composite literal
>>>>> ./gen-go//api/ttypes.go:2624:19: expected '==', found '='
>>>>> WARNING - Running 'gofmt -w ./gen-go//api/ttypes.go' failed.
>>>>>
>>>>> I can modify the golang code by hand, but I would like to play it safe
>>>>> and use the working compiler from the debian repos.
>>>>>
>>>>> Also, when I use thrift v0.9.0, and try to integrate code into a test
>>>>> golang app, it fails to find "thriftlib/api" package. Anyone faced a
>>>>> similar error and gone past it?
>>>>> I have already done a `go get
>>>>> git.apache.org/thrift.git/lib/go/thrift/...`
>>>>> <http://git.apache.org/thrift.git/lib/go/thrift/...>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> κρισhναν
>>>>>
>>>>> On Wed, Mar 16, 2016 at 10:30 PM, Bill Farner <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Regarding documentation - Maxim is correct that there isn't much in
>>>>>> the way of independent/holistic docs for the thrift API. There is,
>>>>>> however, scant javadoc-style documentation within the IDL spec itself:
>>>>>> https://github.com/apache/aurora/blob/master/api/src/main/thrift/org/apache/aurora/gen/api.thrift
>>>>>>
>>>>>> If you are looking to use the thrift API directly, the most difficult
>>>>>> API method will be defining the ExecutorConfig.data value when calling
>>>>>> createJob. Please don't hesitate to ask for assistance if you get to
>>>>>> that
>>>>>> point!
>>>>>>
>>>>>> On Wed, Mar 16, 2016 at 9:19 AM, Maxim Khutornenko <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>> return thrift values only in Response.result field.
>>>>>>>
>>>>>>> Correct. There is also 'details' field that may have additional
>>>>>>> messages (of error or informational nature)
>>>>>>>
>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>> understand Thrift API better?
>>>>>>>
>>>>>>> The thrift API is largely undocumented. There is an effort to bring
>>>>>>> up a fully supported REST API that will presumably get documented and
>>>>>>> become much easier to use. It's mostly in flux now.
>>>>>>>
>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>
>>>>>>> This is the API to use when you a brand new service or adhoc (batch)
>>>>>>> job created. The JobDescription is populated from the .aurora config.
>>>>>>> You
>>>>>>> may want to trace "aurora job create" client command implementation to
>>>>>>> see
>>>>>>> how it happens.
>>>>>>>
>>>>>>> 4. What is the Lock object? I see that some APIs require locking and
>>>>>>>> some don't. For example, createJob needs a Lock object as parameter, &
>>>>>>>> I am
>>>>>>>> assuming that it is required so that one does not create multiple jobs
>>>>>>>> with
>>>>>>>> the same JobKey.
>>>>>>>
>>>>>>> Ignore this object as it's an echo of the old client updater. It's
>>>>>>> now deprecated and will be removed soon. You can pass NULL for now.
>>>>>>>
>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>
>>>>>>> Another echo of the client updater but this time it's got a second
>>>>>>> life. Check out its new signature and comments in the api.thrift. It's
>>>>>>> essentially a "scale-out" API that can add instances to the existing job
>>>>>>> without changing the underlying task assumptions.
>>>>>>>
>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>
>>>>>>> It's actually 'getPendingReason' and is currently used exclusively
>>>>>>> by the UI to get the reason for a task PENDING state.
>>>>>>>
>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>
>>>>>>> This is to set role-level quota. Currently only required for tasks
>>>>>>> with 'production=True'. Search through our docs for more details.
>>>>>>>
>>>>>>> 8. killTasks to kill all running instances of a job in the cluster.
>>>>>>>
>>>>>>> It's quite versatile and can be used to kill some or all instances
>>>>>>> of the job.
>>>>>>>
>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>
>>>>>>> Your observations are correct. This is the main API to change your
>>>>>>> service job in any way (including adding, removing or modifying
>>>>>>> instances).
>>>>>>>
>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>>> and there are resources available to run only 4 of them, does the
>>>>>>>> entire
>>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>
>>>>>>> Scheduler will try to schedule as many instances as it can. Those
>>>>>>> that will not find resources will remain in PENDING state until more
>>>>>>> resources are available. In your particular example only the 5th will
>>>>>>> remain PENDING.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 16, 2016 at 5:54 AM, Krish <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I was going through the Aurora Thrift API to determine how to add
>>>>>>>> new jobs.
>>>>>>>> I am using aurora v0.12 released last month and have upgraded to
>>>>>>>> mesos v0.25 accordingly.
>>>>>>>>
>>>>>>>> Below is a summary of my (very limited) understanding of some APIs,
>>>>>>>> & would help it if someone can point out flaws in my understanding:
>>>>>>>>
>>>>>>>> 1. All APIs require thrift inputs of the structs specified, and
>>>>>>>> return thrift values only in Response.result field.
>>>>>>>>
>>>>>>>> 2. Is there a set of examples in the documentation to help
>>>>>>>> understand Thrift API better?
>>>>>>>>
>>>>>>>> 3. createJob(JobDescription desc, Lock lock):
>>>>>>>> This is basically the API to replace the Aurora DSL/.aurora
>>>>>>>> files for job configuration.
>>>>>>>>
>>>>>>>> 4. What is the Lock object? I see that some APIs require
>>>>>>>> locking and some don't. For example, createJob needs a Lock object
>>>>>>>> as
>>>>>>>> parameter, & I am assuming that it is required so that one does not
>>>>>>>> create
>>>>>>>> multiple jobs with the same JobKey.
>>>>>>>>
>>>>>>>> 5. addInstances(AddInstancesConfig cfg, Lock lock):
>>>>>>>> By the naming convention, it seems this is used to increase the
>>>>>>>> number of instances of a job. It will not result in stopping of
>>>>>>>> current
>>>>>>>> instances of the job.
>>>>>>>>
>>>>>>>> My second explanation for this API: Since it uses a set of
>>>>>>>> instanceIds, this is used for adding already running job in slaves
>>>>>>>> to the
>>>>>>>> internal data structures of Aurora to track the job.
>>>>>>>>
>>>>>>>> 6. getPendingResult(TaskQuery taskquery):
>>>>>>>> Return the reason (in string) about why the job is PENDING. For
>>>>>>>> example: insufficient CPU.
>>>>>>>>
>>>>>>>> 7. setQuota & getQuota for setting user level quotas.
>>>>>>>>
>>>>>>>> 8. killTasks to kill all running instances of a job in the
>>>>>>>> cluster.
>>>>>>>>
>>>>>>>> 9. startJobUpdate(JobUpdateRequest request, string message):
>>>>>>>> Used for updating jobs with the new TaskConfig specified. Can
>>>>>>>> be used if resource requirement changes. For example: If I wanted
>>>>>>>> aurora to
>>>>>>>> update the version of container used for a job using
>>>>>>>> TaskConfig.Container
>>>>>>>> attribute.
>>>>>>>>
>>>>>>>>
>>>>>>>> An aurora scheduling question is if I start a job with 5 instances,
>>>>>>>> and there are resources available to run only 4 of them, does the
>>>>>>>> entire
>>>>>>>> job block, or only the 5th instance of the job blocks?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> --
>>>>>>>> κρισhναν
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
