Re: Build thermos executor on ARM
Minor addition to the explanation of Florian: the folder in question has to be
called `third_party’ (mind the underscore).
From: Florian Pfeiffer <fpfeif...@x8s.de>
Reply-To: "user@aurora.apache.org" <user@aurora.apache.org>
Date: Tuesday 19 July 2016 at 10:19
To: "user@aurora.apache.org" <user@aurora.apache.org>
Subject: Re: Build thermos executor on ARM
if this hasn't changed somewhen during the last versions, you need to create a
directory "thirdparty" (written out.. don't mix it up with the already existing
"3rdparty" directory) and copy the egg there.
2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas
<johngou...@gmail.com<mailto:johngou...@gmail.com>>:
great, so if I understood correctly, it just needs the *.egg files right?
since I have built it myself, what is the directory in which I would have to
copy them?
On 18 July 2016 at 16:52, Erb, Stephan
<stephan@blue-yonder.com<mailto:stephan@blue-yonder.com>> wrote:
The Thermos executor is bundling the Mesos driver necessary for communication
with to the agent. To make this work, you have to build Python egg files
containing the Mesos driver. This is the script we normally use to achieve
this:
https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
From: Yiannis Gkoufas <johngou...@gmail.com<mailto:johngou...@gmail.com>>
Reply-To: "user@aurora.apache.org<mailto:user@aurora.apache.org>"
<user@aurora.apache.org<mailto:user@aurora.apache.org>>
Date: Monday 18 July 2016 at 17:35
To: "user@aurora.apache.org<mailto:user@aurora.apache.org>"
<user@aurora.apache.org<mailto:user@aurora.apache.org>>
Subject: Re: Build thermos executor on ARM
So the error I am getting now is:
15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from:
https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
...
INFO] Attempting to fetch thrift binary from:
http://localhost:/bin/thrift/local/0.9.1/thrift ...
100% .. 20270 KB 0.285s
INFO] Fetched thrift binary from:
http://localhost:/bin/thrift/local/0.9.1/thrift .
Waiting for background workers to finish.
15:33:39 00:32 [complete]
FAILURE
Exception caught: ()
Exception message: Could not satisfy all requirements for mesos.native==0.28.2:
mesos.native==0.28.2
Which is kind of strange, because I have actually already installed mesos 0.28.2
On 18 July 2016 at 16:16, John Sirois
<john.sir...@gmail.com<mailto:john.sir...@gmail.com>> wrote:
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas"
<johngou...@gmail.com<mailto:johngou...@gmail.com>> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas
> <johngou...@gmail.com<mailto:johngou...@gmail.com>> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois
>> <john.sir...@gmail.com<mailto:john.sir...@gmail.com>> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the Aurora
>>> pants build without modifying pants itself to do all this automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built binaries.
>>> You can add a local HTTP server to the list and point that server at the
>>> thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
>>> (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under (here
>>> is an example from my machine with result 'x86_64' - yours will be
Re: Build thermos executor on ARM
The Thermos executor is bundling the Mesos driver necessary for communication
with to the agent. To make this work, you have to build Python egg files
containing the Mesos driver. This is the script we normally use to achieve
this:
https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
From: Yiannis Gkoufas <johngou...@gmail.com>
Reply-To: "user@aurora.apache.org" <user@aurora.apache.org>
Date: Monday 18 July 2016 at 17:35
To: "user@aurora.apache.org" <user@aurora.apache.org>
Subject: Re: Build thermos executor on ARM
So the error I am getting now is:
15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from:
https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift
...
INFO] Attempting to fetch thrift binary from:
http://localhost:/bin/thrift/local/0.9.1/thrift ...
100% .. 20270 KB 0.285s
INFO] Fetched thrift binary from:
http://localhost:/bin/thrift/local/0.9.1/thrift .
Waiting for background workers to finish.
15:33:39 00:32 [complete]
FAILURE
Exception caught: ()
Exception message: Could not satisfy all requirements for mesos.native==0.28.2:
mesos.native==0.28.2
Which is kind of strange, because I have actually already installed mesos 0.28.2
On 18 July 2016 at 16:16, John Sirois
<john.sir...@gmail.com<mailto:john.sir...@gmail.com>> wrote:
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas"
<johngou...@gmail.com<mailto:johngou...@gmail.com>> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas
> <johngou...@gmail.com<mailto:johngou...@gmail.com>> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois
>> <john.sir...@gmail.com<mailto:john.sir...@gmail.com>> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the Aurora
>>> pants build without modifying pants itself to do all this automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built binaries.
>>> You can add a local HTTP server to the list and point that server at the
>>> thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
>>> (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under (here
>>> is an example from my machine with result 'x86_64' - yours will be
>>> different and arm-ey):
>>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
>>> sysname.lower(), machine'
>>> linux x86_64
>>>
>>> 3.) Edit pants.ini to tell pants where to look for binaries for your
>>> machine:
>>> $ git diff -U1
>>> diff --git a/pants.ini b/pants.ini
>>> index de0ab30..e2cd94e 100644
>>> --- a/pants.ini
>>> +++ b/pants.ini
>>> @@ -32,2 +32,6 @@ backend_packages: [
>>>
>>> +[binaries]
>>> +baseurls: +['http://localhost:']
>>> +path_by_id: {('linux', 'x86_64'): ['local']}
>>> +
>>> [thrift-binary]
>>>
>>> 4.) Setup the URL you told pants about above:
>>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
>>> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
>>> >&/tmp/thrift.serve &) && cd -
>>>
>>> 5.)
Re: Build thermos executor on ARM
On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <johngou...@gmail.com> wrote:
>
> Hi John,
>
> just to confirm: this is how this part of pants.ini should look like:
>
> backend_packages: [
> [binaries]
> baseurls: +['http://localhost:']
> path_by_id: {('linux', 'armv7l'): ['local']}
> ]
>
> right?
Not quite, kill the surrounding backend_packages, so:
[binaries]
baseurls: +['http://localhost:']
path_by_id: {('linux', 'armv7l'): ['local']}
>
> On 17 July 2016 at 11:58, Yiannis Gkoufas <johngou...@gmail.com> wrote:
>>
>> Wow, thanks a lot John!
>> Will give it a shot and get back to you!
>>
>> On 15 July 2016 at 18:05, John Sirois <john.sir...@gmail.com> wrote:
>>>
>>> Here's a workaround for you to try. If this works, I can modify the
Aurora pants build without modifying pants itself to do all this
automatically.
>>>
>>> The basic idea is that pants probes a list of URLs for pre-built
binaries. You can add a local HTTP server to the list and point that
server at the thrift binary Aurora builds for gradle via make.
>>> Below is how you do this:
>>>
>>> 0.) Ensure the Aurora gradle thrift is built.
>>> $ ./build-support/thrift/thriftw 0.9.1 --version
>>> Thrift version 0.9.1
>>>
>>> 1.) Just for reference, the pants options in-play here
>>> $ ./pants options | grep binaries
>>> binaries.baseurls = ['
https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED)
>>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>>> binaries.path_by_id = {} (from NONE)
>>> $ ./pants options | grep binary
>>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>>
>>> 2.) Grab machine name pants will try to find your thrift binary under
(here is an example from my machine with result 'x86_64' - yours will be
different and arm-ey):
>>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
sysname.lower(), machine'
>>> linux x86_64
>>>
>>> 3.) Edit pants.ini to tell pants where to look for binaries for your
machine:
>>> $ git diff -U1
>>> diff --git a/pants.ini b/pants.ini
>>> index de0ab30..e2cd94e 100644
>>> --- a/pants.ini
>>> +++ b/pants.ini
>>> @@ -32,2 +32,6 @@ backend_packages: [
>>>
>>> +[binaries]
>>> +baseurls: +['http://localhost:']
>>> +path_by_id: {('linux', 'x86_64'): ['local']}
>>> +
>>> [thrift-binary]
>>>
>>> 4.) Setup the URL you told pants about above:
>>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
>&/tmp/thrift.serve &) && cd -
>>>
>>> 5.) Run pants and confirm download of a thrift binary for your machine:
>>> $ ./pants binary src/main/python/::
>>>
>>> 10:58:44 00:00 [main]
>>>(To run a reporting server: ./pants server)
>>> 10:58:44 00:00 [setup]
>>> 10:58:44 00:00 [parse]
>>>Executing tasks in goals: binary
>>> 10:58:44 00:00 [binary]
>>> 10:58:44 00:00 [binary-prep-command]
>>> 10:58:44 00:00 [python-binary-create]
>>> 10:58:44 00:00 [cache]
>>>No cached artifacts for 8 targets.
>>>Invalidated 8 targets.
>>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
from: http://localhost:/bin/thrift/local/0.9.1/thrift ...
>>> 100% .. 31661 KB 0.364s
>>> INFO] Fetched thrift binary from:
http://localhost:/bin/thrift/local/0.9.1/thrift .
>>> ...
>>>
>>> Let me know what you get!
>>>
>>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <johngou...@gmail.com>
wrote:
>>>>
>>>> Cool, can you imagine a way to work around this?
>>>> I have to customize pants itself right?
>>>>
>>>> On 14 July 2016 at 11:54, Erb, Stephan <stephan@blue-yonder.com>
wrote:
>>>>>
>>>>> Yeah, that sounds like it could be the issue here.
>>>>>
>>>>>
>>>>>
>>>>> From: Yiannis Gkoufas <johngou...@gmail.com>
>>>>> Reply-To: "user@au
