Hi John,
just to confirm: this is how this part of pants.ini should look like:
backend_packages: [
[binaries]
baseurls: +['http://localhost:9999']
path_by_id: {('linux', 'armv7l'): ['local']}
]
right?
On 17 July 2016 at 11:58, Yiannis Gkoufas <[email protected]> wrote:
> Wow, thanks a lot John!
> Will give it a shot and get back to you!
>
> On 15 July 2016 at 18:05, John Sirois <[email protected]> wrote:
>
>> Here's a workaround for you to try. If this works, I can modify the
>> Aurora pants build without modifying pants itself to do all this
>> automatically.
>>
>> The basic idea is that pants probes a list of URLs for pre-built
>> binaries. You can add a local HTTP server to the list and point that
>> server at the thrift binary Aurora builds for gradle via make.
>> Below is how you do this:
>>
>> 0.) Ensure the Aurora gradle thrift is built.
>> $ ./build-support/thrift/thriftw 0.9.1 --version
>> Thrift version 0.9.1
>>
>> 1.) Just for reference, the pants options in-play here
>> $ ./pants options | grep binaries
>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support']
>> (from HARDCODED)
>> binaries.fetch_timeout_secs = 30 (from HARDCODED)
>> binaries.path_by_id = {} (from NONE)
>> $ ./pants options | grep binary
>> thrift-binary.supportdir = bin/thrift (from HARDCODED)
>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini)
>>
>> 2.) Grab machine name pants will try to find your thrift binary under
>> (here is an example from my machine with result 'x86_64' - yours will be
>> different and arm-ey):
>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print
>> sysname.lower(), machine'
>> linux x86_64
>>
>> 3.) Edit pants.ini to tell pants where to look for binaries for your
>> machine:
>> $ git diff -U1
>> diff --git a/pants.ini b/pants.ini
>> index de0ab30..e2cd94e 100644
>> --- a/pants.ini
>> +++ b/pants.ini
>> @@ -32,2 +32,6 @@ backend_packages: [
>>
>> +[binaries]
>> +baseurls: +['http://localhost:9999']
>> +path_by_id: {('linux', 'x86_64'): ['local']}
>> +
>> [thrift-binary]
>>
>> 4.) Setup the URL you told pants about above:
>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1
>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift
>> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift
>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer
>> 9999 >&/tmp/thrift.serve &) && cd -
>>
>> 5.) Run pants and confirm download of a thrift binary for your machine:
>> $ ./pants binary src/main/python/::
>>
>> 10:58:44 00:00 [main]
>> (To run a reporting server: ./pants server)
>> 10:58:44 00:00 [setup]
>> 10:58:44 00:00 [parse]
>> Executing tasks in goals: binary
>> 10:58:44 00:00 [binary]
>> 10:58:44 00:00 [binary-prep-command]
>> 10:58:44 00:00 [python-binary-create]
>> 10:58:44 00:00 [cache]
>> No cached artifacts for 8 targets.
>> Invalidated 8 targets.
>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary
>> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ...
>> 100% .................................................. 31661 KB 0.364s
>> INFO] Fetched thrift binary from:
>> http://localhost:9999/bin/thrift/local/0.9.1/thrift .
>> ...
>>
>> Let me know what you get!
>>
>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas <[email protected]>
>> wrote:
>>
>>> Cool, can you imagine a way to work around this?
>>> I have to customize pants itself right?
>>>
>>> On 14 July 2016 at 11:54, Erb, Stephan <[email protected]>
>>> wrote:
>>>
>>>> Yeah, that sounds like it could be the issue here.
>>>>
>>>>
>>>>
>>>> *From: *Yiannis Gkoufas <[email protected]>
>>>> *Reply-To: *"[email protected]" <[email protected]>
>>>> *Date: *Thursday 14 July 2016 at 12:28
>>>> *To: *"[email protected]" <[email protected]>
>>>> *Subject: *Re: Build thermos executor on ARM
>>>>
>>>>
>>>>
>>>> Hi there,
>>>>
>>>>
>>>>
>>>> sure:
>>>>
>>>>
>>>>
>>>> 10:26:45 00:00 [binary]
>>>>
>>>> 10:26:45 00:00 [binary-jvm-prep-command]
>>>>
>>>> 10:26:46 00:01 [jvm_prep_command]
>>>>
>>>> 10:26:46 00:01 [binary-prep-command]
>>>>
>>>> 10:26:46 00:01 [python-binary-create]
>>>>
>>>> Detected interpreter /usr/bin/python2.7:
>>>> CPython-2.7.6
>>>>
>>>> 10:26:46 00:01 [chroot]
>>>>
>>>> Building chroot for
>>>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>>> thermos_executor))]:
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core))
>>>>
>>>> Dumping library:
>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring))
>>>>
>>>> Dumping library:
>>>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD,
>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)),
>>>> thermos_executor))
>>>>
>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>>> 3.10.40
>>>>
>>>>
>>>>
>>>> If I am not completely wrong, I think pants is looking one of the
>>>> architectures outlined here:
>>>>
>>>>
>>>>
>>>>
>>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24
>>>>
>>>>
>>>>
>>>> and probably that's why it fails
>>>>
>>>>
>>>>
>>>> On 14 July 2016 at 11:17, Erb, Stephan <[email protected]>
>>>> wrote:
>>>>
>>>> Cool, ARM J. Please try to run it like this, so that we get a little
>>>> bit more output:
>>>>
>>>>
>>>>
>>>> PEX_VERBOSE=1 ./pants -ldebug binary
>>>> src/main/python/apache/aurora/executor:thermos_executor
>>>>
>>>>
>>>>
>>>> *From: *Yiannis Gkoufas <[email protected]>
>>>> *Reply-To: *"[email protected]" <[email protected]>
>>>> *Date: *Thursday 14 July 2016 at 10:44
>>>> *To: *"[email protected]" <[email protected]>
>>>> *Subject: *Build thermos executor on ARM
>>>>
>>>>
>>>>
>>>> Hi guys,
>>>>
>>>>
>>>>
>>>> I have been trying to get aurora working on ARM processors.
>>>>
>>>> I had no problem with the scheduler, but I didn't manage to compile the
>>>> thermos executor.
>>>>
>>>>
>>>>
>>>> When I give:
>>>>
>>>>
>>>>
>>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor
>>>>
>>>>
>>>>
>>>> I get back:
>>>>
>>>>
>>>>
>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l
>>>> 3.10.40
>>>>
>>>>
>>>>
>>>> Let me know if you have any hints on how to workaround this.
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
