Minor addition to the explanation of Florian: the folder in question has to be called `third_party’ (mind the underscore).
From: Florian Pfeiffer <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday 19 July 2016 at 10:19 To: "[email protected]" <[email protected]> Subject: Re: Build thermos executor on ARM if this hasn't changed somewhen during the last versions, you need to create a directory "thirdparty" (written out.. don't mix it up with the already existing "3rdparty" directory) and copy the egg there. 2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas <[email protected]<mailto:[email protected]>>: great, so if I understood correctly, it just needs the *.egg files right? since I have built it myself, what is the directory in which I would have to copy them? On 18 July 2016 at 16:52, Erb, Stephan <[email protected]<mailto:[email protected]>> wrote: The Thermos executor is bundling the Mesos driver necessary for communication with to the agent. To make this work, you have to build Python egg files containing the Mesos driver. This is the script we normally use to achieve this: https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg From: Yiannis Gkoufas <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday 18 July 2016 at 17:35 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Build thermos executor on ARM So the error I am getting now is: 15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift ... INFO] Attempting to fetch thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ... 100% .................................................. 20270 KB 0.285s INFO] Fetched thrift binary from: http://localhost:9999/bin/thrift/local/0.9.1/thrift . Waiting for background workers to finish. 15:33:39 00:32 [complete] FAILURE Exception caught: (<class 'pex.resolver.Unsatisfiable'>) Exception message: Could not satisfy all requirements for mesos.native==0.28.2: mesos.native==0.28.2 Which is kind of strange, because I have actually already installed mesos 0.28.2 On 18 July 2016 at 16:16, John Sirois <[email protected]<mailto:[email protected]>> wrote: On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <[email protected]<mailto:[email protected]>> wrote: > > Hi John, > > just to confirm: this is how this part of pants.ini should look like: > > backend_packages: [ > [binaries] > baseurls: +['http://localhost:9999'] > path_by_id: {('linux', 'armv7l'): ['local']} > ] > > right? Not quite, kill the surrounding backend_packages, so: [binaries] baseurls: +['http://localhost:9999'] path_by_id: {('linux', 'armv7l'): ['local']} > > On 17 July 2016 at 11:58, Yiannis Gkoufas > <[email protected]<mailto:[email protected]>> wrote: >> >> Wow, thanks a lot John! >> Will give it a shot and get back to you! >> >> On 15 July 2016 at 18:05, John Sirois >> <[email protected]<mailto:[email protected]>> wrote: >>> >>> Here's a workaround for you to try. If this works, I can modify the Aurora >>> pants build without modifying pants itself to do all this automatically. >>> >>> The basic idea is that pants probes a list of URLs for pre-built binaries. >>> You can add a local HTTP server to the list and point that server at the >>> thrift binary Aurora builds for gradle via make. >>> Below is how you do this: >>> >>> 0.) Ensure the Aurora gradle thrift is built. >>> $ ./build-support/thrift/thriftw 0.9.1 --version >>> Thrift version 0.9.1 >>> >>> 1.) Just for reference, the pants options in-play here >>> $ ./pants options | grep binaries >>> binaries.baseurls = ['https://dl.bintray.com/pantsbuild/bin/build-support'] >>> (from HARDCODED) >>> binaries.fetch_timeout_secs = 30 (from HARDCODED) >>> binaries.path_by_id = {} (from NONE) >>> $ ./pants options | grep binary >>> thrift-binary.supportdir = bin/thrift (from HARDCODED) >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini) >>> >>> 2.) Grab machine name pants will try to find your thrift binary under (here >>> is an example from my machine with result 'x86_64' - yours will be >>> different and arm-ey): >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); print >>> sysname.lower(), machine' >>> linux x86_64 >>> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your >>> machine: >>> $ git diff -U1 >>> diff --git a/pants.ini b/pants.ini >>> index de0ab30..e2cd94e 100644 >>> --- a/pants.ini >>> +++ b/pants.ini >>> @@ -32,2 +32,6 @@ backend_packages: [ >>> >>> +[binaries] >>> +baseurls: +['http://localhost:9999'] >>> +path_by_id: {('linux', 'x86_64'): ['local']} >>> + >>> [thrift-binary] >>> >>> 4.) Setup the URL you told pants about above: >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1 >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift >>> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer 9999 >>> >&/tmp/thrift.serve &) && cd - >>> >>> 5.) Run pants and confirm download of a thrift binary for your machine: >>> $ ./pants binary src/main/python/:: >>> >>> 10:58:44 00:00 [main] >>> (To run a reporting server: ./pants server) >>> 10:58:44 00:00 [setup] >>> 10:58:44 00:00 [parse] >>> Executing tasks in goals: binary >>> 10:58:44 00:00 [binary] >>> 10:58:44 00:00 [binary-prep-command] >>> 10:58:44 00:00 [python-binary-create] >>> 10:58:44 00:00 [cache] >>> No cached artifacts for 8 targets. >>> Invalidated 8 targets. >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary from: >>> http://localhost:9999/bin/thrift/local/0.9.1/thrift ... >>> 100% .................................................. 31661 KB 0.364s >>> INFO] Fetched thrift binary from: >>> http://localhost:9999/bin/thrift/local/0.9.1/thrift . >>> ... >>> >>> Let me know what you get! >>> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas >>> <[email protected]<mailto:[email protected]>> wrote: >>>> >>>> Cool, can you imagine a way to work around this? >>>> I have to customize pants itself right? >>>> >>>> On 14 July 2016 at 11:54, Erb, Stephan >>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> >>>>> Yeah, that sounds like it could be the issue here. >>>>> >>>>> >>>>> >>>>> From: Yiannis Gkoufas <[email protected]<mailto:[email protected]>> >>>>> Reply-To: "[email protected]<mailto:[email protected]>" >>>>> <[email protected]<mailto:[email protected]>> >>>>> Date: Thursday 14 July 2016 at 12:28 >>>>> To: "[email protected]<mailto:[email protected]>" >>>>> <[email protected]<mailto:[email protected]>> >>>>> Subject: Re: Build thermos executor on ARM >>>>> >>>>> >>>>>> >>>>>> Hi there, >>>>>> >>>>>> >>>>>> >>>>>> sure: >>>>>> >>>>>> >>>>>> >>>>>> 10:26:45 00:00 [binary] >>>>>> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command] >>>>>> >>>>>> 10:26:46 00:01 [jvm_prep_command] >>>>>> >>>>>> 10:26:46 00:01 [binary-prep-command] >>>>>> >>>>>> 10:26:46 00:01 [python-binary-create] >>>>>> >>>>>> Detected interpreter /usr/bin/python2.7: CPython-2.7.6 >>>>>> >>>>>> 10:26:46 00:01 [chroot] >>>>>> >>>>>> Building chroot for >>>>>> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), >>>>>> thermos_executor))]: >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core)) >>>>>> >>>>>> Dumping library: >>>>>> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring)) >>>>>> >>>>>> Dumping library: >>>>>> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >>>>>> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), >>>>>> thermos_executor)) >>>>>> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux armv7l >>>>>> 3.10.40 >>>>>> >>>>>> >>>>>> >>>>>> If I am not completely wrong, I think pants is looking one of the >>>>>> architectures outlined here: >>>>>> >>>>>> >>>>>> >>>>>> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24 >>>>>> >>>>>> >>>>>> >>>>>> and probably that's why it fails >>>>>> >>>>>> >>>>>> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan >>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a little >>>>>>> bit more output: >>>>>>> >>>>>>> >>>>>>> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary >>>>>>> src/main/python/apache/aurora/executor:thermos_executor >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Yiannis Gkoufas >>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>> Reply-To: "[email protected]<mailto:[email protected]>" >>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>> Date: Thursday 14 July 2016 at 10:44 >>>>>>> To: "[email protected]<mailto:[email protected]>" >>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>> Subject: Build thermos executor on ARM >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I have been trying to get aurora working on ARM processors. >>>>>>>> >>>>>>>> I had no problem with the scheduler, but I didn't manage to compile >>>>>>>> the thermos executor. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When I give: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ./pants binary src/main/python/apache/aurora/executor:thermos_executor >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I get back: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux >>>>>>>> armv7l 3.10.40 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Let me know if you have any hints on how to workaround this. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks! >>>>>> >>>>>> >>>> >>>> >>> >> >
