if this hasn't changed somewhen during the last versions, you need to create a directory "thirdparty" (written out.. don't mix it up with the already existing "3rdparty" directory) and copy the egg there.
2016-07-18 18:09 GMT+02:00 Yiannis Gkoufas <[email protected]>: > great, so if I understood correctly, it just needs the *.egg files right? > since I have built it myself, what is the directory in which I would have > to copy them? > > On 18 July 2016 at 16:52, Erb, Stephan <[email protected]> > wrote: > >> The Thermos executor is bundling the Mesos driver necessary for >> communication with to the agent. To make this work, you have to build >> Python egg files containing the Mesos driver. This is the script we >> normally use to achieve this: >> https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg >> >> >> >> >> >> *From: *Yiannis Gkoufas <[email protected]> >> *Reply-To: *"[email protected]" <[email protected]> >> *Date: *Monday 18 July 2016 at 17:35 >> >> *To: *"[email protected]" <[email protected]> >> *Subject: *Re: Build thermos executor on ARM >> >> >> >> So the error I am getting now is: >> >> >> >> 15:33:32 00:25 [chroot]INFO] Attempting to fetch thrift binary >> from: >> https://dl.bintray.com/pantsbuild/bin/build-support/bin/thrift/local/0.9.1/thrift >> ... >> >> INFO] Attempting to fetch thrift binary from: >> http://localhost:9999/bin/thrift/local/0.9.1/thrift ... >> >> 100% .................................................. 20270 KB 0.285s >> >> INFO] Fetched thrift binary from: >> http://localhost:9999/bin/thrift/local/0.9.1/thrift . >> >> >> >> Waiting for background workers to finish. >> >> 15:33:39 00:32 [complete] >> >> FAILURE >> >> Exception caught: (<class 'pex.resolver.Unsatisfiable'>) >> >> >> >> Exception message: Could not satisfy all requirements for >> mesos.native==0.28.2: >> >> mesos.native==0.28.2 >> >> >> >> Which is kind of strange, because I have actually already installed mesos >> 0.28.2 >> >> >> >> On 18 July 2016 at 16:16, John Sirois <[email protected]> wrote: >> >> On Jul 18, 2016 9:04 AM, "Yiannis Gkoufas" <[email protected]> wrote: >> > >> > Hi John, >> > >> > just to confirm: this is how this part of pants.ini should look like: >> > >> > backend_packages: [ >> > [binaries] >> > baseurls: +['http://localhost:9999'] >> > path_by_id: {('linux', 'armv7l'): ['local']} >> > ] >> > >> > right? >> >> Not quite, kill the surrounding backend_packages, so: >> >> [binaries] >> baseurls: +['http://localhost:9999'] >> path_by_id: {('linux', 'armv7l'): ['local']} >> >> > >> >> >> > On 17 July 2016 at 11:58, Yiannis Gkoufas <[email protected]> wrote: >> >> >> >> Wow, thanks a lot John! >> >> Will give it a shot and get back to you! >> >> >> >> On 15 July 2016 at 18:05, John Sirois <[email protected]> wrote: >> >>> >> >>> Here's a workaround for you to try. If this works, I can modify the >> Aurora pants build without modifying pants itself to do all this >> automatically. >> >>> >> >>> The basic idea is that pants probes a list of URLs for pre-built >> binaries. You can add a local HTTP server to the list and point that >> server at the thrift binary Aurora builds for gradle via make. >> >>> Below is how you do this: >> >>> >> >>> 0.) Ensure the Aurora gradle thrift is built. >> >>> $ ./build-support/thrift/thriftw 0.9.1 --version >> >>> Thrift version 0.9.1 >> >>> >> >>> 1.) Just for reference, the pants options in-play here >> >>> $ ./pants options | grep binaries >> >>> binaries.baseurls = [' >> https://dl.bintray.com/pantsbuild/bin/build-support'] (from HARDCODED) >> >>> binaries.fetch_timeout_secs = 30 (from HARDCODED) >> >>> binaries.path_by_id = {} (from NONE) >> >>> $ ./pants options | grep binary >> >>> thrift-binary.supportdir = bin/thrift (from HARDCODED) >> >>> thrift-binary.version = 0.9.1 (from CONFIG in pants.ini) >> >>> >> >>> 2.) Grab machine name pants will try to find your thrift binary under >> (here is an example from my machine with result 'x86_64' - yours will be >> different and arm-ey): >> >>> $ python2 -c 'import os; sysname, _, _, _, machine = os.uname(); >> print sysname.lower(), machine' >> >>> linux x86_64 >> >>> >> >>> 3.) Edit pants.ini to tell pants where to look for binaries for your >> machine: >> >>> $ git diff -U1 >> >>> diff --git a/pants.ini b/pants.ini >> >>> index de0ab30..e2cd94e 100644 >> >>> --- a/pants.ini >> >>> +++ b/pants.ini >> >>> @@ -32,2 +32,6 @@ backend_packages: [ >> >>> >> >>> +[binaries] >> >>> +baseurls: +['http://localhost:9999'] >> >>> +path_by_id: {('linux', 'x86_64'): ['local']} >> >>> + >> >>> [thrift-binary] >> >>> >> >>> 4.) Setup the URL you told pants about above: >> >>> $ mkdir -p build-support/thrift/serve/bin/thrift/local/0.9.1 >> >>> $ ln -s $PWD/build-support/thrift/thrift-0.9.1/compiler/cpp/thrift >> build-support/thrift/serve/bin/thrift/local/0.9.1/thrift >> >>> $ cd build-support/thrift/serve && (nohup python2 -m SimpleHTTPServer >> 9999 >&/tmp/thrift.serve &) && cd - >> >>> >> >>> 5.) Run pants and confirm download of a thrift binary for your >> machine: >> >>> $ ./pants binary src/main/python/:: >> >>> >> >>> 10:58:44 00:00 [main] >> >>> (To run a reporting server: ./pants server) >> >>> 10:58:44 00:00 [setup] >> >>> 10:58:44 00:00 [parse] >> >>> Executing tasks in goals: binary >> >>> 10:58:44 00:00 [binary] >> >>> 10:58:44 00:00 [binary-prep-command] >> >>> 10:58:44 00:00 [python-binary-create] >> >>> 10:58:44 00:00 [cache] >> >>> No cached artifacts for 8 targets. >> >>> Invalidated 8 targets. >> >>> 10:58:44 00:00 [chroot]INFO] Attempting to fetch thrift binary >> from: http://localhost:9999/bin/thrift/local/0.9.1/thrift ... >> >>> 100% .................................................. 31661 KB >> 0.364s >> >>> INFO] Fetched thrift binary from: >> http://localhost:9999/bin/thrift/local/0.9.1/thrift . >> >>> ... >> >>> >> >>> Let me know what you get! >> >>> >> >>> On Thu, Jul 14, 2016 at 6:24 AM, Yiannis Gkoufas < >> [email protected]> wrote: >> >>>> >> >>>> Cool, can you imagine a way to work around this? >> >>>> I have to customize pants itself right? >> >>>> >> >>>> On 14 July 2016 at 11:54, Erb, Stephan <[email protected]> >> wrote: >> >>>>> >> >>>>> Yeah, that sounds like it could be the issue here. >> >>>>> >> >>>>> >> >>>>> >> >>>>> From: Yiannis Gkoufas <[email protected]> >> >>>>> Reply-To: "[email protected]" <[email protected]> >> >>>>> Date: Thursday 14 July 2016 at 12:28 >> >>>>> To: "[email protected]" <[email protected]> >> >>>>> Subject: Re: Build thermos executor on ARM >> >>>>> >> >>>>> >> >>>>>> >> >>>>>> Hi there, >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> sure: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> 10:26:45 00:00 [binary] >> >>>>>> >> >>>>>> 10:26:45 00:00 [binary-jvm-prep-command] >> >>>>>> >> >>>>>> 10:26:46 00:01 [jvm_prep_command] >> >>>>>> >> >>>>>> 10:26:46 00:01 [binary-prep-command] >> >>>>>> >> >>>>>> 10:26:46 00:01 [python-binary-create] >> >>>>>> >> >>>>>> Detected interpreter /usr/bin/python2.7: >> CPython-2.7.6 >> >>>>>> >> >>>>>> 10:26:46 00:01 [chroot] >> >>>>>> >> >>>>>> Building chroot for >> [PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), >> thermos_executor))]: >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), _executor)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(api/src/main/thrift/org/apache/aurora/gen/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), gen)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/common/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/config/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/config/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), config)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/common/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), common)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/core/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), core)) >> >>>>>> >> >>>>>> Dumping library: >> PythonLibrary(BuildFileAddress(BuildFile(src/main/python/apache/thermos/monitoring/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), monitoring)) >> >>>>>> >> >>>>>> Dumping library: >> PythonBinary(BuildFileAddress(BuildFile(src/main/python/apache/aurora/executor/BUILD, >> FileSystemProjectTree(/home/ubuntu/apache-aurora-0.15.0)), >> thermos_executor)) >> >>>>>> >> >>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux >> armv7l 3.10.40 >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> If I am not completely wrong, I think pants is looking one of the >> architectures outlined here: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> https://github.com/pantsbuild/pants/blob/master/src/python/pants/binaries/binary_util.py#L24 >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> and probably that's why it fails >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On 14 July 2016 at 11:17, Erb, Stephan < >> [email protected]> wrote: >> >>>>>>> >> >>>>>>> Cool, ARM J. Please try to run it like this, so that we get a >> little bit more output: >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> PEX_VERBOSE=1 ./pants -ldebug binary >> src/main/python/apache/aurora/executor:thermos_executor >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> From: Yiannis Gkoufas <[email protected]> >> >>>>>>> Reply-To: "[email protected]" <[email protected]> >> >>>>>>> Date: Thursday 14 July 2016 at 10:44 >> >>>>>>> To: "[email protected]" <[email protected]> >> >>>>>>> Subject: Build thermos executor on ARM >> >>>>>>> >> >>>>>>> >> >>>>>>>> >> >>>>>>>> Hi guys, >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> I have been trying to get aurora working on ARM processors. >> >>>>>>>> >> >>>>>>>> I had no problem with the scheduler, but I didn't manage to >> compile the thermos executor. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> When I give: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> ./pants binary >> src/main/python/apache/aurora/executor:thermos_executor >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> I get back: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> FAILURE: Update --binaries-path-by-id to find binaries for Linux >> armv7l 3.10.40 >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Let me know if you have any hints on how to workaround this. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Thanks! >> >>>>>> >> >>>>>> >> >>>> >> >>>> >> >>> >> >> >> > >> >> >> >> >
