We eventually should move away from setting a `config_setting` and instead select a `platform` as shown here: https://github.com/aosp-riscv/platform-build-bazel/blob/eba6bae790001f2bc0928117069fa15278ee4182/platforms/BUILD.bazel
This would allow us to have a `linux_amd64`, `osx_amd64` and `osx_m1` platform type. This could also help with the previous ARM64 effort that stalled out. On Sat, Feb 26, 2022 at 11:22 PM Nicholas Nezis <[email protected]> wrote: > Oh, just remembered... Here is a list of Bazel platform values. > https://github.com/bazelbuild/platforms/blob/main/cpu/BUILD > > Bazel auto populates this. Here is a full list. I believe the M1 is > `armv6-m`. > > I think I have a theory as to what's happening, but I'd have to dig into > this a bit. Short summary though, your logic in the Pex builder script is > trying to run the Linux code, and not the Darwin alternative. > > We seem to set the `darwin` and `k8` `config_settings` closures in > `tools/platform/BUILD`. We set each based on the CPU type. > > I wonder if we are mixing CPU type and OS type in our config. > > On Sat, Feb 26, 2022 at 11:09 PM Nicholas Nezis <[email protected]> > wrote: > >> So I'm wondering if it's something special with the M1. I feel like I've >> built on MacOS recently without issue. And when I run `mktemp` from the >> CLI, I do see the MacOS specific arguments. >> >> In the `tools/platform/BUILD` file, we define a `darwin` and `k8` type >> based on matching `cpu` flavor. Not sure what the M1 would register as. >> >> >> On Sat, Feb 26, 2022 at 9:56 PM Windham Wong <[email protected]> >> wrote: >> >>> It looks like happening in macOS only because macOS has the same command >>> of tools but entirely different on the parameters or usage. I personally >>> don't build Heron on my Macbook but I know the frustration of doing that >>> if the project does not support macOS natively. >>> >>> I believe this is just a missing piece for the bazel configure for using >>> special parameters on different distros, just like what I have >>> encountered on Debian as well. I would vote for a yes to add a distro >>> check before running `mktemp` here, if I have a voting right here. :P >>> >>> Regards J, >>> *Windham Wong* >>> OSWE, OSCP, GCIA, Specialist in Cybersecurity >>> Co-Founder, Managing Partner of >>> *Stormeye.io, Hong Kong Managed Security Operation Center Limited* >>> Specialist in Cybersecurity, Log Management and SIEM System >>> <https://www.stormeye.io> >>> Email // [email protected] >>> Phone // +852_3590_2212_|_+852_9832_0707 <tel:+85235902212> >>> Fax // +852_3590_2202 <tel:+852_3590_2202> >>> >>> On 27/2/2022 3:18, Josh Fischer wrote: >>> > Is that the suggested fix for this? Or is this just a short term >>> solution? >>> > >>> > I thought we had Bazel conditionals that used the appropriate tools >>> based >>> > on the distro Heron is being built on. >>> > >>> > On Sat, Feb 26, 2022 at 1:50 AM Nicholas Nezis< >>> [email protected]> >>> > wrote: >>> > >>> >> It seems that `mktemp` on Darwin has different arguments compared to >>> Linux. >>> >> >>> >> If you `brew install coreutils` and set `gnubin` on your `PATH`, you >>> might >>> >> be able to get matching `mktemp` as you'd have on linux. >>> >> >>> >> >>> >> On Fri, Feb 25, 2022 at 9:06 PM Josh Fischer<[email protected]> >>> wrote: >>> >> >>> >>> I tried to build off master expecting errors since I'm on an M1. >>> Anyone >>> >>> seen this "mktmp" error before? Python version is 3.8.x. You'll see >>> the >>> >>> full Python version at the bottom of the shell snippet. >>> >>> >>> >>> bazel build --config=darwin_nostyle scripts/packages:binpkgs >>> >>> --host_javabase=@local_jdk//:jdk --verbose_failures >>> >>> >>> >>> INFO: Analyzed target //scripts/packages:binpkgs (0 packages loaded, >>> 0 >>> >>> targets configured). >>> >>> >>> >>> INFO: Found 1 target... >>> >>> >>> >>> ERROR: >>> >>> >>> >> >>> /Users/joshfischer/Source/apache/incubator-heron/tools/rules/pex/BUILD:57:8: >>> >>> Bootstrapping pex //tools/rules/pex:pex_wrapper [for host] failed: >>> (Exit >>> >>> 1): bash failed: error executing command >>> >>> >>> >>> (cd >>> >>> >>> >>> >>> >> >>> /private/var/tmp/_bazel_joshfischer/70e9bd6aaead2621044e8d790ac25401/execroot/org_apache_heron >>> >>> && \ >>> >>> >>> >>> exec env - \ >>> >>> >>> >>> PATH=' left out for brevity >>> >>> >>> >>> /bin/bash -c 'source >>> >> external/bazel_tools/tools/genrule/genrule-setup.sh; >>> >>> OUTDIR=$(cd bazel-out/host/bin/tools/rules/pex && pwd) >>> >>> >>> >>> # Workaround really long shebang lines breaking on linux: >>> >>> >>> >>> # Use a /tmp path, but keep the actual venv inside the bazel outdir. >>> >>> >>> >>> # Avoids having to worry about cleanup, even if sandboxing is off. >>> >>> >>> >>> TMPF=$(mktemp -d -p /tmp pex.XXXXX) >>> >>> >>> >>> ln -sf "$OUTDIR" "$TMPF" >>> >>> >>> >>> VENV="${TMPF}/venv" >>> >>> >>> >>> python3 -m venv $VENV --clear >>> >>> >>> >>> VIRTUAL_ENV_DISABLE_PROMPT=1 source "$VENV/bin/activate" >>> >>> >>> >>> TEMP="bazel-out/host/bin/tools/rules/pex/pexbuild" >>> >>> >>> >>> pip install pex --quiet --no-cache-dir --no-index >>> >>> --find-links >>> >>> $(dirname external/pex_pkg/file/pex-2.1.62-py2.py3-none-any.whl) >>> >>> --find-links $(dirname >>> external/wheel_pkg/file/wheel-0.36.1.tar.gz) >>> >>> --find-links $(dirname >>> >>> external/setuptools_pkg/file/setuptools-51.0.0-py3-none-any.whl) >>> >>> >>> >>> # Work around setuptools insistance on writing to the source >>> directory, >>> >>> >>> >>> # which is discouraged by Bazel (and annoying) >>> >>> >>> >>> cp -r $(dirname tools/rules/pex/wrapper/setup.py) >>> >>> bazel-out/host/bin/tools/rules/pex/.pex_wrapper >>> >>> >>> >>> # Use the bootstrapped pex to build pex_wrapper.pex >>> >>> >>> >>> pex bazel-out/host/bin/tools/rules/pex/.pex_wrapper >>> >>> --disable-cache >>> >>> --no-index --entry-point=pex_wrapper >>> >>> --output-file=bazel-out/host/bin/tools/rules/pex/pex_wrapper.pex >>> >>> --find-links $(dirname >>> >>> external/pex_pkg/file/pex-2.1.62-py2.py3-none-any.whl) >>> >>> --find-links >>> >>> $(dirname >>> >> external/setuptools_pkg/file/setuptools-51.0.0-py3-none-any.whl) >>> >>> --find-links $(dirname >>> >>> external/requests_pkg/file/requests-2.27.1-py2.py3-none-any.whl) >>> >>> --find-links $(dirname >>> >>> external/charset_pkg/file/charset_normalizer-2.0.10-py3-none-any.whl) >>> >>> --find-links $(dirname >>> >>> external/idna_pkg/file/idna-3.3-py2.py3-none-any.whl) >>> >>> --find-links >>> >>> $(dirname >>> external/urllib3_pkg/file/urllib3-1.26.8-py2.py3-none-any.whl) >>> >>> --find-links $(dirname >>> >>> external/certifi_pkg/file/certifi-2021.10.8-py2.py3-none-any.whl) >>> >>> --find-links $(dirname >>> external/wheel_pkg/file/wheel-0.36.1.tar.gz)') >>> >>> >>> >>> Execution platform: @local_config_platform//:host >>> >>> >>> >>> mktemp: illegal option -- p >>> >>> >>> >>> usage: mktemp [-d] [-q] [-t prefix] [-u] template ... >>> >>> >>> >>> mktemp [-d] [-q] [-u] -t prefix >>> >>> >>> >>> Target //scripts/packages:binpkgs failed to build >>> >>> >>> >>> INFO: Elapsed time: 0.245s, Critical Path: 0.03s >>> >>> >>> >>> INFO: 14 processes: 14 internal. >>> >>> >>> >>> FAILED: Build did NOT complete successfully >>> >>> >>> >>> MacBook-Pro:incubator-heron joshfischer$ python -V >>> >>> >>> >>> Python 3.8.12 >>> >>> >>> >>> On Fri, Feb 25, 2022 at 12:08 PM Saad Ur Rahman < >>> [email protected] >>> >>> >>> >>> wrote: >>> >>> >>> >>>> I just checked the diff on the PR and it is in fact Python 3.8 - I >>> >> stand >>> >>>> corrected. I am not sure about the ASF infra issues but I believe it >>> >> was >>> >>>> related to PEX and Python. The recent PRs should, hopefully, fix the >>> >>> issue. >>> >>>> On Fri, Feb 25, 2022 at 12:09 PM Josh Fischer<[email protected]> >>> >>> wrote: >>> >>>>> Ahh, 3.9, even better. >>> >>>>> >>> >>>>> Random question: did we ever figure out what the build issue was >>> >> when >>> >>>>> building heron in containers on the Apache jenkins instance? >>> >>>>> >>> >>>>> On Fri, Feb 25, 2022 at 11:05 AM Saad Ur Rahman < >>> >>> [email protected] >>> >>>>> wrote: >>> >>>>> >>> >>>>>> It is a drastic improvement, but a minor correction on the base >>> >>> version >>> >>>>> of >>> >>>>>> Python: it is currently 3.9. We really should run a test build on >>> >> the >>> >>>> ASF >>> >>>>>> build infrastructure to see if it builds smoothly. >>> >>>>>> >>> >>>>>> On Fri, Feb 25, 2022 at 11:48 AM Josh Fischer < >>> [email protected] >>> >>>>> wrote: >>> >>>>>>> Fantastic. This helps remove the pressure of the native Python >>> >>> rules >>> >>>>> for >>> >>>>>>> the moment. >>> >>>>>>> >>> >>>>>>> On Fri, Feb 25, 2022 at 10:44 AM Saad Ur Rahman < >>> >>>>> [email protected] >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>>> Correct, the base Python version was bumped up to 3.8. It was >>> >> the >>> >>>>> last >>> >>>>>>>> major PR merged in #3646 ( >>> >>>>>>>> https://github.com/apache/incubator-heron/pull/3646). I >>> >> believe >>> >>>> the >>> >>>>>> base >>> >>>>>>>> Ubuntu container version is now 20.04 LTS. >>> >>>>>>>> >>> >>>>>>>> On Fri, Feb 25, 2022 at 11:36 AM Josh Fischer < >>> >>> [email protected] >>> >>>>>>> wrote: >>> >>>>>>>>> I remember reading somewhere that during a meet up that some >>> >>> work >>> >>>>> was >>> >>>>>>>> done >>> >>>>>>>>> to upgrade the current supported version of Python within the >>> >>>> Heron >>> >>>>>>> code >>> >>>>>>>>> base. Am I remembering this correctly?-- >>> >>>>>>>>> Sent from A Mobile Device >>> >>>>>>>>> >>> >>>>>>> -- >>> >>>>>>> Sent from A Mobile Device >>> >>>>>>> >>> >>>>> -- >>> >>>>> Sent from A Mobile Device >>> >>>>> >> >>
