Thanks for the work! I feel moving away from pex might be ok, as long as the final output can run in the same way. There are quite a few modules in Python: python API, cli, executor, tracker, UI, etc.
It seems this is a blocker to the bazel upgrade? In that case maybe we should focus on python package first. Does any bazel has internal python packaging support? On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <[email protected]> wrote: > Hey Josh, > > I had a look into this/pex/subpar a bit more: > > - external pex rules aren't used (e.g. benley/bazel_rules_pex > <https://github.com/benley/bazel_rules_pex> - although it looks like > the > heron tools/rules/pex/pex_rules.bzl > < > https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl > > > was copied+pasted from the same source at some point - the earliest > copy in > heron doesn't appear in the remote repo so I guess with changes). > Although > the public repo hasn't been touched in months, there isn't an issue > (yet?) > for bazel 0.20.0 > - content of pex files are imported from by the pex_loader module > maintained in heron - I couldn't find the code via google, but maybe > it's > something that could live in pex > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex format > appears stable (or inactive?) so hopefully we'll be ok for now > - pex's design documentation < > https://www.pantsbuild.org/pex_design.html> > doesn't list subpar as an alternative - I made an issue for > clarification > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10> > > > I'm under the impression that switching out pex for par would be quite a > bit of work (and not sure yet if it introduces more limitations), including > a making a par_loader and also deciding on maintaining both for a while or > dropping pex. Moving away from the loaders may be nice so we aren't tied to > a language if possible, but I haven't looked into the how it is used, I > think elsewhere the interface is a program argument/flag and protobuf so > that might be an option. > > I don't know if there's any required customisation within the heron copy of > pex_rules.bzl (I hope not), so it may be worth trying to drop in the public > rules and using those, then at least the rules should be easy to update and > there's a better shot at community support. > > Does using bazel_pex_rules repo sound passable? > > Thanks, > Oliver > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <[email protected]> wrote: > > > I'm open to any suggestions.. I'm not much of a python developer, so my > > knowledge is limited in this part of the project. I appreciate the > > thoughts and ideas. Thanks Oliver. Let us know what you come up with. > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow < > [email protected]> > > wrote: > > > > > Google's subpar may be an option as a replacement for .pex - I imagine > > it'd > > > have better support in Bazel/generally. Has anyone got any thoughts on > > > pex/subpar? I can ponder more and look up stuff next week > > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <[email protected] wrote: > > > > > > > Hey All, > > > > > > > > I've spent little bit of time over the past day or two working to > allow > > > > Heron to be built with newer versions of Bazel (at this time > v-0.20.0). > > > > I've managed to restructure certain files within the Heron to > project > > to > > > > comply with what Bazel is expecting.. I think I have most of it > worked > > > out > > > > with one big exception (I think) of the python pex rules that were > > > written > > > > by the community and seem to be several months old. I do not know > > enough > > > > about PEX to update this custom rule and verify it works as > expected.. > > I > > > am > > > > afraid that if we don't address this issue soon will be stuck with an > > > older > > > > version of Bazel moving forward.. Another thought I had: "is it > > > required > > > > that we use pex"? Is there a way we could get away from it and just > > use > > > > supported official Bazel rules if this issue is too hard to fix or > > > > maintain? > > > > > > > > Here is my output when building > > > > > > > > bazel build --config=darwin heron/... > > > > --incompatible_remove_native_http_archive=false > > > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded > > > > > > > > INFO: Build options have changed, discarding analysis cache. > > > > > > > > ERROR: > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > in eggs attribute of pex_binary rule > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs files > > > > (expected .egg or .whl). Since this rule was created by the macro > > > > 'pex_pytest', the error might have been caused by the macro > > > implementation > > > > in > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > ERROR: > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > in eggs attribute of pex_binary rule > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > '@py_whl//file:file' does not produce any pex_binary eggs files > > (expected > > > > .egg or .whl). Since this rule was created by the macro 'pex_pytest', > > the > > > > error might have been caused by the macro implementation in > > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > ERROR: Analysis of target > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > failed; > > > > build aborted: Analysis of target > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > failed; > > > > build aborted > > > > > > > > ignore the flag --incompatible_remove_native_http_archive=false I'm > > > using > > > > that ignore a warning with http_archive targets temporarily while I > > work > > > > through this python issue. I have a fork of heron at > > > > *https://github.com/joshfischer1108/incubator-heron > > > > <https://github.com/joshfischer1108/incubator-heron>. *The branch I > > am > > > > working off of within that repo is support-bazel-0.20.0 > > > > < > > > > > > > > > > https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0 > > > > >. > > > > If anyone has any experience with python and could possibly look over > > > this > > > > and help me get Heron up to speed with the most recent version of > > Bazel I > > > > would greatly appreciate it.. Just a FYI, as this is still a work in > > > > progress you'll notice I have some duplicate folder structures and > > files > > > in > > > > the repo right now.. This on purpose.. Bazel has made some changes > that > > > > were breaking.. > > > > > > > > > > > > Thanks, > > > > > > > > Josh > > > > > > > > > >
