Hey Oliver, Thanks for looking into this. I apologize for my bad choice of words describing how the rule was used.. I think you are correct, it looks as the rule was copy and pasted.
I would prefer to change as little as possible for the Bazel upgrade. I really wanted someone with more Python experience to point my in the right direction (which I think you have done).. Maybe the first thing to do is swap out the parts we need to update the rule.. If that doesn't work then we can investigate moving away from PEX. Everyone ok with that? -Josh On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <[email protected]> wrote: > Thanks for the work! > > I feel moving away from pex might be ok, as long as the final output can > run in the same way. There are quite a few modules in Python: python API, > cli, executor, tracker, UI, etc. > > It seems this is a blocker to the bazel upgrade? In that case maybe we > should focus on python package first. Does any bazel has internal python > packaging support? > > > > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <[email protected] > > > wrote: > > > Hey Josh, > > > > I had a look into this/pex/subpar a bit more: > > > > - external pex rules aren't used (e.g. benley/bazel_rules_pex > > <https://github.com/benley/bazel_rules_pex> - although it looks like > > the > > heron tools/rules/pex/pex_rules.bzl > > < > > > https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl > > > > > was copied+pasted from the same source at some point - the earliest > > copy in > > heron doesn't appear in the remote repo so I guess with changes). > > Although > > the public repo hasn't been touched in months, there isn't an issue > > (yet?) > > for bazel 0.20.0 > > - content of pex files are imported from by the pex_loader module > > maintained in heron - I couldn't find the code via google, but maybe > > it's > > something that could live in pex > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex format > > appears stable (or inactive?) so hopefully we'll be ok for now > > - pex's design documentation < > > https://www.pantsbuild.org/pex_design.html> > > doesn't list subpar as an alternative - I made an issue for > > clarification > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10> > > > > > > I'm under the impression that switching out pex for par would be quite a > > bit of work (and not sure yet if it introduces more limitations), > including > > a making a par_loader and also deciding on maintaining both for a while > or > > dropping pex. Moving away from the loaders may be nice so we aren't tied > to > > a language if possible, but I haven't looked into the how it is used, I > > think elsewhere the interface is a program argument/flag and protobuf so > > that might be an option. > > > > I don't know if there's any required customisation within the heron copy > of > > pex_rules.bzl (I hope not), so it may be worth trying to drop in the > public > > rules and using those, then at least the rules should be easy to update > and > > there's a better shot at community support. > > > > Does using bazel_pex_rules repo sound passable? > > > > Thanks, > > Oliver > > > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <[email protected]> wrote: > > > > > I'm open to any suggestions.. I'm not much of a python developer, so my > > > knowledge is limited in this part of the project. I appreciate the > > > thoughts and ideas. Thanks Oliver. Let us know what you come up with. > > > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow < > > [email protected]> > > > wrote: > > > > > > > Google's subpar may be an option as a replacement for .pex - I > imagine > > > it'd > > > > have better support in Bazel/generally. Has anyone got any thoughts > on > > > > pex/subpar? I can ponder more and look up stuff next week > > > > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <[email protected] wrote: > > > > > > > > > Hey All, > > > > > > > > > > I've spent little bit of time over the past day or two working to > > allow > > > > > Heron to be built with newer versions of Bazel (at this time > > v-0.20.0). > > > > > I've managed to restructure certain files within the Heron to > > project > > > to > > > > > comply with what Bazel is expecting.. I think I have most of it > > worked > > > > out > > > > > with one big exception (I think) of the python pex rules that were > > > > written > > > > > by the community and seem to be several months old. I do not know > > > enough > > > > > about PEX to update this custom rule and verify it works as > > expected.. > > > I > > > > am > > > > > afraid that if we don't address this issue soon will be stuck with > an > > > > older > > > > > version of Bazel moving forward.. Another thought I had: "is it > > > > required > > > > > that we use pex"? Is there a way we could get away from it and > just > > > use > > > > > supported official Bazel rules if this issue is too hard to fix or > > > > > maintain? > > > > > > > > > > Here is my output when building > > > > > > > > > > bazel build --config=darwin heron/... > > > > > --incompatible_remove_native_http_archive=false > > > > > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded > > > > > > > > > > INFO: Build options have changed, discarding analysis cache. > > > > > > > > > > ERROR: > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > in eggs attribute of pex_binary rule > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs files > > > > > (expected .egg or .whl). Since this rule was created by the macro > > > > > 'pex_pytest', the error might have been caused by the macro > > > > implementation > > > > > in > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > ERROR: > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > in eggs attribute of pex_binary rule > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > '@py_whl//file:file' does not produce any pex_binary eggs files > > > (expected > > > > > .egg or .whl). Since this rule was created by the macro > 'pex_pytest', > > > the > > > > > error might have been caused by the macro implementation in > > > > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > ERROR: Analysis of target > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > failed; > > > > > build aborted: Analysis of target > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > failed; > > > > > build aborted > > > > > > > > > > ignore the flag --incompatible_remove_native_http_archive=false > I'm > > > > using > > > > > that ignore a warning with http_archive targets temporarily while I > > > work > > > > > through this python issue. I have a fork of heron at > > > > > *https://github.com/joshfischer1108/incubator-heron > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The > branch I > > > am > > > > > working off of within that repo is support-bazel-0.20.0 > > > > > < > > > > > > > > > > > > > > > https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0 > > > > > >. > > > > > If anyone has any experience with python and could possibly look > over > > > > this > > > > > and help me get Heron up to speed with the most recent version of > > > Bazel I > > > > > would greatly appreciate it.. Just a FYI, as this is still a work > in > > > > > progress you'll notice I have some duplicate folder structures and > > > files > > > > in > > > > > the repo right now.. This on purpose.. Bazel has made some changes > > that > > > > > were breaking.. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Josh > > > > > > > > > > > > > > >
