+1 On Tue, 18 Dec 2018 at 11:13, Josh Fischer <j...@joshfischer.io> wrote:
> Hey Oliver, > > Thanks for looking into this. I apologize for my bad choice of words > describing how the rule was used.. I think you are correct, it looks as the > rule was copy and pasted. > > I would prefer to change as little as possible for the Bazel upgrade. I > really wanted someone with more Python experience to point my in the right > direction (which I think you have done).. > > Maybe the first thing to do is swap out the parts we need to update the > rule.. If that doesn't work then we can investigate moving away from PEX. > Everyone ok with that? > > -Josh > > On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wangnin...@gmail.com> wrote: > > > Thanks for the work! > > > > I feel moving away from pex might be ok, as long as the final output can > > run in the same way. There are quite a few modules in Python: python API, > > cli, executor, tracker, UI, etc. > > > > It seems this is a blocker to the bazel upgrade? In that case maybe we > > should focus on python package first. Does any bazel has internal python > > packaging support? > > > > > > > > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow < > oli...@oliverbristow.co.uk > > > > > wrote: > > > > > Hey Josh, > > > > > > I had a look into this/pex/subpar a bit more: > > > > > > - external pex rules aren't used (e.g. benley/bazel_rules_pex > > > <https://github.com/benley/bazel_rules_pex> - although it looks > like > > > the > > > heron tools/rules/pex/pex_rules.bzl > > > < > > > > > > https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl > > > > > > > was copied+pasted from the same source at some point - the earliest > > > copy in > > > heron doesn't appear in the remote repo so I guess with changes). > > > Although > > > the public repo hasn't been touched in months, there isn't an issue > > > (yet?) > > > for bazel 0.20.0 > > > - content of pex files are imported from by the pex_loader module > > > maintained in heron - I couldn't find the code via google, but maybe > > > it's > > > something that could live in pex > > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex > format > > > appears stable (or inactive?) so hopefully we'll be ok for now > > > - pex's design documentation < > > > https://www.pantsbuild.org/pex_design.html> > > > doesn't list subpar as an alternative - I made an issue for > > > clarification > > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10> > > > > > > > > > I'm under the impression that switching out pex for par would be quite > a > > > bit of work (and not sure yet if it introduces more limitations), > > including > > > a making a par_loader and also deciding on maintaining both for a while > > or > > > dropping pex. Moving away from the loaders may be nice so we aren't > tied > > to > > > a language if possible, but I haven't looked into the how it is used, I > > > think elsewhere the interface is a program argument/flag and protobuf > so > > > that might be an option. > > > > > > I don't know if there's any required customisation within the heron > copy > > of > > > pex_rules.bzl (I hope not), so it may be worth trying to drop in the > > public > > > rules and using those, then at least the rules should be easy to update > > and > > > there's a better shot at community support. > > > > > > Does using bazel_pex_rules repo sound passable? > > > > > > Thanks, > > > Oliver > > > > > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <j...@joshfischer.io> wrote: > > > > > > > I'm open to any suggestions.. I'm not much of a python developer, so > my > > > > knowledge is limited in this part of the project. I appreciate the > > > > thoughts and ideas. Thanks Oliver. Let us know what you come up > with. > > > > > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow < > > > oli...@oliverbristow.co.uk> > > > > wrote: > > > > > > > > > Google's subpar may be an option as a replacement for .pex - I > > imagine > > > > it'd > > > > > have better support in Bazel/generally. Has anyone got any thoughts > > on > > > > > pex/subpar? I can ponder more and look up stuff next week > > > > > > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <j...@joshfischer.io wrote: > > > > > > > > > > > Hey All, > > > > > > > > > > > > I've spent little bit of time over the past day or two working to > > > allow > > > > > > Heron to be built with newer versions of Bazel (at this time > > > v-0.20.0). > > > > > > I've managed to restructure certain files within the Heron to > > > project > > > > to > > > > > > comply with what Bazel is expecting.. I think I have most of it > > > worked > > > > > out > > > > > > with one big exception (I think) of the python pex rules that > were > > > > > written > > > > > > by the community and seem to be several months old. I do not know > > > > enough > > > > > > about PEX to update this custom rule and verify it works as > > > expected.. > > > > I > > > > > am > > > > > > afraid that if we don't address this issue soon will be stuck > with > > an > > > > > older > > > > > > version of Bazel moving forward.. Another thought I had: "is it > > > > > required > > > > > > that we use pex"? Is there a way we could get away from it and > > just > > > > use > > > > > > supported official Bazel rules if this issue is too hard to fix > or > > > > > > maintain? > > > > > > > > > > > > Here is my output when building > > > > > > > > > > > > bazel build --config=darwin heron/... > > > > > > --incompatible_remove_native_http_archive=false > > > > > > > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded > > > > > > > > > > > > INFO: Build options have changed, discarding analysis cache. > > > > > > > > > > > > ERROR: > > > > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > > in eggs attribute of pex_binary rule > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs > files > > > > > > (expected .egg or .whl). Since this rule was created by the macro > > > > > > 'pex_pytest', the error might have been caused by the macro > > > > > implementation > > > > > > in > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > > > ERROR: > > > > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > > in eggs attribute of pex_binary rule > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > > '@py_whl//file:file' does not produce any pex_binary eggs files > > > > (expected > > > > > > .egg or .whl). Since this rule was created by the macro > > 'pex_pytest', > > > > the > > > > > > error might have been caused by the macro implementation in > > > > > > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > > > ERROR: Analysis of target > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > > failed; > > > > > > build aborted: Analysis of target > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > > failed; > > > > > > build aborted > > > > > > > > > > > > ignore the flag --incompatible_remove_native_http_archive=false > > I'm > > > > > using > > > > > > that ignore a warning with http_archive targets temporarily > while I > > > > work > > > > > > through this python issue. I have a fork of heron at > > > > > > *https://github.com/joshfischer1108/incubator-heron > > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The > > branch I > > > > am > > > > > > working off of within that repo is support-bazel-0.20.0 > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0 > > > > > > >. > > > > > > If anyone has any experience with python and could possibly look > > over > > > > > this > > > > > > and help me get Heron up to speed with the most recent version of > > > > Bazel I > > > > > > would greatly appreciate it.. Just a FYI, as this is still a work > > in > > > > > > progress you'll notice I have some duplicate folder structures > and > > > > files > > > > > in > > > > > > the repo right now.. This on purpose.. Bazel has made some > changes > > > that > > > > > > were breaking.. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Josh > > > > > > > > > > > > > > > > > > > > >
