Last night I updated the Pex rules with what is in the github repository at https://github.com/benley/bazel_rules_pex
I'm getting an error stating that the macro implementation does produce any pex_binary eggs Here is my command and some output: ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: in eggs attribute of pex_binary rule //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: '@pytest_whl//file:file' does not produce any pex_binary eggs files (expected .egg or .whl). Since this rule was created by the macro 'pex_pytest', the error might have been caused by the macro implementation in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14 ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: in eggs attribute of pex_binary rule //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: '@py_whl//file:file' does not produce any pex_binary eggs files (expected .egg or .whl). Since this rule was created by the macro 'pex_pytest', the error might have been caused by the macro implementation in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14 ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: in eggs attribute of pex_binary rule //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: '@setuptools_src//file:file' does not produce any pex_binary eggs files (expected .egg or .whl). Since this rule was created by the macro 'pex_pytest', the error might have been caused by the macro implementation in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14 INFO: SHA256 ( http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2-tests.jar) = d87c8dcfd609085090e5ced0249c5ac064c5d50bda29eb8633794fd91789d163 ERROR: Analysis of target '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed; build aborted: Analysis of target '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed; build aborted Think we should continue down the path of trying to fix this or go a different route? On Wed, Dec 19, 2018 at 2:35 AM thinker0 <[email protected]> wrote: > +1 > > 2018년 12월 19일 (수) 오전 2:42, Ning Wang <[email protected]>님이 작성: > > > +1 > > > > On Tue, Dec 18, 2018 at 9:13 AM Josh Fischer <[email protected]> > wrote: > > > > > Hey Oliver, > > > > > > Thanks for looking into this. I apologize for my bad choice of words > > > describing how the rule was used.. I think you are correct, it looks as > > the > > > rule was copy and pasted. > > > > > > I would prefer to change as little as possible for the Bazel upgrade. > I > > > really wanted someone with more Python experience to point my in the > > right > > > direction (which I think you have done).. > > > > > > Maybe the first thing to do is swap out the parts we need to update the > > > rule.. If that doesn't work then we can investigate moving away from > PEX. > > > Everyone ok with that? > > > > > > -Josh > > > > > > On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <[email protected]> > wrote: > > > > > > > Thanks for the work! > > > > > > > > I feel moving away from pex might be ok, as long as the final output > > can > > > > run in the same way. There are quite a few modules in Python: python > > API, > > > > cli, executor, tracker, UI, etc. > > > > > > > > It seems this is a blocker to the bazel upgrade? In that case maybe > we > > > > should focus on python package first. Does any bazel has internal > > python > > > > packaging support? > > > > > > > > > > > > > > > > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Hey Josh, > > > > > > > > > > I had a look into this/pex/subpar a bit more: > > > > > > > > > > - external pex rules aren't used (e.g. benley/bazel_rules_pex > > > > > <https://github.com/benley/bazel_rules_pex> - although it looks > > > like > > > > > the > > > > > heron tools/rules/pex/pex_rules.bzl > > > > > < > > > > > > > > > > > > > > > https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl > > > > > > > > > > > was copied+pasted from the same source at some point - the > > earliest > > > > > copy in > > > > > heron doesn't appear in the remote repo so I guess with > changes). > > > > > Although > > > > > the public repo hasn't been touched in months, there isn't an > > issue > > > > > (yet?) > > > > > for bazel 0.20.0 > > > > > - content of pex files are imported from by the pex_loader > module > > > > > maintained in heron - I couldn't find the code via google, but > > maybe > > > > > it's > > > > > something that could live in pex > > > > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex > > > format > > > > > appears stable (or inactive?) so hopefully we'll be ok for now > > > > > - pex's design documentation < > > > > > https://www.pantsbuild.org/pex_design.html> > > > > > doesn't list subpar as an alternative - I made an issue for > > > > > clarification > > > > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10> > > > > > > > > > > > > > > > I'm under the impression that switching out pex for par would be > > quite > > > a > > > > > bit of work (and not sure yet if it introduces more limitations), > > > > including > > > > > a making a par_loader and also deciding on maintaining both for a > > while > > > > or > > > > > dropping pex. Moving away from the loaders may be nice so we aren't > > > tied > > > > to > > > > > a language if possible, but I haven't looked into the how it is > > used, I > > > > > think elsewhere the interface is a program argument/flag and > protobuf > > > so > > > > > that might be an option. > > > > > > > > > > I don't know if there's any required customisation within the heron > > > copy > > > > of > > > > > pex_rules.bzl (I hope not), so it may be worth trying to drop in > the > > > > public > > > > > rules and using those, then at least the rules should be easy to > > update > > > > and > > > > > there's a better shot at community support. > > > > > > > > > > Does using bazel_pex_rules repo sound passable? > > > > > > > > > > Thanks, > > > > > Oliver > > > > > > > > > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <[email protected]> > > wrote: > > > > > > > > > > > I'm open to any suggestions.. I'm not much of a python developer, > > so > > > my > > > > > > knowledge is limited in this part of the project. I appreciate > the > > > > > > thoughts and ideas. Thanks Oliver. Let us know what you come up > > > with. > > > > > > > > > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Google's subpar may be an option as a replacement for .pex - I > > > > imagine > > > > > > it'd > > > > > > > have better support in Bazel/generally. Has anyone got any > > thoughts > > > > on > > > > > > > pex/subpar? I can ponder more and look up stuff next week > > > > > > > > > > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <[email protected] > > wrote: > > > > > > > > > > > > > > > Hey All, > > > > > > > > > > > > > > > > I've spent little bit of time over the past day or two > working > > to > > > > > allow > > > > > > > > Heron to be built with newer versions of Bazel (at this time > > > > > v-0.20.0). > > > > > > > > I've managed to restructure certain files within the Heron > to > > > > > project > > > > > > to > > > > > > > > comply with what Bazel is expecting.. I think I have most of > it > > > > > worked > > > > > > > out > > > > > > > > with one big exception (I think) of the python pex rules that > > > were > > > > > > > written > > > > > > > > by the community and seem to be several months old. I do not > > know > > > > > > enough > > > > > > > > about PEX to update this custom rule and verify it works as > > > > > expected.. > > > > > > I > > > > > > > am > > > > > > > > afraid that if we don't address this issue soon will be stuck > > > with > > > > an > > > > > > > older > > > > > > > > version of Bazel moving forward.. Another thought I had: > "is > > it > > > > > > > required > > > > > > > > that we use pex"? Is there a way we could get away from it > and > > > > just > > > > > > use > > > > > > > > supported official Bazel rules if this issue is too hard to > fix > > > or > > > > > > > > maintain? > > > > > > > > > > > > > > > > Here is my output when building > > > > > > > > > > > > > > > > bazel build --config=darwin heron/... > > > > > > > > --incompatible_remove_native_http_archive=false > > > > > > > > > > > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded > > > > > > > > > > > > > > > > INFO: Build options have changed, discarding analysis cache. > > > > > > > > > > > > > > > > ERROR: > > > > > > > > > > > > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > > > > in eggs attribute of pex_binary rule > > > > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs > > > files > > > > > > > > (expected .egg or .whl). Since this rule was created by the > > macro > > > > > > > > 'pex_pytest', the error might have been caused by the macro > > > > > > > implementation > > > > > > > > in > > > > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > > > > > > > ERROR: > > > > > > > > > > > > > > > > > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > > > > > > in eggs attribute of pex_binary rule > > > > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > > > > > > '@py_whl//file:file' does not produce any pex_binary eggs > files > > > > > > (expected > > > > > > > > .egg or .whl). Since this rule was created by the macro > > > > 'pex_pytest', > > > > > > the > > > > > > > > error might have been caused by the macro implementation in > > > > > > > > > > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > > > > > > > > > > > ERROR: Analysis of target > > > > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > > > > failed; > > > > > > > > build aborted: Analysis of target > > > > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' > > > > > failed; > > > > > > > > build aborted > > > > > > > > > > > > > > > > ignore the flag > --incompatible_remove_native_http_archive=false > > > > I'm > > > > > > > using > > > > > > > > that ignore a warning with http_archive targets temporarily > > > while I > > > > > > work > > > > > > > > through this python issue. I have a fork of heron at > > > > > > > > *https://github.com/joshfischer1108/incubator-heron > > > > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The > > > > branch I > > > > > > am > > > > > > > > working off of within that repo is support-bazel-0.20.0 > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0 > > > > > > > > >. > > > > > > > > If anyone has any experience with python and could possibly > > look > > > > over > > > > > > > this > > > > > > > > and help me get Heron up to speed with the most recent > version > > of > > > > > > Bazel I > > > > > > > > would greatly appreciate it.. Just a FYI, as this is still a > > work > > > > in > > > > > > > > progress you'll notice I have some duplicate folder > structures > > > and > > > > > > files > > > > > > > in > > > > > > > > the repo right now.. This on purpose.. Bazel has made some > > > changes > > > > > that > > > > > > > > were breaking.. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Josh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
