Agree with everything Robert said. So if we just rebuild the Python zip then this should g2g?
On Wed, Nov 8, 2017 at 12:37 PM, Robert Bradshaw < rober...@google.com.invalid> wrote: > Let me try to clarify the state of the world (with regards to Python > and proto files). > > * When Python setup.py is run, it checks to see if the generated pb2 > files exist. If not, it attempts to generate them by installing the > proto compiler and looking up the .proto definitions in its parent > directory. This works great for the developer that checked out the > full pristine sources from git (or otherwise obtained them). > > * For the sdist tarball uploaded to PyPi (aka Python Artifact), we > ship the generated pb2 files both because (1) we don't want to force > the user to install the proto compiler and (2) the "parent" directory > doesn't exist as we're just shipping the sdks/python/... portion of > the full git repository. > > * All previous "releases" in > https://dist.apache.org/repos/dist/release/beam/ post the Python > artifact (which is Python sources + generated proto files, but notably > no source proto files) in addition to the full source artifact (which > contains some snapshot of the full git repository, Python and proto > files included). We also separately publish Java artifacts offsite > which is what people will install from. > > So it seems the purpose of the -python.zip file is just to stage what > we intend to release on PyPi (e.g. for testing); it is not a source > distribution (that is taken care of by the adjacent -source.zip file) > and so there's no issue with it containing generated files. It should > be the output of "python setup.py sdist" (possibly invoked by the mvn > release commands, if you can get those to work). On the other hand, > creating a separate python-only source distribution would serve no > purpose, as it would be redundant with the existing > everything-source-distribution which is just a manually taken snapshot > of the entire git repository. The confusion is around the role of the > -python.zip file, and if we clarify that it's the proposed Python PyPi > artifact, and *not* some kind of python-only source distribution, the > release process is WAI. > > - Robert > > > On Wed, Nov 8, 2017 at 11:55 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > Let me take a look. Afair I didn't touch those files in last release. > > > > I keep you posted. > > > > Regards > > JB > > > > On Nov 8, 2017, 20:50, at 20:50, Reuven Lax <re...@google.com.INVALID> > wrote: > >>I explicitly removed the pb2 files as I thought we determined they > >>shouldn't be in the source release, and they caused RAT failures. What > >>should I be doing here? > >> > >>On Wed, Nov 8, 2017 at 10:21 AM, Robert Bradshaw < > >>rober...@google.com.invalid> wrote: > >> > >>> This is due to having removed the auto-generated pb2 files. > >>> > >>> On Wed, Nov 8, 2017 at 9:37 AM, Valentyn Tymofieiev > >>> <valen...@google.com.invalid> wrote: > >>> > Confirming Ismaël's finding - I also see this error and it did not > >>see it > >>> > on a candidate that was in the staging area yesterday. > >>> > > >>> > On Wed, Nov 8, 2017 at 9:07 AM, Ismaël Mejía <ieme...@gmail.com> > >>wrote: > >>> > > >>> >> I tested the python version of the release I just created a new > >>> >> virtualenv and run > >>> >> > >>> >> python setup.py install and it gave me this message: > >>> >> > >>> >> Traceback (most recent call last): > >>> >> File "setup.py", line 203, in <module> > >>> >> 'test': generate_protos_first(test), > >>> >> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > >>> >> dist.run_commands() > >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 953, in > >>> run_commands > >>> >> self.run_command(cmd) > >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in > >>run_command > >>> >> cmd_obj.run() > >>> >> File > >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si > >>> >> te-packages/setuptools/command/install.py", > >>> >> line 67, in run > >>> >> self.do_egg_install() > >>> >> File > >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si > >>> >> te-packages/setuptools/command/install.py", > >>> >> line 109, in do_egg_install > >>> >> self.run_command('bdist_egg') > >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in > >>run_command > >>> >> self.distribution.run_command(command) > >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in > >>run_command > >>> >> cmd_obj.run() > >>> >> File > >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si > >>> >> te-packages/setuptools/command/bdist_egg.py", > >>> >> line 169, in run > >>> >> cmd = self.call_command('install_lib', warn_dir=0) > >>> >> File > >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si > >>> >> te-packages/setuptools/command/bdist_egg.py", > >>> >> line 155, in call_command > >>> >> self.run_command(cmdname) > >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in > >>run_command > >>> >> self.distribution.run_command(command) > >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in > >>run_command > >>> >> cmd_obj.run() > >>> >> File > >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si > >>> >> te-packages/setuptools/command/install_lib.py", > >>> >> line 11, in run > >>> >> self.build() > >>> >> File "/usr/lib/python2.7/distutils/command/install_lib.py", line > >>109, > >>> >> in build > >>> >> self.run_command('build_py') > >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in > >>run_command > >>> >> self.distribution.run_command(command) > >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in > >>run_command > >>> >> cmd_obj.run() > >>> >> File "setup.py", line 143, in run > >>> >> gen_protos.generate_proto_files() > >>> >> File > >>"/home/ismael/releases/votes/beam/apache-beam-2.2.0-python/g > >>> >> en_protos.py", > >>> >> line 66, in generate_proto_files > >>> >> 'Not in apache git tree; unable to find proto definitions.') > >>> >> RuntimeError: Not in apache git tree; unable to find proto > >>definitions. > >>> >> > >>> >> Not sure if this is something in my environment, but this passed > >>when > >>> >> I validated the previous release (2.1.0). > >>> >> > >>> >> > >>> >> On Wed, Nov 8, 2017 at 11:30 AM, Reuven Lax > >><re...@google.com.invalid> > >>> >> wrote: > >>> >> > Hi everyone, > >>> >> > > >>> >> > Please review and vote on the release candidate #2 for the > >>version > >>> 2.2.0, > >>> >> > as follows: > >>> >> > [ ] +1, Approve the release > >>> >> > [ ] -1, Do not approve the release (please provide specific > >>> comments) > >>> >> > > >>> >> > > >>> >> > The complete staging area is available for your review, which > >>> includes: > >>> >> > * JIRA release notes [1], > >>> >> > * the official Apache source release to be deployed to > >>> dist.apache.org > >>> >> [2], > >>> >> > which is signed with the key with fingerprint B98B7708 [3], > >>> >> > * all artifacts to be deployed to the Maven Central Repository > >>[4], > >>> >> > * source code tag "v2.2.0-RC3" [5], > >>> >> > * website pull request listing the release and publishing the > >>API > >>> >> > reference manual [6]. > >>> >> > * Java artifacts were built with Maven 3.5.0 and > >>OpenJDK/Oracle JDK > >>> >> > 1.8.0_144. > >>> >> > * Python artifacts are deployed along with the source release > >>to the > >>> >> > dist.apache.org [2]. > >>> >> > > >>> >> > The vote will be open for at least 72 hours. It is adopted by > >>majority > >>> >> > approval, with at least 3 PMC affirmative votes. > >>> >> > > >>> >> > Thanks, > >>> >> > Reuven > >>> >> > > >>> >> > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > >>> >> > projectId=12319527&version=12341044 > >>> >> > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/ > >>> >> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS > >>> >> > [4] https://repository.apache.org/content/repositories/orgapache > >>> >> beam-1023/ > >>> >> > <https://repository.apache.org/content/repositories/ > >>> orgapachebeam-1022/> > >>> >> > [5] https://github.com/apache/beam/tree/v2.2.0-RC > >>> >> > <https://github.com/apache/beam/tree/v2.2.0-RC2>3 > >>> >> > [6] https://github.com/apache/beam-site/pull/337 > >>> >> > >>> >
