Let me try to clarify the state of the world (with regards to Python and proto files).
* When Python setup.py is run, it checks to see if the generated pb2 files exist. If not, it attempts to generate them by installing the proto compiler and looking up the .proto definitions in its parent directory. This works great for the developer that checked out the full pristine sources from git (or otherwise obtained them). * For the sdist tarball uploaded to PyPi (aka Python Artifact), we ship the generated pb2 files both because (1) we don't want to force the user to install the proto compiler and (2) the "parent" directory doesn't exist as we're just shipping the sdks/python/... portion of the full git repository. * All previous "releases" in https://dist.apache.org/repos/dist/release/beam/ post the Python artifact (which is Python sources + generated proto files, but notably no source proto files) in addition to the full source artifact (which contains some snapshot of the full git repository, Python and proto files included). We also separately publish Java artifacts offsite which is what people will install from. So it seems the purpose of the -python.zip file is just to stage what we intend to release on PyPi (e.g. for testing); it is not a source distribution (that is taken care of by the adjacent -source.zip file) and so there's no issue with it containing generated files. It should be the output of "python setup.py sdist" (possibly invoked by the mvn release commands, if you can get those to work). On the other hand, creating a separate python-only source distribution would serve no purpose, as it would be redundant with the existing everything-source-distribution which is just a manually taken snapshot of the entire git repository. The confusion is around the role of the -python.zip file, and if we clarify that it's the proposed Python PyPi artifact, and *not* some kind of python-only source distribution, the release process is WAI. - Robert On Wed, Nov 8, 2017 at 11:55 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Let me take a look. Afair I didn't touch those files in last release. > > I keep you posted. > > Regards > JB > > On Nov 8, 2017, 20:50, at 20:50, Reuven Lax <re...@google.com.INVALID> wrote: >>I explicitly removed the pb2 files as I thought we determined they >>shouldn't be in the source release, and they caused RAT failures. What >>should I be doing here? >> >>On Wed, Nov 8, 2017 at 10:21 AM, Robert Bradshaw < >>rober...@google.com.invalid> wrote: >> >>> This is due to having removed the auto-generated pb2 files. >>> >>> On Wed, Nov 8, 2017 at 9:37 AM, Valentyn Tymofieiev >>> <valen...@google.com.invalid> wrote: >>> > Confirming Ismaël's finding - I also see this error and it did not >>see it >>> > on a candidate that was in the staging area yesterday. >>> > >>> > On Wed, Nov 8, 2017 at 9:07 AM, Ismaël Mejía <ieme...@gmail.com> >>wrote: >>> > >>> >> I tested the python version of the release I just created a new >>> >> virtualenv and run >>> >> >>> >> python setup.py install and it gave me this message: >>> >> >>> >> Traceback (most recent call last): >>> >> File "setup.py", line 203, in <module> >>> >> 'test': generate_protos_first(test), >>> >> File "/usr/lib/python2.7/distutils/core.py", line 151, in setup >>> >> dist.run_commands() >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 953, in >>> run_commands >>> >> self.run_command(cmd) >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in >>run_command >>> >> cmd_obj.run() >>> >> File >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si >>> >> te-packages/setuptools/command/install.py", >>> >> line 67, in run >>> >> self.do_egg_install() >>> >> File >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si >>> >> te-packages/setuptools/command/install.py", >>> >> line 109, in do_egg_install >>> >> self.run_command('bdist_egg') >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in >>run_command >>> >> self.distribution.run_command(command) >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in >>run_command >>> >> cmd_obj.run() >>> >> File >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si >>> >> te-packages/setuptools/command/bdist_egg.py", >>> >> line 169, in run >>> >> cmd = self.call_command('install_lib', warn_dir=0) >>> >> File >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si >>> >> te-packages/setuptools/command/bdist_egg.py", >>> >> line 155, in call_command >>> >> self.run_command(cmdname) >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in >>run_command >>> >> self.distribution.run_command(command) >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in >>run_command >>> >> cmd_obj.run() >>> >> File >>"/home/ismael/.virtualenvs/beam-vote2/local/lib/python2.7/si >>> >> te-packages/setuptools/command/install_lib.py", >>> >> line 11, in run >>> >> self.build() >>> >> File "/usr/lib/python2.7/distutils/command/install_lib.py", line >>109, >>> >> in build >>> >> self.run_command('build_py') >>> >> File "/usr/lib/python2.7/distutils/cmd.py", line 326, in >>run_command >>> >> self.distribution.run_command(command) >>> >> File "/usr/lib/python2.7/distutils/dist.py", line 972, in >>run_command >>> >> cmd_obj.run() >>> >> File "setup.py", line 143, in run >>> >> gen_protos.generate_proto_files() >>> >> File >>"/home/ismael/releases/votes/beam/apache-beam-2.2.0-python/g >>> >> en_protos.py", >>> >> line 66, in generate_proto_files >>> >> 'Not in apache git tree; unable to find proto definitions.') >>> >> RuntimeError: Not in apache git tree; unable to find proto >>definitions. >>> >> >>> >> Not sure if this is something in my environment, but this passed >>when >>> >> I validated the previous release (2.1.0). >>> >> >>> >> >>> >> On Wed, Nov 8, 2017 at 11:30 AM, Reuven Lax >><re...@google.com.invalid> >>> >> wrote: >>> >> > Hi everyone, >>> >> > >>> >> > Please review and vote on the release candidate #2 for the >>version >>> 2.2.0, >>> >> > as follows: >>> >> > [ ] +1, Approve the release >>> >> > [ ] -1, Do not approve the release (please provide specific >>> comments) >>> >> > >>> >> > >>> >> > The complete staging area is available for your review, which >>> includes: >>> >> > * JIRA release notes [1], >>> >> > * the official Apache source release to be deployed to >>> dist.apache.org >>> >> [2], >>> >> > which is signed with the key with fingerprint B98B7708 [3], >>> >> > * all artifacts to be deployed to the Maven Central Repository >>[4], >>> >> > * source code tag "v2.2.0-RC3" [5], >>> >> > * website pull request listing the release and publishing the >>API >>> >> > reference manual [6]. >>> >> > * Java artifacts were built with Maven 3.5.0 and >>OpenJDK/Oracle JDK >>> >> > 1.8.0_144. >>> >> > * Python artifacts are deployed along with the source release >>to the >>> >> > dist.apache.org [2]. >>> >> > >>> >> > The vote will be open for at least 72 hours. It is adopted by >>majority >>> >> > approval, with at least 3 PMC affirmative votes. >>> >> > >>> >> > Thanks, >>> >> > Reuven >>> >> > >>> >> > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? >>> >> > projectId=12319527&version=12341044 >>> >> > [2] https://dist.apache.org/repos/dist/dev/beam/2.2.0/ >>> >> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>> >> > [4] https://repository.apache.org/content/repositories/orgapache >>> >> beam-1023/ >>> >> > <https://repository.apache.org/content/repositories/ >>> orgapachebeam-1022/> >>> >> > [5] https://github.com/apache/beam/tree/v2.2.0-RC >>> >> > <https://github.com/apache/beam/tree/v2.2.0-RC2>3 >>> >> > [6] https://github.com/apache/beam-site/pull/337 >>> >> >>>
