+1 (binding) I ran another battery of new IO jobs that were not included in mass_comment.py and I ran them against the RC1 commit itself rather than just the release branch (only difference between version string, which alters containers pulled in at least some cases)
On Wed, Jan 11, 2023 at 3:47 PM Ahmet Altay <al...@google.com> wrote: > +1 (binding) - my concerns are addressed. > > On Wed, Jan 11, 2023 at 3:44 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> +1. I validated that Dataflow and Beam Python containers include >> necessary dependencies of Apache Beam and did additional validation (see >> inline). >> >> On Wed, Jan 11, 2023 at 12:48 AM Ahmet Altay <al...@google.com> wrote: >> >>> I validated python quick starts (direct, dataflow) X (batch, streaming). >>> I ran into an issue with the dataflow batch case, running the wordcount >>> with the standard: >>> >>> python -m apache_beam.examples.wordcount \ >>> --output <GCS> \ >>> --staging_location <GCS> \ >>> --temp_location <GCS>\ >>> --runner DataflowRunner \ >>> --job_name wordcount-$USER \ >>> --project <PROJECT> \ >>> --num_workers 1 \ >>> --region us-central1 \ >>> --sdk_location apache-beam-2.44.0.zip >>> >>> results in: >>> >>> "/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py", >>> line 589, in __enter__ raise >>> RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE) RuntimeError: This pipeline >>> requires Dataflow Runner v2 in order to run with currently used version of >>> Apache Beam on Python 3.10+. Please verify that the Dataflow Runner v2 is >>> not disabled in the pipeline options or enable it explicitly via: >>> --dataflow_service_option=use_runner_v2. Alternatively, downgrade to Python >>> 3.9 to use Dataflow Runner v1. >>> >>> Questions: >>> - I am not explicitly opting out of runner v2, and this is a standard >>> wordcount example, I expected it to just work. >>> >> You are most likely using a google-internal project for which Runner v2 >> is explicitly disabled, to enable Runner v1 test coverage within Google. I >> can repro this error as well (maybe on the same project as you), but don't >> repro it on other projects, such as apache-beam-testing. Runner v1 is not >> supported on Python 3.10 (this is documented). >> >> Such behavior is WAI as far as Beam is concerned and the difference is >> due to configuration details in Dataflow. >> > > Yes, I was using a Google owned project for testing. That makes sense. And > it is fine as long as it is not a user visible behavior. Thank you for the > clarification. > >> >> >> >>> >>> Then I tried to add --dataflow_service_option=use_runner_v2 to the above >>> wordcount command, which results in the following error: >>> >>> "message": "Dataflow Runner v2 requires a valid FnApi job, Please >>> resubmit your job with a valid configuration. Note that if using Templates, >>> you may need to regenerate your template with the '--use_runner_v2'." >>> >>> Maybe I am doing something wrong and it is an error on my end. It would >>> be good for someone else with python experience to check this. >>> >>> /cc @Valentyn Tymofieiev <valen...@google.com> >>> >>> Ahmet >>> >>> >>> >>> >>> On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles <k...@apache.org> >>> wrote: >>> >>>> I have published a new maven staging repository: >>>> https://repository.apache.org/content/repositories/orgapachebeam-1290/ >>>> >>>> It looks like it has everything, though I did not automate a check. At >>>> least there were no errors during publish which I ran with --no-parallel >>>> overnight, and some specific things that were missing from >>>> orgapachebeam-1289 are present. >>>> >>>> I will restart the 72 hour waiting period, since the RC is only now >>>> usable. >>>> >>>> Kenn >>>> >>>> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles <k...@apache.org> wrote: >>>> >>>>> I have discovered that many pom files are missing from the nexus >>>>> repository. I should be able to re-publish a new one. It will take some >>>>> time as this is one of the longest-running processes. >>>>> >>>>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles <k...@apache.org> >>>>> wrote: >>>>> >>>>>> Correction: this is release candidate #1. >>>>>> >>>>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles <k...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> Please review and vote on the release candidate #3 for the version >>>>>>> 2.44.0, as follows: >>>>>>> [ ] +1, Approve the release >>>>>>> [ ] -1, Do not approve the release (please provide specific comments) >>>>>>> >>>>>>> Reviewers are encouraged to test their own use cases with the >>>>>>> release candidate, and vote +1 if >>>>>>> no issues are found. >>>>>>> >>>>>>> The complete staging area is available for your review, which >>>>>>> includes: >>>>>>> * GitHub Release notes [1], >>>>>>> * the official Apache source release to be deployed to >>>>>>> dist.apache.org [2], which is signed with the key with fingerprint >>>>>>> 6ED551A8AE02461C [3], >>>>>>> * all artifacts to be deployed to the Maven Central Repository [4], >>>>>>> * source code tag "v2.44.0-RC1" [5], >>>>>>> * website pull request listing the release [6], the blog post [6], >>>>>>> and publishing the API reference manual [7]. >>>>>>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_232. >>>>>>> * Python artifacts are deployed along with the source release to the >>>>>>> dist.apache.org [2] and PyPI [8]. >>>>>>> * Go artifacts and documentation are available at pkg.go.dev [9] >>>>>>> (waiting on these to appear) >>>>>>> * Validation sheet with a tab for 2.44.0 release to help with >>>>>>> validation [10]. >>>>>>> * Docker images published to Docker Hub [11]. >>>>>>> >>>>>>> The vote will be open for at least 72 hours. It is adopted by >>>>>>> majority approval, with at least 3 PMC affirmative votes. >>>>>>> >>>>>>> For guidelines on how to try the release in your projects, check out >>>>>>> our blog post at /blog/validate-beam-release/. >>>>>>> >>>>>>> Thanks, >>>>>>> Kenn >>>>>>> >>>>>>> [1] https://github.com/apache/beam/milestone/7 >>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.44.0/ >>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>>> [4] >>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1289/ >>>>>>> [5] https://github.com/apache/beam/tree/v2.44.0-RC1 >>>>>>> [6] https://github.com/apache/beam/pull/24951 >>>>>>> [7] https://github.com/apache/beam-site/pull/638 >>>>>>> [8] https://pypi.org/project/apache-beam/2.44.0rc1/ >>>>>>> [9] >>>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam >>>>>>> [10] >>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910 >>>>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image >>>>>>> >>>>>>
