+1 (binding) - my concerns are addressed. On Wed, Jan 11, 2023 at 3:44 PM Valentyn Tymofieiev <valen...@google.com> wrote:
> +1. I validated that Dataflow and Beam Python containers include necessary > dependencies of Apache Beam and did additional validation (see inline). > > On Wed, Jan 11, 2023 at 12:48 AM Ahmet Altay <al...@google.com> wrote: > >> I validated python quick starts (direct, dataflow) X (batch, streaming). >> I ran into an issue with the dataflow batch case, running the wordcount >> with the standard: >> >> python -m apache_beam.examples.wordcount \ >> --output <GCS> \ >> --staging_location <GCS> \ >> --temp_location <GCS>\ >> --runner DataflowRunner \ >> --job_name wordcount-$USER \ >> --project <PROJECT> \ >> --num_workers 1 \ >> --region us-central1 \ >> --sdk_location apache-beam-2.44.0.zip >> >> results in: >> >> "/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py", >> line 589, in __enter__ raise >> RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE) RuntimeError: This pipeline >> requires Dataflow Runner v2 in order to run with currently used version of >> Apache Beam on Python 3.10+. Please verify that the Dataflow Runner v2 is >> not disabled in the pipeline options or enable it explicitly via: >> --dataflow_service_option=use_runner_v2. Alternatively, downgrade to Python >> 3.9 to use Dataflow Runner v1. >> >> Questions: >> - I am not explicitly opting out of runner v2, and this is a standard >> wordcount example, I expected it to just work. >> > You are most likely using a google-internal project for which Runner v2 is > explicitly disabled, to enable Runner v1 test coverage within Google. I > can repro this error as well (maybe on the same project as you), but don't > repro it on other projects, such as apache-beam-testing. Runner v1 is not > supported on Python 3.10 (this is documented). > > Such behavior is WAI as far as Beam is concerned and the difference is due > to configuration details in Dataflow. > Yes, I was using a Google owned project for testing. That makes sense. And it is fine as long as it is not a user visible behavior. Thank you for the clarification. > > > >> >> Then I tried to add --dataflow_service_option=use_runner_v2 to the above >> wordcount command, which results in the following error: >> >> "message": "Dataflow Runner v2 requires a valid FnApi job, Please >> resubmit your job with a valid configuration. Note that if using Templates, >> you may need to regenerate your template with the '--use_runner_v2'." >> >> Maybe I am doing something wrong and it is an error on my end. It would >> be good for someone else with python experience to check this. >> >> /cc @Valentyn Tymofieiev <valen...@google.com> >> >> Ahmet >> >> >> >> >> On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> I have published a new maven staging repository: >>> https://repository.apache.org/content/repositories/orgapachebeam-1290/ >>> >>> It looks like it has everything, though I did not automate a check. At >>> least there were no errors during publish which I ran with --no-parallel >>> overnight, and some specific things that were missing from >>> orgapachebeam-1289 are present. >>> >>> I will restart the 72 hour waiting period, since the RC is only now >>> usable. >>> >>> Kenn >>> >>> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> I have discovered that many pom files are missing from the nexus >>>> repository. I should be able to re-publish a new one. It will take some >>>> time as this is one of the longest-running processes. >>>> >>>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles <k...@apache.org> wrote: >>>> >>>>> Correction: this is release candidate #1. >>>>> >>>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles <k...@apache.org> >>>>> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> Please review and vote on the release candidate #3 for the version >>>>>> 2.44.0, as follows: >>>>>> [ ] +1, Approve the release >>>>>> [ ] -1, Do not approve the release (please provide specific comments) >>>>>> >>>>>> Reviewers are encouraged to test their own use cases with the release >>>>>> candidate, and vote +1 if >>>>>> no issues are found. >>>>>> >>>>>> The complete staging area is available for your review, which >>>>>> includes: >>>>>> * GitHub Release notes [1], >>>>>> * the official Apache source release to be deployed to >>>>>> dist.apache.org [2], which is signed with the key with fingerprint >>>>>> 6ED551A8AE02461C [3], >>>>>> * all artifacts to be deployed to the Maven Central Repository [4], >>>>>> * source code tag "v2.44.0-RC1" [5], >>>>>> * website pull request listing the release [6], the blog post [6], >>>>>> and publishing the API reference manual [7]. >>>>>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_232. >>>>>> * Python artifacts are deployed along with the source release to the >>>>>> dist.apache.org [2] and PyPI [8]. >>>>>> * Go artifacts and documentation are available at pkg.go.dev [9] >>>>>> (waiting on these to appear) >>>>>> * Validation sheet with a tab for 2.44.0 release to help with >>>>>> validation [10]. >>>>>> * Docker images published to Docker Hub [11]. >>>>>> >>>>>> The vote will be open for at least 72 hours. It is adopted by >>>>>> majority approval, with at least 3 PMC affirmative votes. >>>>>> >>>>>> For guidelines on how to try the release in your projects, check out >>>>>> our blog post at /blog/validate-beam-release/. >>>>>> >>>>>> Thanks, >>>>>> Kenn >>>>>> >>>>>> [1] https://github.com/apache/beam/milestone/7 >>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.44.0/ >>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>> [4] >>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1289/ >>>>>> [5] https://github.com/apache/beam/tree/v2.44.0-RC1 >>>>>> [6] https://github.com/apache/beam/pull/24951 >>>>>> [7] https://github.com/apache/beam-site/pull/638 >>>>>> [8] https://pypi.org/project/apache-beam/2.44.0rc1/ >>>>>> [9] >>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam >>>>>> [10] >>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910 >>>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image >>>>>> >>>>>
