On 11 Jan 2023, at 16:53, Ritesh Ghorse via dev
<dev@beam.apache.org> wrote:
+1 (non-binding)
Validated Go Dataframe Transform wrapper on Dataflow runner and
Go SDK quickstart on Direct and Dataflow Runner.
Thanks!
On Wed, Jan 11, 2023 at 12:51 AM Anand Inguva via dev
<dev@beam.apache.org> wrote:
I ran the Python word count on DirectRunner and Dataflow Runner.
Steps:
1. pip install --pre apache_beam in a fresh virtualenv.
2. Run the command Ahmet provided except removing the
sdk_location from CMD args.
The job was successful. <image.png>
On Tue, Jan 10, 2023 at 6:48 PM Ahmet Altay via dev
<dev@beam.apache.org> wrote:
I validated python quick starts (direct, dataflow) X
(batch, streaming). I ran into an issue with the dataflow
batch case, running the wordcount with the standard:
python -m apache_beam.examples.wordcount \
--output <GCS> \
--staging_location <GCS> \
--temp_location <GCS>\
--runner DataflowRunner \
--job_name wordcount-$USER \
--project <PROJECT> \
--num_workers 1 \
--region us-central1 \
--sdk_location apache-beam-2.44.0.zip
results in:
"/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py",
line 589, in __enter__ raise
RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE)
RuntimeError: This pipeline requires Dataflow Runner v2
in order to run with currently used version of Apache
Beam on Python 3.10+. Please verify that the Dataflow
Runner v2 is not disabled in the pipeline options or
enable it explicitly via:
--dataflow_service_option=use_runner_v2. Alternatively,
downgrade to Python 3.9 to use Dataflow Runner v1.
Questions:
- I am not explicitly opting out of runner v2, and this
is a standard wordcount example, I expected it to just work.
Then I tried to
add --dataflow_service_option=use_runner_v2 to the above
wordcount command, which results in the following error:
"message": "Dataflow Runner v2 requires a valid FnApi
job, Please resubmit your job with a valid configuration.
Note that if using Templates, you may need to regenerate
your template with the '--use_runner_v2'."
Maybe I am doing something wrong and it is an error on my
end. It would be good for someone else with python
experience to check this.
/cc @Valentyn Tymofieiev <mailto:valen...@google.com>
Ahmet
On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles
<k...@apache.org> wrote:
I have published a new maven staging repository:
https://repository.apache.org/content/repositories/orgapachebeam-1290/
It looks like it has everything, though I did not
automate a check. At least there were no errors
during publish which I ran with --no-parallel
overnight, and some specific things that were missing
from orgapachebeam-1289 are present.
I will restart the 72 hour waiting period, since the
RC is only now usable.
Kenn
On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles
<k...@apache.org> wrote:
I have discovered that many pom files are missing
from the nexus repository. I should be able to
re-publish a new one. It will take some time as
this is one of the longest-running processes.
On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles
<k...@apache.org> wrote:
Correction: this is release candidate #1.
On Mon, Jan 9, 2023 at 1:25 PM Kenneth
Knowles <k...@apache.org> wrote:
Hi everyone,
Please review and vote on the release
candidate #3 for the version 2.44.0, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release
(please provide specific comments)
Reviewers are encouraged to test their
own use cases with the release candidate,
and vote +1 if
no issues are found.
The complete staging area is available
for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to
be deployed to dist.apache.org
<http://dist.apache.org/> [2], which is
signed with the key with fingerprint
6ED551A8AE02461C [3],
* all artifacts to be deployed to the
Maven Central Repository [4],
* source code tag "v2.44.0-RC1" [5],
* website pull request listing the
release [6], the blog post [6], and
publishing the API reference manual [7].
* Java artifacts were built with Gradle
7.5.1 and OpenJDK 1.8.0_232.
* Python artifacts are deployed along
with the source release to the
dist.apache.org <http://dist.apache.org/>
[2] and PyPI [8].
* Go artifacts and documentation are
available at pkg.go.dev
<http://pkg.go.dev/> [9] (waiting on
these to appear)
* Validation sheet with a tab for 2.44.0
release to help with validation [10].
* Docker images published to Docker Hub [11].
The vote will be open for at least 72
hours. It is adopted by majority
approval, with at least 3 PMC affirmative
votes.
For guidelines on how to try the release
in your projects, check out our blog post
at /blog/validate-beam-release/.
Thanks,
Kenn
[1]
https://github.com/apache/beam/milestone/7
[2]
https://dist.apache.org/repos/dist/dev/beam/2.44.0/
[3]
https://dist.apache.org/repos/dist/release/beam/KEYS
[4]
https://repository.apache.org/content/repositories/orgapachebeam-1289/
[5]
https://github.com/apache/beam/tree/v2.44.0-RC1
[6] https://github.com/apache/beam/pull/24951
[7]
https://github.com/apache/beam-site/pull/638
[8]
https://pypi.org/project/apache-beam/2.44.0rc1/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910
[11]
https://hub.docker.com/search?q=apache%2Fbeam&type=image
<https://hub.docker.com/search?q=apache%2Fbeam&type=image>
