from:"tvalentyn"

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 93a5bc76577 [Python] Use retry decorator for retrying (#31273)
93a5bc76577 is described below

commit 93a5bc765773c168ada907ba78cd0d04b1c5e390
Author: Ritesh Ghorse 
AuthorDate: Thu May 16 20:33:17 2024 -0400

[Python] Use retry decorator for retrying (#31273)
---
 sdks/python/apache_beam/io/requestresponse_test.py | 61 ++
 1 file changed, 29 insertions(+), 32 deletions(-)

diff --git a/sdks/python/apache_beam/io/requestresponse_test.py 
b/sdks/python/apache_beam/io/requestresponse_test.py
index a4427bcefa2..3bc85a5e103 100644
--- a/sdks/python/apache_beam/io/requestresponse_test.py
+++ b/sdks/python/apache_beam/io/requestresponse_test.py
@@ -18,6 +18,10 @@ import logging
 import time
 import unittest
 
+from tenacity import retry
+from tenacity import retry_if_exception_type
+from tenacity import stop_after_attempt
+
 import apache_beam as beam
 from apache_beam.testing.test_pipeline import TestPipeline
 
@@ -134,39 +138,32 @@ class TestCaller(unittest.TestCase):
 | RequestResponseIO(caller=caller, repeater=None))
 self.assertRegex(cm.exception.message, 'retries = 0')
 
+  @retry(
+  retry=retry_if_exception_type(IndexError),
+  reraise=True,
+  stop=stop_after_attempt(MAX_TEST_RETRIES))
   def test_default_throttler(self):
-for i in range(MAX_TEST_RETRIES):
-  try:
-caller = CallerWithTimeout()
-throttler = DefaultThrottler(
-window_ms=1, bucket_ms=5000, overload_ratio=1)
-# manually override the number of received requests for testing.
-throttler.throttler._all_requests.add(time.time() * 1000, 100)
-test_pipeline = TestPipeline()
-_ = (
-test_pipeline
-| beam.Create(['sample_request'])
-| RequestResponseIO(caller=caller, throttler=throttler))
-result = test_pipeline.run()
-result.wait_until_finish()
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name('throttled_requests'))
-self.assertEqual(metrics['counters'][0].committed, 1)
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name(
-'cumulativeThrottlingSeconds'))
-self.assertGreater(metrics['counters'][0].committed, 0)
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name('responses'))
-self.assertEqual(metrics['counters'][0].committed, 1)
-  except IndexError:
-if i != MAX_TEST_RETRIES - 1:
-  _LOGGER.warning(
-  'Retry %s: Metrics were expected but not found' % i + 1)
-  continue
-raise IndexError('List index out of range. Metrics not found.')
-  else:
-break
+caller = CallerWithTimeout()
+throttler = DefaultThrottler(
+window_ms=1, bucket_ms=5000, overload_ratio=1)
+# manually override the number of received requests for testing.
+throttler.throttler._all_requests.add(time.time() * 1000, 100)
+test_pipeline = TestPipeline()
+_ = (
+test_pipeline
+| beam.Create(['sample_request'])
+| RequestResponseIO(caller=caller, throttler=throttler))
+result = test_pipeline.run()
+result.wait_until_finish()
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('throttled_requests'))
+self.assertEqual(metrics['counters'][0].committed, 1)
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('cumulativeThrottlingSeconds'))
+self.assertGreater(metrics['counters'][0].committed, 0)
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('responses'))
+self.assertEqual(metrics['counters'][0].committed, 1)
 
 
 if __name__ == '__main__':

(beam) branch tvalentyn-gha updated (e1ce6602619 -> 97991328554)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from e1ce6602619 Upgrade cibuildwheel.
 add 97991328554 Also install setuptools.

No new revisions were added by this update.

Summary of changes:
 sdks/python/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch tvalentyn-gha updated (485913128fe -> e1ce6602619)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from 485913128fe Support Python 3.12 in Beam.
 add e1ce6602619 Upgrade cibuildwheel.

No new revisions were added by this update.

Summary of changes:
 sdks/python/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch tvalentyn-gha updated (dbd91a046f8 -> 485913128fe)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard dbd91a046f8 Support Python 3.12 in Beam.
 discard 7a8a7a4a98a Support Python 3.12 in Dataflow.
 discard cabe9c8cd36 Build Python 3.12 wheels.
 discard ead13ba7ef8 Switch remaining GHA workflows to test Python 3.12.
 discard e878ca40b0a Switch Py3.11 Gradle suites to use Python 3.12.
 add 02fe949438f Switch Py3.11 Gradle suites to use Python 3.12.
 add ca99deabe27 Switch remaining GHA workflows to test Python 3.12.
 add 98010e16018 Build Python 3.12 wheels.
 add f4417e1395f Support Python 3.12 in Dataflow.
 add 485913128fe Support Python 3.12 in Beam.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (dbd91a046f8)
\
 N -- N -- N   refs/heads/tvalentyn-gha (485913128fe)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/beam_PostCommit_Python_Examples_Dataflow.yml |  4 ++--
 .github/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml  |  6 +++---
 .../workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml|  2 +-
 .github/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml  |  2 +-
 .github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml |  2 +-
 .../beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml|  4 ++--
 .github/workflows/beam_PreCommit_Python_PVR_Flink.yml  | 10 +-
 .github/workflows/python_dependency_tests.yml  |  1 +
 .github/workflows/update_python_dependencies.yml   |  9 +
 sdks/python/test-suites/gradle.properties  |  2 +-
 10 files changed, 22 insertions(+), 20 deletions(-)

(beam) branch tvalentyn-gha updated (2af26187399 -> dbd91a046f8)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard 2af26187399 Support Python 3.12 in Beam.
 discard 376c0fafe6e Support Python 3.12 in Dataflow.
 discard a71b5108e79 Build Python 3.12 wheels.
 discard 885f65ecbf0 Switch remaining GHA workflows to test Python 3.12.
 discard b0d6d0169e0 Switch Py3.11 Gradle suites to use Python 3.12.
 discard 060b9b35d79 Gradle definitions for Python 3.12
 discard 357796d4b26 Add a tox suite for Py312.
 discard 902bf98c6f2 Define Python 3.12 test environments in tox.
 add 2babd0aee5b Merge pull request #31120: Update Guava version to 
33.1.0-jre to match the GCP Libraries BOM 26.38.0
 add b30085d10de Define Python 3.12 test environments in tox.
 add 69abdd16576 Add a tox suite for Py312.
 add 232633fae6d Gradle definitions for Python 3.12
 add e878ca40b0a Switch Py3.11 Gradle suites to use Python 3.12.
 add ead13ba7ef8 Switch remaining GHA workflows to test Python 3.12.
 add cabe9c8cd36 Build Python 3.12 wheels.
 add 7a8a7a4a98a Support Python 3.12 in Dataflow.
 add dbd91a046f8 Support Python 3.12 in Beam.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2af26187399)
\
 N -- N -- N   refs/heads/tvalentyn-gha (dbd91a046f8)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 gradle.properties  | 2 +-
 settings.gradle.kts| 3 +++
 3 files changed, 5 insertions(+), 2 deletions(-)

(beam) branch tvalentyn-gha updated (c84e23a623c -> 2af26187399)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


omit c84e23a623c Support Python 3.12 in Beam.
omit ce22bd1b778 Support Python 3.12 in Dataflow.
omit fca3febef73 Build Python 3.12 wheels.
omit 6b4848a1099 Switch remaining GHA workflows to test Python 3.12.
 add b0d6d0169e0 Switch Py3.11 Gradle suites to use Python 3.12.
 add 885f65ecbf0 Switch remaining GHA workflows to test Python 3.12.
 add a71b5108e79 Build Python 3.12 wheels.
 add 376c0fafe6e Support Python 3.12 in Dataflow.
 add 2af26187399 Support Python 3.12 in Beam.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (c84e23a623c)
\
 N -- N -- N   refs/heads/tvalentyn-gha (2af26187399)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 sdks/python/test-suites/gradle.properties | 20 ++--
 1 file changed, 10 insertions(+), 10 deletions(-)

(beam) branch tvalentyn-gha updated (57aa3e6f689 -> c84e23a623c)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard 57aa3e6f689 Support Python 3.12 in Beam.
 discard 8b6741ad1e3 Support Python 3.12 in Dataflow.
 discard a2e489db53c Build Python 3.12 wheels.
 discard f9f8b59b3d3 Switch remaining GHA workflows to test Python 3.12.
 discard be08e19d246 Gradle definitions for Python 3.12
 discard 1391921c7d5 Add a tox suite for Py312.
 discard f5fe5a5fe0d Define Python 3.12 test environments in tox.
 add 999f7deb94e [Python] Add job name to GCS custom audit info (#31300)
 add 902bf98c6f2 Define Python 3.12 test environments in tox.
 add 357796d4b26 Add a tox suite for Py312.
 add 060b9b35d79 Gradle definitions for Python 3.12
 add 6b4848a1099 Switch remaining GHA workflows to test Python 3.12.
 add fca3febef73 Build Python 3.12 wheels.
 add ce22bd1b778 Support Python 3.12 in Dataflow.
 add c84e23a623c Support Python 3.12 in Beam.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (57aa3e6f689)
\
 N -- N -- N   refs/heads/tvalentyn-gha (c84e23a623c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/io/gcp/gcsio.py| 48 --
 sdks/python/apache_beam/io/gcp/gcsio_test.py   | 47 +
 .../runners/dataflow/internal/apiclient.py | 20 ++---
 .../apache_beam/runners/interactive/utils.py   | 18 +---
 .../runners/portability/sdk_container_builder.py   | 14 ++-
 .../dataflow/{py310 => py312}/build.gradle |  2 +-
 .../direct/{py38 => py312}/build.gradle|  2 +-
 .../portable/{py310 => py312}/build.gradle |  2 +-
 8 files changed, 94 insertions(+), 59 deletions(-)
 copy sdks/python/test-suites/dataflow/{py310 => py312}/build.gradle (97%)
 copy sdks/python/test-suites/direct/{py38 => py312}/build.gradle (97%)
 copy sdks/python/test-suites/portable/{py310 => py312}/build.gradle (97%)

(beam) branch tvalentyn-gha updated (82558028498 -> 57aa3e6f689)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard 82558028498 Support Python 3.12 in Beam.
 discard 9e0a5d49fce Support Python 3.12 in Dataflow.
 discard 911fd417af8 Build Python 3.12 wheels.
 discard 2194aa644d5 Switch remaining GHA workflows to test Python 3.12.
 discard f88089de8a3 Add a tox suite for Py312.
 discard 2e8c62d54d2 Define Python 3.12 test environments in tox.
 discard ede1d831152 Increase pyarrow upper bound. Add compat tests, and remove 
some of the compat suites for pyarrow to reduce test suite runtime.
 add 118c51404d9 Implement DeduplicateTensorPerRow in MLTransform (#31307)
 add 084f23b2a8d Install Beam from wheels in Dependency Compat Test Suite. 
(#31308)
 add 743e34e0098 Fix iceberg unit tests (#31314)
 add 126d92249f9 Support newer versions of Pyarrow in Beam. (#31305)
 add 840faea2262 Create option to specify temp query project, and wire into 
source tab… (#31128)
 add f5fe5a5fe0d Define Python 3.12 test environments in tox.
 add 1391921c7d5 Add a tox suite for Py312.
 add be08e19d246 Gradle definitions for Python 3.12
 add f9f8b59b3d3 Switch remaining GHA workflows to test Python 3.12.
 add a2e489db53c Build Python 3.12 wheels.
 add 8b6741ad1e3 Support Python 3.12 in Dataflow.
 add 57aa3e6f689 Support Python 3.12 in Beam.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (82558028498)
\
 N -- N -- N   refs/heads/tvalentyn-gha (57aa3e6f689)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 ..._Gcp_Direct.json => IO_Iceberg_Unit_Tests.json} |  0
 ...json => beam_PostCommit_Python_Dependency.json} |  0
 .github/workflows/IO_Iceberg_Unit_Tests.yml|  4 +-
 .github/workflows/beam_PostCommit_Python_Arm.yml   |  2 +-
 .../beam_PostCommit_Python_Dependency.yml  |  1 +
 .../beam_PostCommit_Python_Examples_Spark.yml  |  2 +-
 ..._PostCommit_Python_ValidatesRunner_Dataflow.yml |  2 +-
 .../beam_PostCommit_TransformService_Direct.yml|  2 +-
 .github/workflows/beam_PostCommit_XVR_Flink.yml|  4 +-
 ...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml |  4 +-
 ...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml |  4 +-
 .github/workflows/beam_PostCommit_XVR_Samza.yml|  4 +-
 .github/workflows/beam_PostCommit_XVR_Spark3.yml   |  4 +-
 .github/workflows/beam_PreCommit_PythonDocker.yml  |  4 +-
 .github/workflows/beam_PreCommit_Python_ML.yml |  2 +-
 build.gradle.kts   | 12 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java   | 39 +++---
 .../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 10 +++-
 .../sdk/io/gcp/bigquery/BigQueryQueryHelper.java   | 15 +++---
 .../io/gcp/bigquery/BigQueryQuerySourceDef.java| 26 +++--
 .../gcp/bigquery/BigQueryStorageQuerySource.java   |  8 +++
 .../sdk/io/gcp/bigquery/BigQueryIOReadTest.java| 26 -
 .../gcp/bigquery/BigQueryIOStorageQueryTest.java   |  6 +++
 .../io/gcp/bigquery/BigQueryIOTranslationTest.java |  1 +
 sdks/python/apache_beam/ml/transforms/tft.py   | 22 
 sdks/python/apache_beam/ml/transforms/tft_test.py  | 62 ++
 26 files changed, 225 insertions(+), 41 deletions(-)
 copy .github/trigger_files/{beam_PostCommit_Python_Xlang_Gcp_Direct.json => 
IO_Iceberg_Unit_Tests.json} (100%)
 copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json 
=> beam_PostCommit_Python_Dependency.json} (100%)

(beam) branch master updated: Support newer versions of Pyarrow in Beam. (#31305)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 126d92249f9 Support newer versions of Pyarrow in Beam. (#31305)
126d92249f9 is described below

commit 126d92249f9babb1977087df277940f8646916d9
Author: tvalentyn 
AuthorDate: Thu May 16 08:50:52 2024 -0700

Support newer versions of Pyarrow in Beam. (#31305)

* Increase pyarrow upper bound. Add compat tests, and remove some of the 
compat suites for pyarrow to reduce test suite runtime.
---
 .../beam_PostCommit_Python_Dependency.json |  0
 sdks/python/setup.py   | 14 +
 sdks/python/test-suites/tox/py38/build.gradle  | 63 +++---
 sdks/python/tox.ini| 30 ---
 4 files changed, 54 insertions(+), 53 deletions(-)

diff --git a/.github/trigger_files/beam_PostCommit_Python_Dependency.json 
b/.github/trigger_files/beam_PostCommit_Python_Dependency.json
new file mode 100644
index 000..e69de29bb2d
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 3e1bfda0dd3..852f14117d8 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -143,21 +143,9 @@ except ImportError:
 # [BEAM-8181] pyarrow cannot be installed on 32-bit Windows platforms.
 if sys.platform == 'win32' and sys.maxsize <= 2**32:
   pyarrow_dependency = ['']
-elif sys.platform == 'win32' or sys.platform == 'cygwin':
-  # https://github.com/apache/beam/issues/28410 - pyarrow>=13 seeing issues
-  # on windows with error
-  # C:\arrow\cpp\src\arrow\filesystem\s3fs.cc:2904:  arrow::fs::FinalizeS3 was
-  # not called even though S3 was initialized.  This could lead to a
-  # segmentation fault at exit. Keep pyarrow<13 until this is resolved.
-  pyarrow_dependency = [
-  'pyarrow>=3.0.0,<12.0.0',
-  # NOTE: We can remove this once Beam increases the pyarrow lower bound
-  # to a version that fixes CVE.
-  'pyarrow-hotfix<1'
-  ]
 else:
   pyarrow_dependency = [
-  'pyarrow>=3.0.0,<15.0.0',
+  'pyarrow>=3.0.0,<17.0.0',
   # NOTE(https://github.com/apache/beam/issues/29392): We can remove this
   # once Beam increases the pyarrow lower bound to a version that fixes 
CVE.
   'pyarrow-hotfix<1'
diff --git a/sdks/python/test-suites/tox/py38/build.gradle 
b/sdks/python/test-suites/tox/py38/build.gradle
index 52299d8c31f..2ca82d3d926 100644
--- a/sdks/python/test-suites/tox/py38/build.gradle
+++ b/sdks/python/test-suites/tox/py38/build.gradle
@@ -44,36 +44,48 @@ project.tasks.register("preCommitPyCoverage") {
 // e.g. pyarrow and pandas also run on PreCommit Dataframe and Coverage
 project.tasks.register("postCommitPyDep") {}
 
-// Create a test task for each supported major version of pyarrow
+// Create a test task for supported major versions of pyarrow
+// We should have a test for the lowest supported version and
+// For versions that we would like to prioritize for testing,
+// for example versions released in a timeframe of last 1-2 years.
+
 toxTask "testPy38pyarrow-3", "py38-pyarrow-3", "${posargs}"
 test.dependsOn "testPy38pyarrow-3"
 postCommitPyDep.dependsOn "testPy38pyarrow-3"
 
-toxTask "testPy38pyarrow-4", "py38-pyarrow-4", "${posargs}"
-test.dependsOn "testPy38pyarrow-4"
-postCommitPyDep.dependsOn "testPy38pyarrow-4"
+toxTask "testPy38pyarrow-9", "py38-pyarrow-9", "${posargs}"
+test.dependsOn "testPy38pyarrow-9"
+postCommitPyDep.dependsOn "testPy38pyarrow-9"
 
-toxTask "testPy38pyarrow-5", "py38-pyarrow-5", "${posargs}"
-test.dependsOn "testPy38pyarrow-5"
-postCommitPyDep.dependsOn "testPy38pyarrow-5"
+toxTask "testPy38pyarrow-10", "py38-pyarrow-10", "${posargs}"
+test.dependsOn "testPy38pyarrow-10"
+postCommitPyDep.dependsOn "testPy38pyarrow-10"
 
-toxTask "testPy38pyarrow-6", "py38-pyarrow-6", "${posargs}"
-test.dependsOn "testPy38pyarrow-6"
-postCommitPyDep.dependsOn "testPy38pyarrow-6"
+toxTask "testPy38pyarrow-11", "py38-pyarrow-11", "${posargs}"
+test.dependsOn "testPy38pyarrow-11"
+postCommitPyDep.dependsOn "testPy38pyarrow-11"
 
-toxTask "testPy38pyarrow-7", "py38-pyarrow-7", "${posargs}"
-test.dependsOn "testPy38pyarrow-7"
-postCommitPyDep.dependsOn "testPy38pyarrow-7"
+toxTask "testPy38pyarrow-12", "py38-pyarrow-12", "${posargs}"
+test.dependsOn "testPy38pyarrow-12"
+postCommitPyDep.dependsOn "testPy38pyarrow-12"
 
-toxTask &quo

(beam) branch tvalentyn-gha updated (2194aa644d5 -> 82558028498)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from 2194aa644d5 Switch remaining GHA workflows to test Python 3.12.
 add 911fd417af8 Build Python 3.12 wheels.
 add 9e0a5d49fce Support Python 3.12 in Dataflow.
 add 82558028498 Support Python 3.12 in Beam.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build_wheels.yml  | 2 +-
 sdks/python/apache_beam/__init__.py | 2 +-
 sdks/python/apache_beam/runners/dataflow/internal/apiclient.py  | 2 +-
 sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py | 2 +-
 sdks/python/setup.py| 1 +
 5 files changed, 5 insertions(+), 4 deletions(-)

(beam) branch tvalentyn-gha updated (f88089de8a3 -> 2194aa644d5)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from f88089de8a3 Add a tox suite for Py312.
 add 2194aa644d5 Switch remaining GHA workflows to test Python 3.12.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/beam_PostCommit_Python.yml  | 2 +-
 .github/workflows/beam_PostCommit_Python_Examples_Flink.yml   | 2 +-
 .../workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml  | 2 +-
 .../beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml| 2 +-
 .github/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml| 2 +-
 .github/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml| 2 +-
 .github/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml| 2 +-
 .github/workflows/beam_PostCommit_Sickbay_Python.yml  | 2 +-
 .github/workflows/beam_PostCommit_TransformService_Direct.yml | 2 +-
 .github/workflows/beam_PostCommit_XVR_Direct.yml  | 4 ++--
 .github/workflows/beam_PreCommit_Portable_Python.yml  | 2 +-
 .github/workflows/beam_PreCommit_Python_Examples.yml  | 2 +-
 .github/workflows/beam_PreCommit_Python_Integration.yml   | 2 +-
 .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml | 4 ++--
 14 files changed, 16 insertions(+), 16 deletions(-)

(beam) branch tvalentyn-gha updated (ead701cf765 -> f88089de8a3)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


omit ead701cf765 Install Beam from wheels in Dependency Compat Test Suite.
omit 3dac627e918 trigger
omit 7593608e219 Increase pyarrow upper bound. Add compat tests, and remove 
some of the compat suites for pyarrow to reduce test suite runtime.
 add 041a8dd1173 Touch Samza ValidatesRunner trigger files
 add f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
 add 2196758c20b Merge pull request #31270: Reapply "Add Redistribute 
translation to Samza runner"
 add 6cb30cc5c86 setup GCP auth before running tests (#31306)
 add ede1d831152 Increase pyarrow upper bound. Add compat tests, and remove 
some of the compat suites for pyarrow to reduce test suite runtime.
 add 2e8c62d54d2 Define Python 3.12 test environments in tox.
 add f88089de8a3 Add a tox suite for Py312.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (ead701cf765)
\
 N -- N -- N   refs/heads/tvalentyn-gha (f88089de8a3)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../beam_PostCommit_Java_PVR_Samza.json|  3 +-
 ...beam_PostCommit_Java_ValidatesRunner_Samza.json |  3 +-
 .../beam_PostCommit_Python_Dependency.json |  0
 .github/workflows/IO_Iceberg_Unit_Tests.yml|  6 ++
 .../beam_PostCommit_Python_Dependency.yml  |  1 -
 .../beam_PostCommit_Python_Examples_Direct.yml |  2 +-
 .github/workflows/beam_PreCommit_Python.yml|  2 +-
 .../workflows/beam_PreCommit_Python_Dataframes.yml |  2 +-
 .../workflows/beam_PreCommit_Python_Runners.yml|  2 +-
 .../workflows/beam_PreCommit_Python_Transforms.yml |  2 +-
 .github/workflows/python_tests.yml |  5 +-
 build.gradle.kts   |  1 +
 .../translation/RedistributeByKeyTranslator.java   | 64 ++
 .../samza/translation/ReshuffleTranslator.java | 14 -
 .../samza/translation/SamzaPipelineTranslator.java |  1 +
 .../test-suites/tox/{py311 => py312}/build.gradle  |  2 +-
 sdks/python/tox.ini| 30 +-
 settings.gradle.kts|  1 +
 18 files changed, 113 insertions(+), 28 deletions(-)
 delete mode 100644 .github/trigger_files/beam_PostCommit_Python_Dependency.json
 create mode 100644 
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
 copy sdks/python/test-suites/tox/{py311 => py312}/build.gradle (97%)

(beam) branch tvalentyn-gha updated (db6a0ec8340 -> ead701cf765)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git

discard db6a0ec8340 Define Python 3.12 test environments in tox.
discard a01bcfd53a8 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
omit 2196758c20b Merge pull request #31270: Reapply "Add Redistribute
translation to Samza runner"
omit f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
omit 041a8dd1173 Touch Samza ValidatesRunner trigger files
add 7593608e219 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
add 3dac627e918 trigger
add ead701cf765 Install Beam from wheels in Dependency Compat Test Suite.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

* -- * -- B -- O -- O -- O (db6a0ec8340)
\
N -- N -- N refs/heads/tvalentyn-gha (ead701cf765)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
.../beam_PostCommit_Java_PVR_Samza.json| 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
...json => beam_PostCommit_Python_Dependency.json} | 0
.../beam_PostCommit_Python_Dependency.yml | 1 +
.../translation/RedistributeByKeyTranslator.java | 64 --
.../samza/translation/ReshuffleTranslator.java | 14 +
.../samza/translation/SamzaPipelineTranslator.java | 1 -
sdks/python/tox.ini| 30 +-
8 files changed, 20 insertions(+), 96 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json
=> beam_PostCommit_Python_Dependency.json} (100%)
delete mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java

(beam) branch tvalentyn-gha updated (b3bf42b65e6 -> db6a0ec8340)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard b3bf42b65e6 trigger
 discard 00ad85d284d Attmpt to update pyarrow and add compat test suite 
definitions.
 discard 1ba37cd6d69 Define Python 3.12 test environments in tox.
 add 5a42d54bfd1 Catch org.elasticsearch.client.ResponseException and 
return document with withHasError(true)
 add 29b47e69716 Remove status
 add 84b1d92a73e Fix Style
 add ff858b8b966 doc -> doc.withHasError(true)
 add 3cbf7de5796 Add tests
 add 1647872e054 Fix styles via spotlessApply
 add 41772b015c2 rename INJECT_SOME_INVALID_DOCS and insert numDoc
 add 00c59b251ec Catch with java.io.IOExecption
 add 591d2c75679 explicit import
 add 0f6a79cd2eb Removed unused import
 add a7380297011 java util explicit import
 add 035c88da6cf spotlessapply
 add c6eabd19faf Fix max batch size
 add 77da5add258 Rename message from 
org.elasticsearch.client.ResponseException to java.io.IOException
 add 1ee2f0ec5e6 Add test to verify response items message
 add e010a3ecc66 spotlessapply
 add 5ee569f930b fix comment
 add ed84ba0e0dc Fix naming
 add 1a399b96977 Verify the whole error message string
 add e97064aa14c Remove mapToInputDoc
 add 946f4959b7b retry 500-range http except 501
 add 68790eb242f Add elasticResponseExceptionMessage predicate
 add 3d897b27635 Merge pull request #31151 from andyzhangdialpad/master
 add 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python 
3.11+ (#31288)
 add 041a8dd1173 Touch Samza ValidatesRunner trigger files
 add f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
 add 2196758c20b Merge pull request #31270: Reapply "Add Redistribute 
translation to Samza runner"
 add a01bcfd53a8 Increase pyarrow upper bound. Add compat tests, and remove 
some of the compat suites for pyarrow to reduce test suite runtime.
 add db6a0ec8340 Define Python 3.12 test environments in tox.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (b3bf42b65e6)
\
 N -- N -- N   refs/heads/tvalentyn-gha (db6a0ec8340)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../beam_PostCommit_Java_PVR_Samza.json|  3 +-
 ...beam_PostCommit_Java_ValidatesRunner_Samza.json |  3 +-
 .../beam_PostCommit_Python_Dependency.json |  0
 .../translation/RedistributeByKeyTranslator.java   | 64 ++
 .../samza/translation/ReshuffleTranslator.java | 14 -
 .../samza/translation/SamzaPipelineTranslator.java |  1 +
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |  6 ++
 .../elasticsearch/ElasticsearchIOTestCommon.java   | 52 ++
 .../io/elasticsearch/ElasticsearchIOTestUtils.java | 26 -
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 54 ++
 .../ml/transforms/embeddings/huggingface_test.py   |  2 +
 sdks/python/setup.py   | 20 ++-
 sdks/python/test-suites/tox/py38/build.gradle  | 39 ++---
 sdks/python/tox.ini| 45 +--
 14 files changed, 247 insertions(+), 82 deletions(-)
 delete mode 100644 .github/trigger_files/beam_PostCommit_Python_Dependency.json
 create mode 100644 
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java

(beam) branch master updated: Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python 
3.11+ (#31288)
6de9a60dc9e is described below

commit 6de9a60dc9e456d34b3fcd0583e29372656c995c
Author: tvalentyn 
AuthorDate: Wed May 15 08:38:01 2024 -0700

Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288)

* Don't install TFT on Python 3.11+

* Add an upper bound to TF due to #31294 .

* Run huggingface tests without parallelism: see comments in #31287
---
 .../apache_beam/ml/transforms/embeddings/huggingface_test.py  | 2 ++
 sdks/python/setup.py  | 8 +---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git 
a/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py 
b/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py
index f38fc8e8542..f94e747c5ed 100644
--- a/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py
+++ b/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py
@@ -21,6 +21,7 @@ import unittest
 import uuid
 
 import numpy as np
+import pytest
 from parameterized import parameterized
 
 import apache_beam as beam
@@ -80,6 +81,7 @@ _parameterized_inputs = [
 ]
 
 
+@pytest.mark.no_xdist
 @unittest.skipIf(
 SentenceTransformerEmbeddings is None,
 'sentence-transformers is not installed.')
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 181d810971a..3e1bfda0dd3 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -490,16 +490,18 @@ if __name__ == '__main__':
   # We don't expect users to install this extra. Users should install
   # necessary dependencies individually, or we should create targeted
   # extras. Keeping the bounds open as much as possible so that we
-  # can find out early when using Beam with new versions doesn't work.
+  # can find out early when Beam doesn't work with new versions.
   'ml_test': [
   'datatable',
   'embeddings',
   'onnxruntime',
   'sentence-transformers',
   'skl2onnx',
-  'tensorflow',
+  # https://github.com/apache/beam/issues/31294
+  'tensorflow<2.16.0',
   'tensorflow-hub',
-  'tensorflow_transform',
+  # https://github.com/tensorflow/transform/issues/313
+  'tensorflow-transform;python_version<"3.11"',
   'tf2onnx',
   'torch',
   'transformers',

(beam) branch tvalentyn-gha updated (bd832576057 -> b3bf42b65e6)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard bd832576057 Trigger tests
 discard 15c077fac15 Don't install TFT on Python 3.11+
 add 785855e6c4c Exclude testRedistributePreservesMetadata from Spark PVR 
batch (#31286)
 add 54db453b032 Fix postrelease revert leftover (#31292)
 add 1ba37cd6d69 Define Python 3.12 test environments in tox.
 add 00ad85d284d Attmpt to update pyarrow and add compat test suite 
definitions.
 add b3bf42b65e6 trigger

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (bd832576057)
\
 N -- N -- N   refs/heads/tvalentyn-gha (b3bf42b65e6)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../beam_PostCommit_Java_PVR_Spark3_Batch.json |  4 ---
 .../beam_PostCommit_Java_PVR_Spark_Batch.json  |  3 +-
 ...json => beam_PostCommit_Python_Dependency.json} |  0
 release/build.gradle.kts   |  2 +-
 runners/flink/flink_runner.gradle  |  8 --
 runners/spark/job-server/spark_job_server.gradle   |  3 +-
 sdks/python/setup.py   |  7 ++---
 sdks/python/test-suites/tox/py38/build.gradle  | 28 ++
 sdks/python/tox.ini| 33 +-
 9 files changed, 61 insertions(+), 27 deletions(-)
 delete mode 100644 
.github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json
 rename .github/trigger_files/{beam_PostCommit_Python_ML.json => 
beam_PostCommit_Python_Dependency.json} (100%)

(beam) branch tvalentyn-gha updated (15c077fac15 -> bd832576057)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from 15c077fac15 Don't install TFT on Python 3.11+
 add bd832576057 Trigger tests

No new revisions were added by this update.

Summary of changes:
 ...t_Python_ValidatesRunner_Flink.json => beam_PostCommit_Python_ML.json} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json 
=> beam_PostCommit_Python_ML.json} (100%)

(beam) branch tvalentyn-gha updated (0a827f31b7a -> 15c077fac15)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


omit 0a827f31b7a Include ML deps for coverage analysis.
omit 404b57a7fa1 Move ML compatibility tests to postcommits.
omit 13c133176f1 Lint
omit 05451b62a97 Add a subsuite of unit tests for ML package
 add fbb3ae1ebef add retries for flaky test (#31247)
 add 45c730ad0c7 Update version tag for dataflow dev containers. (#31248)
 add 2e5f69c4bb9 Update try-apache-beam-yaml.ipynb (#31257)
 add 66da760357c Update yaml to non-beta link (#31210)
 add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and 
PVR suites
 add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest
 add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK"
 add 737be230347 Merge pull request #31156: Reapply "Add Redistribute 
transform to Java SDK"
 add 466516b016c [Python] Use BufferedLogger instead of logger to log in 
boot.go (#31265)
 add 749c128bef1 Disclaim direct runner result can be different with remote 
runners (#31250)
 add c230655a1e5 Implement the hash_words TFT operation (#31249)
 add 296c13a291d Avoid side input issue where Latest.globally produces 2 
elements in a window (#31263)
 add c88a863870c Add a subsuite of unit tests for ML package and remove 
them from 'coverage' GHA workflow. (#31255)
 add 31e81ffe12a add windmill computation state cache, config loaders, and 
status pages (#31133)
 add 459c87af706 Revert "[flink] Add documentation for Flink runner 
versions updates"
 add 557b5bad1b0 Revert "[#30789] Add support for Flink 1.18 (#31062)"
 add 0542c83fc1a Un-revert doc fixes
 add bcac88b815e Merge pull request #31274 from apache/debugpostrelease
 add c7937001981 Beam YAML day zero notebook save file fix (#31266)
 add c1d1bd642d1 Fix a typo in skip condition. (#31267)
 add 4a15af019cc Touch trigger files for Dataflow ValidatesRunner tests
 add 238792d7c65 Reapply "Add direct Dataflow translation of Redistribute 
transform"
 add 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow 
translation of Redistribute transform"
 add 8869175a887 Fix nullness errors in TestPipeline
 add 7f9264ad91d Merge pull request #31230: Fix nullness errors in 
TestPipeline
 add ee170e142b9 Set backlog in gauge metric (#31137)
 add 5c5bcccf516 Copy edit the code contribution guide (#31279)
 add c061b2776b0 Revert "Don't specify an unnecessary config given that tox 
task and gradle config already defines the Python version." (#31283)
 add 15c077fac15 Don't install TFT on Python 3.11+

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (0a827f31b7a)
\
 N -- N -- N   refs/heads/tvalentyn-gha (15c077fac15)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../test-properties.json   |   2 +-
 .../beam_PostCommit_Java_PVR_Flink_Batch.json  |   4 +
 .../beam_PostCommit_Java_PVR_Flink_Docker.json |   4 +
 .../beam_PostCommit_Java_PVR_Flink_Streaming.json  |   4 +
 .../beam_PostCommit_Java_PVR_Samza.json|   4 +
 .../beam_PostCommit_Java_PVR_Spark3_Batch.json |   4 +
 .../beam_PostCommit_Java_PVR_Spark3_Streaming.json |   3 +-
 ...m_PostCommit_Java_ValidatesRunner_Dataflow.json |   4 +-
 ...Java_ValidatesRunner_Dataflow_JavaVersions.json |   4 +
 ...it_Java_ValidatesRunner_Dataflow_Streaming.json |   4 +-
 ...ostCommit_Java_ValidatesRunner_Dataflow_V2.json |   4 +-
 ...Java_ValidatesRunner_Dataflow_V2_Streaming.json |   4 +-
 ...eam_PostCommit_Java_ValidatesRunner_Direct.json |   4 +
 ...t_Java_ValidatesRunner_Direct_JavaVersions.json |   4 +
 ...beam_PostCommit_Java_ValidatesRunner_Flink.json |   3 +-
 ...stCommit_Java_ValidatesRunner_Flink_Java11.json |   3 +-
 ...beam_PostCommit_Java_ValidatesRunner_Samza.json |   3 +-
 ...beam_PostCommit_Java_ValidatesRunner_Spark.json |   3 +-
 ...a_ValidatesRunner_SparkStructuredStreaming.json |   3 +-
 ...stCommit_Java_ValidatesRunner_Spark_Java11.json |   4 +
 ...m_PostCommit_Java_ValidatesRunner_Twister2.json |   3 +-
 .../beam_PostCommit_Java_ValidatesRunner_ULR.json  |   3 +-
 .github/workflows/README.m

(beam) branch master updated (5c5bcccf516 -> c061b2776b0)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 5c5bcccf516 Copy edit the code contribution guide (#31279)
 add c061b2776b0 Revert "Don't specify an unnecessary config given that tox 
task and gradle config already defines the Python version." (#31283)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/beam_PreCommit_Python.yml| 3 ++-
 .github/workflows/beam_PreCommit_Python_Dataframes.yml | 3 ++-
 .github/workflows/beam_PreCommit_Python_Examples.yml   | 3 ++-
 .github/workflows/beam_PreCommit_Python_ML.yml | 3 ++-
 .github/workflows/beam_PreCommit_Python_Runners.yml| 3 ++-
 .github/workflows/beam_PreCommit_Python_Transforms.yml | 3 ++-
 sdks/python/apache_beam/transforms/sql.py  | 2 +-
 7 files changed, 13 insertions(+), 7 deletions(-)

(beam) branch master updated: Fix a typo in skip condition. (#31267)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new c1d1bd642d1 Fix a typo in skip condition. (#31267)
c1d1bd642d1 is described below

commit c1d1bd642d152f05ff5bcf64b4630e059c681ba7
Author: tvalentyn 
AuthorDate: Tue May 14 08:04:27 2024 -0700

Fix a typo in skip condition. (#31267)
---
 sdks/python/apache_beam/dataframe/frames_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/dataframe/frames_test.py 
b/sdks/python/apache_beam/dataframe/frames_test.py
index b3ddfee95fc..076ab504add 100644
--- a/sdks/python/apache_beam/dataframe/frames_test.py
+++ b/sdks/python/apache_beam/dataframe/frames_test.py
@@ -1449,7 +1449,7 @@ class DeferredFrameTest(_AbstractFrameTest):
 self._run_test(lambda s: s.unstack(level=0), s)
 
   @unittest.skipIf(
-  sys.version_info >= (2, 12) and PD_VERSION < (2, 3),
+  sys.version_info >= (3, 12) and PD_VERSION < (2, 3),
   'https://github.com/pandas-dev/pandas/issues/58604')
   def test_unstack_pandas_example3(self):
 index = self._unstack_get_categorical_index()

(beam) branch master updated (296c13a291d -> c88a863870c)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 296c13a291d Avoid side input issue where Latest.globally produces 2 
elements in a window (#31263)
 add c88a863870c Add a subsuite of unit tests for ML package and remove 
them from 'coverage' GHA workflow. (#31255)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/README.md   |  1 +
 .github/workflows/beam_PreCommit_Python.yml   |  3 +--
 .../workflows/beam_PreCommit_Python_Dataframes.yml|  3 +--
 .github/workflows/beam_PreCommit_Python_Examples.yml  |  3 +--
 ...ython_Runners.yml => beam_PreCommit_Python_ML.yml} | 19 +--
 .github/workflows/beam_PreCommit_Python_Runners.yml   |  3 +--
 .../workflows/beam_PreCommit_Python_Transforms.yml|  3 +--
 .../apache_beam/ml/inference/huggingface_inference.py |  2 +-
 .../ml/inference/huggingface_inference_test.py|  2 +-
 .../apache_beam/ml/inference/onnx_inference_test.py   |  4 
 sdks/python/setup.py  | 19 +++
 sdks/python/test-suites/tox/common.gradle |  3 +++
 sdks/python/test-suites/tox/py38/build.gradle | 18 ++
 sdks/python/test-suites/tox/pycommon/build.gradle |  4 ++--
 sdks/python/tox.ini   |  8 
 15 files changed, 63 insertions(+), 32 deletions(-)
 copy .github/workflows/{beam_PreCommit_Python_Runners.yml => 
beam_PreCommit_Python_ML.yml} (88%)

(beam) branch master updated (737be230347 -> 466516b016c)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 737be230347 Merge pull request #31156: Reapply "Add Redistribute 
transform to Java SDK"
 add 466516b016c [Python] Use BufferedLogger instead of logger to log in 
boot.go (#31265)

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/boot.go | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(beam) branch tvalentyn-patch-6 created (now 9d655d9583c)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-6
in repository https://gitbox.apache.org/repos/asf/beam.git


  at 9d655d9583c Fix a typo in skip condition.

This branch includes the following new commits:

 new 9d655d9583c Fix a typo in skip condition.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) 01/01: Fix a typo in skip condition.

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-6
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9d655d9583ccb948a219164e9760c74691425055
Author: tvalentyn 
AuthorDate: Mon May 13 08:44:59 2024 -0700

Fix a typo in skip condition.
---
 sdks/python/apache_beam/dataframe/frames_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/dataframe/frames_test.py 
b/sdks/python/apache_beam/dataframe/frames_test.py
index b3ddfee95fc..076ab504add 100644
--- a/sdks/python/apache_beam/dataframe/frames_test.py
+++ b/sdks/python/apache_beam/dataframe/frames_test.py
@@ -1449,7 +1449,7 @@ class DeferredFrameTest(_AbstractFrameTest):
 self._run_test(lambda s: s.unstack(level=0), s)
 
   @unittest.skipIf(
-  sys.version_info >= (2, 12) and PD_VERSION < (2, 3),
+  sys.version_info >= (3, 12) and PD_VERSION < (2, 3),
   'https://github.com/pandas-dev/pandas/issues/58604')
   def test_unstack_pandas_example3(self):
 index = self._unstack_get_categorical_index()

(beam) branch tvalentyn-gha created (now 0a827f31b7a)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


  at 0a827f31b7a Include ML deps for coverage analysis.

No new revisions were added by this update.

(beam) branch master updated: Update version tag for dataflow dev containers. (#31248)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 45c730ad0c7 Update version tag for dataflow dev containers. (#31248)
45c730ad0c7 is described below

commit 45c730ad0c706489ad435fc3210ae0fac606f07e
Author: tvalentyn 
AuthorDate: Fri May 10 13:05:18 2024 -0700

Update version tag for dataflow dev containers. (#31248)
---
 sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py 
b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 4870bcef5e5..bc2d1f5b403 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -34,6 +34,6 @@ SERIALIZED_SOURCE_KEY = 'serialized_source'
 # Unreleased sdks use container image tag specified below.
 # Update this tag whenever there is a change that
 # requires changes to SDK harness container or SDK harness launcher.
-BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240417'
+BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240510'
 
 DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'

(beam) branch master updated: Fix Python 3.12 python hinting errors on MacOS (#31246)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 8b5d0009a1d Fix Python 3.12 python hinting errors on MacOS (#31246)
8b5d0009a1d is described below

commit 8b5d0009a1d2561d4d04df7d952d5c33655f7a15
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Fri May 10 14:03:24 2024 -0400

Fix Python 3.12 python hinting errors on MacOS (#31246)

* Fix typo in Intrinsic Operations test

* Fix load_fast_and_clear
---
 sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py | 4 +---
 sdks/python/apache_beam/typehints/opcodes.py| 2 +-
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py 
b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py
index 8291ef2df5b..adffc945baa 100644
--- a/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py
+++ b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py
@@ -30,9 +30,7 @@ class IntrinsicOneOpsTest(unittest.TestCase):
   def test_unary_intrinsic_ops_are_in_the_same_order_as_in_cpython(self):
 if sys.version_info >= (3, 12):
   dis_order = dis.__dict__['_intrinsic_1_descs']
-  beam_ops = [fn.__name_upper() for fn in intrinsic_one_ops.INT_ONE_OPS]
-  for fn in intrinsic_one_ops.INT_ONE_OPS:
-beam_ops.append(fn.__name__.upper())
+  beam_ops = [fn.__name__.upper() for fn in intrinsic_one_ops.INT_ONE_OPS]
   self.assertListEqual(dis_order, beam_ops)
 
 
diff --git a/sdks/python/apache_beam/typehints/opcodes.py 
b/sdks/python/apache_beam/typehints/opcodes.py
index 11e607543e9..62c7a8fadc3 100644
--- a/sdks/python/apache_beam/typehints/opcodes.py
+++ b/sdks/python/apache_beam/typehints/opcodes.py
@@ -471,7 +471,7 @@ load_fast_check = load_fast
 
 def load_fast_and_clear(state, arg):
   state.stack.append(state.vars[arg])
-  del state.vars[arg]
+  state.vars[arg] = None
 
 
 def store_fast(state, arg):

(beam) 01/01: Update version tag for dataflow dev containres.

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-5
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c885fd413777f151448372b3d01e59330dedaaa9
Author: tvalentyn 
AuthorDate: Fri May 10 10:59:23 2024 -0700

Update version tag for dataflow dev containres.
---
 sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py 
b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 4870bcef5e5..bc2d1f5b403 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -34,6 +34,6 @@ SERIALIZED_SOURCE_KEY = 'serialized_source'
 # Unreleased sdks use container image tag specified below.
 # Update this tag whenever there is a change that
 # requires changes to SDK harness container or SDK harness launcher.
-BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240417'
+BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240510'
 
 DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'

(beam) branch tvalentyn-patch-5 created (now c885fd41377)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-5
in repository https://gitbox.apache.org/repos/asf/beam.git


  at c885fd41377 Update version tag for dataflow dev containres.

This branch includes the following new commits:

 new c885fd41377 Update version tag for dataflow dev containres.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) branch master updated: Remove a deprecated test helper. (#31236)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 0a58d7d2ff4 Remove a deprecated test helper. (#31236)
0a58d7d2ff4 is described below

commit 0a58d7d2ff4dc121acae461c18e9460f7ce6b117
Author: tvalentyn 
AuthorDate: Fri May 10 09:42:23 2024 -0700

Remove a deprecated test helper. (#31236)

* Remove a deprecated test helper.

* yapf
---
 sdks/python/apache_beam/options/pipeline_options_test.py | 4 ++--
 sdks/python/apache_beam/testing/test_pipeline_test.py| 5 +++--
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/sdks/python/apache_beam/options/pipeline_options_test.py 
b/sdks/python/apache_beam/options/pipeline_options_test.py
index c7a56b4739b..61b227d9a24 100644
--- a/sdks/python/apache_beam/options/pipeline_options_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_test.py
@@ -205,7 +205,7 @@ class PipelineOptionsTest(unittest.TestCase):
   @parameterized.expand(TEST_CASES)
   def test_get_all_options_subclass(self, flags, expected, _):
 options = PipelineOptionsTest.MockOptions(flags=flags)
-self.assertDictContainsSubset(expected, options.get_all_options())
+self.assertLessEqual(expected.items(), options.get_all_options().items())
 self.assertEqual(
 options.view_as(PipelineOptionsTest.MockOptions).mock_flag,
 expected['mock_flag'])
@@ -219,7 +219,7 @@ class PipelineOptionsTest(unittest.TestCase):
   @parameterized.expand(TEST_CASES)
   def test_get_all_options(self, flags, expected, _):
 options = PipelineOptions(flags=flags)
-self.assertDictContainsSubset(expected, options.get_all_options())
+self.assertLessEqual(expected.items(), options.get_all_options().items())
 self.assertEqual(
 options.view_as(PipelineOptionsTest.MockOptions).mock_flag,
 expected['mock_flag'])
diff --git a/sdks/python/apache_beam/testing/test_pipeline_test.py 
b/sdks/python/apache_beam/testing/test_pipeline_test.py
index c38b802dc90..06946c7a7ef 100644
--- a/sdks/python/apache_beam/testing/test_pipeline_test.py
+++ b/sdks/python/apache_beam/testing/test_pipeline_test.py
@@ -68,8 +68,9 @@ class TestPipelineTest(unittest.TestCase):
   def test_create_test_pipeline_options(self):
 test_pipeline = TestPipeline(argv=self.TEST_CASE['options'])
 test_options = PipelineOptions(test_pipeline.get_full_options_as_args())
-self.assertDictContainsSubset(
-self.TEST_CASE['expected_dict'], test_options.get_all_options())
+self.assertLessEqual(
+self.TEST_CASE['expected_dict'].items(),
+test_options.get_all_options().items())
 
   EXTRA_OPT_CASES = [{
   'options': {

(beam) branch master updated: Exclude a test that doesn't pass due to an upstream issue. (#31240)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 5ee63c0e04e Exclude a test that doesn't pass due to an upstream issue. 
(#31240)
5ee63c0e04e is described below

commit 5ee63c0e04e433388943922b1578fa16fea0aaa4
Author: tvalentyn 
AuthorDate: Fri May 10 09:00:10 2024 -0700

Exclude a test that doesn't pass due to an upstream issue. (#31240)
---
 sdks/python/apache_beam/dataframe/frames_test.py | 4 
 1 file changed, 4 insertions(+)

diff --git a/sdks/python/apache_beam/dataframe/frames_test.py 
b/sdks/python/apache_beam/dataframe/frames_test.py
index 3ee9adc2bfe..b3ddfee95fc 100644
--- a/sdks/python/apache_beam/dataframe/frames_test.py
+++ b/sdks/python/apache_beam/dataframe/frames_test.py
@@ -15,6 +15,7 @@
 # limitations under the License.
 
 import re
+import sys
 import unittest
 import warnings
 from typing import Dict
@@ -1447,6 +1448,9 @@ class DeferredFrameTest(_AbstractFrameTest):
 s = pd.Series(np.arange(1.0, 5.0), index=index)
 self._run_test(lambda s: s.unstack(level=0), s)
 
+  @unittest.skipIf(
+  sys.version_info >= (2, 12) and PD_VERSION < (2, 3),
+  'https://github.com/pandas-dev/pandas/issues/58604')
   def test_unstack_pandas_example3(self):
 index = self._unstack_get_categorical_index()
 s = pd.Series(np.arange(1.0, 5.0), index=index)

(beam) branch tvalentyn-gha deleted (was 5809f23fc72)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 was 5809f23fc72 fixup

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated: Publish snapshots for Python 3.12 container images. (#31237)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 929f41aea34 Publish snapshots for Python 3.12 container images. 
(#31237)
929f41aea34 is described below

commit 929f41aea347d230459ab40223161acabe08e686
Author: tvalentyn 
AuthorDate: Fri May 10 08:59:19 2024 -0700

Publish snapshots for Python 3.12 container images. (#31237)

* Publish snapshots for Py3.12 container.

* Add a license for a new dep.

* fixup
---
 .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml  | 1 +
 sdks/python/container/license_scripts/dep_urls_py.yaml | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml 
b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
index 0f9d22c3ff5..23b78c36d7b 100644
--- a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
+++ b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
@@ -70,6 +70,7 @@ jobs:
   - "python:container:py39"
   - "python:container:py310"
   - "python:container:py311"
+  - "python:container:py312"
 steps:
   - uses: actions/checkout@v4
   - name: Setup repository
diff --git a/sdks/python/container/license_scripts/dep_urls_py.yaml 
b/sdks/python/container/license_scripts/dep_urls_py.yaml
index 36efb36c321..6fc5129e35c 100644
--- a/sdks/python/container/license_scripts/dep_urls_py.yaml
+++ b/sdks/python/container/license_scripts/dep_urls_py.yaml
@@ -133,6 +133,8 @@ pip_dependencies:
 license: "https://github.com/PiotrDabkowski/pyjsparser/blob/master/LICENSE;
   pymongo:
 license: 
"https://raw.githubusercontent.com/mongodb/mongo-python-driver/master/LICENSE;
+  pyproject_hooks:
+license: 
"https://raw.githubusercontent.com/pypa/pyproject-hooks/main/LICENSE;
   python-gflags:
 license: 
"https://raw.githubusercontent.com/google/python-gflags/master/COPYING;
   pyvcf:

(beam) branch tvalentyn-gha updated (2536cc276db -> 5809f23fc72)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from 2536cc276db Add a license for a new dep.
 add 5809f23fc72 fixup

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/license_scripts/dep_urls_py.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch tvalentyn-gha updated (6688a500924 -> 2536cc276db)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


from 6688a500924 Publish snapshots for Py3.12 container.
 add 2536cc276db Add a license for a new dep.

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/license_scripts/dep_urls_py.yaml | 2 ++
 1 file changed, 2 insertions(+)

(beam) branch tvalentyn-gha updated (2f295969f16 -> 6688a500924)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard 2f295969f16 Publish snapshots for Py3.12 container.
omit d71dd0ba60d Remove a deprecated test helper.
 add 6688a500924 Publish snapshots for Py3.12 container.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2f295969f16)
\
 N -- N -- N   refs/heads/tvalentyn-gha (6688a500924)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/options/pipeline_options_test.py | 4 ++--
 sdks/python/apache_beam/testing/test_pipeline_test.py| 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

(beam) branch tvalentyn-gha updated: Publish snapshots for Py3.12 container.

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/tvalentyn-gha by this push:
 new 2f295969f16 Publish snapshots for Py3.12 container.
2f295969f16 is described below

commit 2f295969f16191bb50c39d8d833368890fc55c71
Author: Valentyn Tymofieiev 
AuthorDate: Thu May 9 16:09:55 2024 -0700

Publish snapshots for Py3.12 container.
---
 .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml 
b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
index 0f9d22c3ff5..23b78c36d7b 100644
--- a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
+++ b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
@@ -70,6 +70,7 @@ jobs:
   - "python:container:py39"
   - "python:container:py310"
   - "python:container:py311"
+  - "python:container:py312"
 steps:
   - uses: actions/checkout@v4
   - name: Setup repository

(beam) branch tvalentyn-gha created (now d71dd0ba60d)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git


  at d71dd0ba60d Remove a deprecated test helper.

No new revisions were added by this update.

(beam) branch tvalentyn-gha-py39 deleted (was 550b5d7a172)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha-py39
in repository https://gitbox.apache.org/repos/asf/beam.git


 was 550b5d7a172 Move lint, yapf and dependency compat tests to from py38 
to py39

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated: Add a container for Python 3.12. (#31216)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 057eb06514a Add a container for Python 3.12. (#31216)
057eb06514a is described below

commit 057eb06514ad47383ffe33ee1faf0469833dc37d
Author: tvalentyn 
AuthorDate: Thu May 9 09:32:10 2024 -0700

Add a container for Python 3.12. (#31216)

* Add a container for py312.

* fixup: bump the upper version.
---
 sdks/python/container/build.gradle |  23 ++--
 .../container/py312/base_image_requirements.txt| 152 +
 sdks/python/container/py312/build.gradle   |  28 
 settings.gradle.kts|   1 +
 4 files changed, 191 insertions(+), 13 deletions(-)

diff --git a/sdks/python/container/build.gradle 
b/sdks/python/container/build.gradle
index 161d343b303..f07b6f743fa 100644
--- a/sdks/python/container/build.gradle
+++ b/sdks/python/container/build.gradle
@@ -21,7 +21,7 @@ applyGoNature()
 
 description = "Apache Beam :: SDKs :: Python :: Container"
 int min_python_version=8
-int max_python_version=11
+int max_python_version=12
 
 configurations {
   sdkSourceTarball
@@ -38,10 +38,9 @@ goBuild {
 }
 
 tasks.register("buildAll") {
-  dependsOn ':sdks:python:container:py38:docker'
-  dependsOn ':sdks:python:container:py39:docker'
-  dependsOn ':sdks:python:container:py310:docker'
-  dependsOn ':sdks:python:container:py311:docker'
+  for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+  dependsOn ':sdks:python:container:py3' + ver + ':docker'
+  }
 }
 
 for(int i=min_python_version; i<=max_python_version; ++i) {
@@ -68,17 +67,15 @@ for(int i=min_python_version; i<=max_python_version; ++i) {
 }
 
 tasks.register("pushAll") {
-  dependsOn ':sdks:python:container:push38'
-  dependsOn ':sdks:python:container:push39'
-  dependsOn ':sdks:python:container:push310'
-  dependsOn ':sdks:python:container:push311'
+  for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+  dependsOn ':sdks:python:container:push3' + ver
+  }
 }
 
 tasks.register("generatePythonRequirementsAll") {
-  dependsOn ':sdks:python:container:py38:generatePythonRequirements'
-  dependsOn ':sdks:python:container:py39:generatePythonRequirements'
-  dependsOn ':sdks:python:container:py310:generatePythonRequirements'
-  dependsOn ':sdks:python:container:py311:generatePythonRequirements'
+  for(int ver=min_python_version; ver<=max_python_version; ++ver) {
+  dependsOn ':sdks:python:container:py3' + ver + 
':generatePythonRequirements'
+  }
 }
 
 artifacts {
diff --git a/sdks/python/container/py312/base_image_requirements.txt 
b/sdks/python/container/py312/base_image_requirements.txt
new file mode 100644
index 000..4a6147b573b
--- /dev/null
+++ b/sdks/python/container/py312/base_image_requirements.txt
@@ -0,0 +1,152 @@
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+
+# Autogenerated requirements file for Apache Beam py312 container image.
+# Run ./gradlew :sdks:python:container:generatePythonRequirementsAll to update.
+# Do not edit manually, adjust ../base_image_requirements_manual.txt or
+# Apache Beam's setup.py instead, and regenerate the list.
+# You will need Python interpreters for all versions supported by Beam, see:
+# https://s.apache.org/beam-python-dev-wiki
+# Reach out to a committer if you need help.
+
+annotated-types==0.6.0
+attrs==23.2.0
+beautifulsoup4==4.12.3
+bs4==0.0.2
+build==1.2.1
+cachetools==5.3.3
+certifi==2024.2.2
+cffi==1.16.0
+charset-normalizer==3.3.2
+click==8.1.7
+cloudpickle==2.2.1
+cramjam==2.8.3
+crcmod==1.7
+cryptography==42.0.7
+Cython==0.29.37
+deprecation==2.1.0
+dill==0.3.1.1
+dnspython==2.6.1
+docker==7.0.0
+docopt==0.6.2
+docstring_parser==0.16
+execnet==2.1.1
+fastavro==1.9.4
+fasteners==0.19
+freezegun==1.5.0
+future==1.0.0
+google-api-core==2.19.0
+google-api-python-client==2.128.0
+google-apitools==0.5.31
+google-auth==2.29.0
+google-auth-httplib2==0.2.0
+google-cloud-aiplatform==1

(beam) branch master updated (287ed388a68 -> 7ed94383d2e)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 287ed388a68 Fix trivial inference tests for Python 3.12 support 
(#31170)
 add 7ed94383d2e Don't hardcode python version in tox.ini for lint, mypy 
and docs. (#31222)

No new revisions were added by this update.

Summary of changes:
 build.gradle.kts  |  5 ++---
 sdks/python/test-suites/tox/py38/build.gradle | 14 --
 sdks/python/test-suites/tox/pycommon/build.gradle | 18 --
 sdks/python/tox.ini   |  8 
 4 files changed, 22 insertions(+), 23 deletions(-)

(beam) branch master updated: Fix trivial inference tests for Python 3.12 support (#31170)

2024-05-08 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 287ed388a68 Fix trivial inference tests for Python 3.12 support 
(#31170)
287ed388a68 is described below

commit 287ed388a68e2013dab29f17e9f470c279645d3a
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Wed May 8 20:29:16 2024 -0400

Fix trivial inference tests for Python 3.12 support (#31170)

* Fix trivial inference for Python 3.12 support

* remove debugging prints

* Address comments

* add unit test for intrinsic op order

* linting

* add unittest.main()

* suggestions
---
 .../apache_beam/typehints/intrinsic_one_ops.py | 100 +
 .../typehints/intrinsic_one_ops_test.py|  40 +
 sdks/python/apache_beam/typehints/opcodes.py   |  65 ++
 .../apache_beam/typehints/trivial_inference.py |  30 ++-
 .../typehints/trivial_inference_test.py|   1 +
 5 files changed, 235 insertions(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/typehints/intrinsic_one_ops.py 
b/sdks/python/apache_beam/typehints/intrinsic_one_ops.py
new file mode 100644
index 000..c82141cb404
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/intrinsic_one_ops.py
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Defines the actions intrinsic bytecodes have on the frame.
+
+Each function here corresponds to a bytecode documented in
+https://docs.python.org/3/library/dis.html . The first argument is a (mutable)
+FrameState object, the second the integer opcode argument.
+
+Bytecodes with more complicated behavior (e.g. modifying the program counter)
+are handled inline rather than here.
+
+For internal use only; no backwards-compatibility guarantees.
+"""
+# pytype: skip-file
+
+from . import opcodes
+
+
+def intrinsic_1_invalid(state, arg):
+  pass
+
+
+def intrinsic_print(state, arg):
+  pass
+
+
+def intrinsic_import_star(state, arg):
+  pass
+
+
+def intrinsic_stopiteration_error(state, arg):
+  pass
+
+
+def intrinsic_async_gen_wrap(state, arg):
+  pass
+
+
+def intrinsic_unary_positive(state, arg):
+  opcodes.unary_positive(state, arg)
+  pass
+
+
+def intrinsic_list_to_tuple(state, arg):
+  opcodes.list_to_tuple(state, arg)
+  pass
+
+
+def intrinsic_typevar(state, arg):
+  pass
+
+
+def intrinsic_paramspec(state, arg):
+  pass
+
+
+def intrinsic_typevartuple(state, arg):
+  pass
+
+
+def intrinsic_subscript_generic(state, arg):
+  pass
+
+
+def intrinsic_typealias(state, arg):
+  pass
+
+
+# The order of operations in the table of the intrinsic one operations is
+# defined in https://docs.python.org/3/library/dis.html#opcode-CALL_INTRINSIC_1
+# and may change between minor versions.
+INT_ONE_OPS = tuple([
+intrinsic_1_invalid,
+intrinsic_print,
+intrinsic_import_star,
+intrinsic_stopiteration_error,
+intrinsic_async_gen_wrap,
+intrinsic_unary_positive,
+intrinsic_list_to_tuple,
+intrinsic_typevar,
+intrinsic_paramspec,
+intrinsic_typevartuple,
+intrinsic_subscript_generic,
+intrinsic_typealias
+])
diff --git a/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py 
b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py
new file mode 100644
index 000..8291ef2df5b
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distribute

(beam) branch master updated: Targeted dependency update for Pandas 2.1 (#31209)

2024-05-07 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2ad45739acd Targeted dependency update for Pandas 2.1 (#31209)
2ad45739acd is described below

commit 2ad45739acc6fa7eed47a1f5bb3d8ef6aa81
Author: tvalentyn 
AuthorDate: Tue May 7 17:32:03 2024 -0700

Targeted dependency update for Pandas 2.1 (#31209)
---
 CHANGES.md  | 1 +
 sdks/python/container/py310/base_image_requirements.txt | 2 +-
 sdks/python/container/py311/base_image_requirements.txt | 2 +-
 sdks/python/container/py39/base_image_requirements.txt  | 2 +-
 4 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index b64cc5a8f6e..80b702eb711 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -69,6 +69,7 @@
 
 * Added Feast feature store handler for enrichment transform (Python) 
([#30957](https://github.com/apache/beam/issues/30964)).
 * BigQuery per-worker metrics are reported by default for Streaming Dataflow 
Jobs (Java) ([#31015](https://github.com/apache/beam/pull/31015))
+* DataFrame API now supports pandas 2.1.x and adds 12 more string functions 
for Series.([#31185](https://github.com/apache/beam/pull/31185)).
 
 ## Breaking Changes
 
diff --git a/sdks/python/container/py310/base_image_requirements.txt 
b/sdks/python/container/py310/base_image_requirements.txt
index 980dd99d5b7..e2806270e54 100644
--- a/sdks/python/container/py310/base_image_requirements.txt
+++ b/sdks/python/container/py310/base_image_requirements.txt
@@ -101,7 +101,7 @@ objsize==0.7.0
 orjson==3.10.1
 overrides==7.7.0
 packaging==24.0
-pandas==2.0.3
+pandas==2.1.4
 parameterized==0.9.0
 pluggy==1.4.0
 proto-plus==1.23.0
diff --git a/sdks/python/container/py311/base_image_requirements.txt 
b/sdks/python/container/py311/base_image_requirements.txt
index f0615b45b29..1cba006ad32 100644
--- a/sdks/python/container/py311/base_image_requirements.txt
+++ b/sdks/python/container/py311/base_image_requirements.txt
@@ -99,7 +99,7 @@ objsize==0.7.0
 orjson==3.10.1
 overrides==7.7.0
 packaging==24.0
-pandas==2.0.3
+pandas==2.1.4
 parameterized==0.9.0
 pluggy==1.4.0
 proto-plus==1.23.0
diff --git a/sdks/python/container/py39/base_image_requirements.txt 
b/sdks/python/container/py39/base_image_requirements.txt
index bd63ce55de0..39b888dd6ab 100644
--- a/sdks/python/container/py39/base_image_requirements.txt
+++ b/sdks/python/container/py39/base_image_requirements.txt
@@ -102,7 +102,7 @@ objsize==0.7.0
 orjson==3.10.1
 overrides==7.7.0
 packaging==24.0
-pandas==2.0.3
+pandas==2.1.4
 parameterized==0.9.0
 pluggy==1.4.0
 proto-plus==1.23.0

(beam) branch master updated: Upgrade Pandas dependency to 2.1 (#31185)

2024-05-07 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2ca9af8ecbc Upgrade Pandas dependency to 2.1 (#31185)
2ca9af8ecbc is described below

commit 2ca9af8ecbcb3d0cf403bb4ce7d90a8a362b124d
Author: tvalentyn 
AuthorDate: Tue May 7 10:19:11 2024 -0700

Upgrade Pandas dependency to 2.1 (#31185)

* Upgrade to Pandas 2.1

* Pandas 2.1: Disable interchange protocol tests.

* Exclude attrs tests as it is not supported.

* Exclude new doctests that exercise unsupported order-sensitive ops.

* Iteration over deferred DFs is not supported

* Skip 'mul' op when index is used as an axis

* Exclude new tests that use index.

* Exclude shift test as order-sensitive.

* Exclude known failure modes.

* Exclude failures that existed on Pandas 1.

* Allow bulk-exclusion of an example in all tests.

* Exclude examples that use to_timedelta.

* Exclude the test that evaluates an inferred .tz value.

* Exclude more tz and timedelta tests.

* Exclude a tests exercision PeriodProporties.end_time

* Exclude tests exercising unsupported GroupBy operations.

* Expand the list of elementwise string methods.

* Exclude known WontImpl ops

* Fix test output normalization.

* Exclude remaining new tests that didn't work

* Remove test that uses values, an unsupported non-deferred op.

* lint
---
 sdks/python/apache_beam/dataframe/doctests.py  |  16 ++-
 sdks/python/apache_beam/dataframe/frames.py|  19 ++-
 sdks/python/apache_beam/dataframe/frames_test.py   |  14 ++
 .../apache_beam/dataframe/pandas_doctests_test.py  | 150 -
 sdks/python/setup.py   |   6 +-
 5 files changed, 190 insertions(+), 15 deletions(-)

diff --git a/sdks/python/apache_beam/dataframe/doctests.py 
b/sdks/python/apache_beam/dataframe/doctests.py
index a7cff5af5ab..61e904c4f9d 100644
--- a/sdks/python/apache_beam/dataframe/doctests.py
+++ b/sdks/python/apache_beam/dataframe/doctests.py
@@ -225,6 +225,8 @@ class 
_DeferrredDataframeOutputChecker(doctest.OutputChecker):
 
   def fix(self, want, got):
 if 'DeferredBase' in got:
+  # When we have a tuple of Dataframes, pandas prints each from a new line.
+  got = re.sub(r'DeferredBase\[(\d+)\],', '\\g<0>\n', got)
   try:
 to_compute = {
 m.group(0): self._env._all_frames[int(m.group(1))]
@@ -381,20 +383,23 @@ class BeamDataframeDoctestRunner(doctest.DocTestRunner):
 self._skipped_set = set()
 
   def _is_wont_implement_ok(self, example, test):
+always_wont_implement = self._wont_implement_ok.get('*', [])
 return any(
-wont_implement(example)
-for wont_implement in self._wont_implement_ok.get(test.name, []))
+wont_implement(example) for wont_implement in (
+self._wont_implement_ok.get(test.name, []) + 
always_wont_implement))
 
   def _is_not_implemented_ok(self, example, test):
+always_not_impl = self._not_implemented_ok.get('*', [])
 return any(
-not_implemented(example)
-for not_implemented in self._not_implemented_ok.get(test.name, []))
+not_implemented(example) for not_implemented in (
+self._not_implemented_ok.get(test.name, []) + always_not_impl))
 
   def run(self, test, **kwargs):
 self._checker.reset()
+always_skip = self._skip.get('*', [])
 for example in test.examples:
   if any(should_skip(example)
- for should_skip in self._skip.get(test.name, [])):
+ for should_skip in self._skip.get(test.name, []) + always_skip):
 self._skipped_set.add(example)
 example.source = 'pass'
 example.want = ''
@@ -726,6 +731,7 @@ def with_run_patched_docstring(target=None):
 verify the examples, else use PartitioningSession to simulate
 distributed execution.
   skip (Dict[str,str]): A set of examples to skip entirely.
+If a key is '*', an example will be skipped in all test scenarios.
   wont_implement_ok (Dict[str,str]): A set of examples that are allowed to
 raise WontImplementError.
   not_implemented_ok (Dict[str,str]): A set of examples that are allowed to
diff --git a/sdks/python/apache_beam/dataframe/frames.py 
b/sdks/python/apache_beam/dataframe/frames.py
index b7aa130fbbd..421430ec972 100644
--- a/sdks/python/apache_beam/dataframe/frames.py
+++ b/sdks/python/apache_beam/dataframe/frames.py
@@ -1181,8 +1181,11 @@ class 
DeferredDataFrameOrSeries(frame_base.DeferredFrame):
   pd.DataFrame, 'hist', reason="plotting-tools")
 
   attrs = property(
-  frame_base.wont_implement_method(
-  pd.DataFrame, 'attrs', reason='experimenta

(beam) branch tvalentyn-gha-py39 created (now 550b5d7a172)

2024-05-06 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-gha-py39
in repository https://gitbox.apache.org/repos/asf/beam.git


  at 550b5d7a172 Move lint, yapf and dependency compat tests to from py38 
to py39

This branch includes the following new commits:

 new 550b5d7a172 Move lint, yapf and dependency compat tests to from py38 
to py39

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) 01/01: Move lint, yapf and dependency compat tests to from py38 to py39

2024-05-06 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-gha-py39
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 550b5d7a172b20d3d791100e22a5a71ae52f03d5
Author: Valentyn Tymofieiev 
AuthorDate: Mon May 6 11:26:55 2024 -0700

Move lint, yapf and dependency compat tests to from py38 to py39
---
 .../beam_PostCommit_Python_Dependency.yml  |   4 +-
 .../workflows/beam_PreCommit_PythonFormatter.yml   |   2 +-
 .github/workflows/beam_PreCommit_PythonLint.yml|   2 +-
 .../workflows/beam_PreCommit_Python_Coverage.yml   |   4 +-
 .github/workflows/beam_PreCommit_Whitespace.yml|   2 +-
 build.gradle.kts   |  14 +-
 sdks/python/test-suites/tox/py38/build.gradle  | 184 -
 sdks/python/test-suites/tox/py39/build.gradle  | 182 
 8 files changed, 196 insertions(+), 198 deletions(-)

diff --git a/.github/workflows/beam_PostCommit_Python_Dependency.yml 
b/.github/workflows/beam_PostCommit_Python_Dependency.yml
index 51a3a572234..12f353c4152 100644
--- a/.github/workflows/beam_PostCommit_Python_Dependency.yml
+++ b/.github/workflows/beam_PostCommit_Python_Dependency.yml
@@ -77,11 +77,11 @@ jobs:
 uses: ./.github/actions/setup-environment-action
 with:
   java-version: 8
-  python-version: 3.8
+  python-version: 3.9
   - name: Run postCommitPyDep
 uses: ./.github/actions/gradle-command-self-hosted-action
 with:
-  gradle-command: :sdks:python:test-suites:tox:py38:postCommitPyDep
+  gradle-command: :sdks:python:test-suites:tox:py39:postCommitPyDep
   - name: Archive Python Test Results
 uses: actions/upload-artifact@v4
 if: failure()
diff --git a/.github/workflows/beam_PreCommit_PythonFormatter.yml 
b/.github/workflows/beam_PreCommit_PythonFormatter.yml
index 8eaf99ba83a..79d30b04ba0 100644
--- a/.github/workflows/beam_PreCommit_PythonFormatter.yml
+++ b/.github/workflows/beam_PreCommit_PythonFormatter.yml
@@ -81,7 +81,7 @@ jobs:
 uses: ./.github/actions/setup-environment-action
 with:
   java-version: 8
-  python-version: 3.8
+  python-version: 3.9
   - name: run pythonFormatterPreCommit script
 uses: ./.github/actions/gradle-command-self-hosted-action
 with:
diff --git a/.github/workflows/beam_PreCommit_PythonLint.yml 
b/.github/workflows/beam_PreCommit_PythonLint.yml
index 2cc469170d5..b69adfdea4c 100644
--- a/.github/workflows/beam_PreCommit_PythonLint.yml
+++ b/.github/workflows/beam_PreCommit_PythonLint.yml
@@ -81,7 +81,7 @@ jobs:
 uses: ./.github/actions/setup-environment-action
 with:
   java-version: 8
-  python-version: 3.8
+  python-version: 3.9
   go-version: 1.16
   - name: run pythonLintPreCommit script
 uses: ./.github/actions/gradle-command-self-hosted-action
diff --git a/.github/workflows/beam_PreCommit_Python_Coverage.yml 
b/.github/workflows/beam_PreCommit_Python_Coverage.yml
index 9d836ef6332..c71edef8acf 100644
--- a/.github/workflows/beam_PreCommit_Python_Coverage.yml
+++ b/.github/workflows/beam_PreCommit_Python_Coverage.yml
@@ -83,11 +83,11 @@ jobs:
 uses: ./.github/actions/setup-environment-action
 with:
   java-version: 8
-  python-version: 3.8
+  python-version: 3.9
   - name: Run preCommitPyCoverage
 uses: ./.github/actions/gradle-command-self-hosted-action
 with:
-  gradle-command: :sdks:python:test-suites:tox:py38:preCommitPyCoverage
+  gradle-command: :sdks:python:test-suites:tox:py39:preCommitPyCoverage
   - uses: codecov/codecov-action@v3
 with:
   flags: python
diff --git a/.github/workflows/beam_PreCommit_Whitespace.yml 
b/.github/workflows/beam_PreCommit_Whitespace.yml
index 4eba39f0824..f0b37e9fbd7 100644
--- a/.github/workflows/beam_PreCommit_Whitespace.yml
+++ b/.github/workflows/beam_PreCommit_Whitespace.yml
@@ -81,7 +81,7 @@ jobs:
 uses: ./.github/actions/setup-environment-action
 with:
   java-version: 8
-  python-version: 3.8
+  python-version: 3.9
   - name: run whitespacePreCommit script
 uses: ./.github/actions/gradle-command-self-hosted-action
 with:
diff --git a/build.gradle.kts b/build.gradle.kts
index 0fd98ad3b05..8780131f62b 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -492,11 +492,11 @@ tasks.register("pythonDockerBuildPreCommit") {
 
 tasks.register("pythonLintPreCommit") {
   // TODO(https://github.com/apache/beam/issues/20209): Find a better way to 
specify lint and formatter tasks without hardcoding py version.
-  dependsOn(":sdks:python:test-suites:tox:py38:lint")
+  dependsOn(":sdks:python:test-suites:tox:py39:lint")
 }
 
 tasks.register("pythonFormatterPreCommit"

(beam) branch master updated: Update game_stats.py (#31134)

2024-04-30 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 43bf99ddcef Update game_stats.py (#31134)
43bf99ddcef is described below

commit 43bf99ddcefbb0eac0b8ecbe2eaf99d409719a99
Author: Ahmet Altay 
AuthorDate: Tue Apr 30 15:31:03 2024 -0700

Update game_stats.py (#31134)

Fixing a typo in examples & docs.
---
 sdks/python/apache_beam/examples/complete/game/game_stats.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py 
b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index d6f5aab3e7b..233d22b7542 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -196,7 +196,7 @@ class WriteToBigQuery(beam.PTransform):
 # [START abuse_detect]
 class CalculateSpammyUsers(beam.PTransform):
   """Filter out all but those users with a high clickrate, which we will
-  consider as 'spammy' uesrs.
+  consider as 'spammy' users.
 
   We do this by finding the mean total score per user, then using that
   information as a side input to filter out all but those user scores that are

(beam) branch aaltay-patch-1 deleted (was 1bb3931eb6b)

2024-04-30 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch aaltay-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git


 was 1bb3931eb6b Update game_stats.py

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated (0696d00d09b -> 970109e0553)

2024-04-29 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 0696d00d09b Add ReadChangeStream IO param to adjust backlog estimates 
for replication delay (#30995)
 add 970109e0553 Also allow links to Drive materials. (#31131)

No new revisions were added by this update.

Summary of changes:
 website/www/site/static/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch tvalentyn-patch-1 created (now 32f8fe73189)

2024-04-29 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git


  at 32f8fe73189 Also allow links to Drive materials.

This branch includes the following new commits:

 new 32f8fe73189 Also allow links to Drive materials.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) 01/01: Also allow links to Drive materials.

2024-04-29 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 32f8fe73189492fabb2a1e63a2628937ccd1fc2f
Author: tvalentyn 
AuthorDate: Mon Apr 29 10:34:25 2024 -0700

Also allow links to Drive materials.
---
 website/www/site/static/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/website/www/site/static/.htaccess 
b/website/www/site/static/.htaccess
index 51e28c9a274..857b83db3bd 100644
--- a/website/www/site/static/.htaccess
+++ b/website/www/site/static/.htaccess
@@ -27,4 +27,4 @@ RedirectMatch "/contribute/release-guide" 
"https://github.com/apache/beam/blob/m
 
 RedirectMatch "/contribute/committer-guide" 
"https://github.com/apache/beam/blob/master/contributor-docs/committer-guide.md;
 
-Header set Content-Security-Policy "frame-src 'self' 
https://play.beam.apache.org/ https://www.youtube.com/ ;"
+Header set Content-Security-Policy "frame-src 'self' 
https://play.beam.apache.org/ https://www.youtube.com/ 
https://drive.google.com/ ;"

(beam) branch master updated: Mention known issues in release notes. (#31088)

2024-04-24 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2db9b8097b7 Mention known issues in release notes. (#31088)
2db9b8097b7 is described below

commit 2db9b8097b79618c2d3d8c76c45a2fc979196ab3
Author: tvalentyn 
AuthorDate: Wed Apr 24 14:16:12 2024 -0700

Mention known issues in release notes. (#31088)
---
 CHANGES.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/CHANGES.md b/CHANGES.md
index 5fc2509ade8..97894b08374 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -84,6 +84,7 @@
 
 * Fixed locking issue when shutting down inactive bundle processors. Symptoms 
of this issue include slowness or stuckness in long-running jobs (Python) 
([#30679](https://github.com/apache/beam/pull/30679)).
 * Fixed logging issue that caused silecing the pip output when installing of 
dependencies provided in `--requirements_file` (Python).
+* Fixed pipeline stuckness issue by disallowing versions of grpcio that can 
cause the stuckness (Python) 
([#30867](https://github.com/apache/beam/issues/30867)).
 
 ## Security Fixes
 * Fixed (CVE--)[https://www.cve.org/CVERecord?id=CVE--] 
(Java/Python/Go) ([#X](https://github.com/apache/beam/issues/X)).
@@ -144,6 +145,7 @@
 
 * In Python pipelines, when shutting down inactive bundle processors, shutdown 
logic can overaggressively hold the lock, blocking acceptance of new work. 
Symptoms of this issue include slowness or stuckness in long-running jobs. 
Fixed in 2.56.0 ([#30679](https://github.com/apache/beam/pull/30679)).
 * WriteToJson broken in languages other than Java (X-lang) 
([#30776](https://github.com/apache/beam/issues/30776)).
+* Python pipelines might occasionally become stuck due to a regression in 
grpcio ([#30867](https://github.com/apache/beam/issues/30867)). The issue 
manifests frequently with Bigtable IO connector, but might also affect other 
GCP connectors. Fixed in 2.56.0.
 
 # [2.54.0] - 2024-02-14
 
@@ -274,6 +276,7 @@ as a workaround, a copy of "old" `CountingSource` class 
should be placed into a
 
 * MLTransform drops the identical elements in the output PCollection. For any 
duplicate elements, a single element will be emitted downstream. 
([#29600](https://github.com/apache/beam/issues/29600)).
 * Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large 
materialized side inputs might be affected by a performance regression. To 
restore the prior behavior on these SDK versions, supply the 
`--max_cache_memory_usage_mb=0` pipeline option. (Python) 
([#30360](https://github.com/apache/beam/issues/30360)).
+* Users who lauch Python pipelines in an environment without internet access 
and use the `--setup_file` pipeline option might experience an increase in 
pipeline submission time. This has been fixed in 2.56.0 
([#31070](https://github.com/apache/beam/pull/31070)).
 
 # [2.51.0] - 2023-10-03

(beam) branch release-2.56.0 updated: [release-2.56.0] Cherry-picks #31070 (#31072)

2024-04-22 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch release-2.56.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.56.0 by this push:
 new bba3ac36676 [release-2.56.0] Cherry-picks #31070 (#31072)
bba3ac36676 is described below

commit bba3ac366769c73858c0447462af1bdc7ce08406
Author: tvalentyn 
AuthorDate: Mon Apr 22 14:16:21 2024 -0700

[release-2.56.0] Cherry-picks #31070 (#31072)
---
 sdks/python/apache_beam/runners/portability/stager.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sdks/python/apache_beam/runners/portability/stager.py 
b/sdks/python/apache_beam/runners/portability/stager.py
index f6207d80a9d..48dabe18aa3 100644
--- a/sdks/python/apache_beam/runners/portability/stager.py
+++ b/sdks/python/apache_beam/runners/portability/stager.py
@@ -791,6 +791,7 @@ class Stager(object):
   Stager._get_python_executable(),
   '-m',
   'build',
+  '--no-isolation',  # Otherwise, we need internet access to PyPI.
   '--sdist',
   '--outdir',
   temp_dir,

(beam) branch master updated: Don't use isolated builds when building an sdist for a pipeline package supplied in --setup_file option. (#31070)

2024-04-22 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 718af0d7652 Don't use isolated builds when building an sdist for a 
pipeline package supplied in --setup_file option. (#31070)
718af0d7652 is described below

commit 718af0d76528a8b6e6d5dcc99eb3979988960252
Author: tvalentyn 
AuthorDate: Mon Apr 22 14:16:16 2024 -0700

Don't use isolated builds when building an sdist for a pipeline package 
supplied in --setup_file option. (#31070)
---
 sdks/python/apache_beam/runners/portability/stager.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sdks/python/apache_beam/runners/portability/stager.py 
b/sdks/python/apache_beam/runners/portability/stager.py
index f6207d80a9d..48dabe18aa3 100644
--- a/sdks/python/apache_beam/runners/portability/stager.py
+++ b/sdks/python/apache_beam/runners/portability/stager.py
@@ -791,6 +791,7 @@ class Stager(object):
   Stager._get_python_executable(),
   '-m',
   'build',
+  '--no-isolation',  # Otherwise, we need internet access to PyPI.
   '--sdist',
   '--outdir',
   temp_dir,

(beam) branch master updated: fix website embedding issue (#31019)

2024-04-19 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 40ec29ad289 fix website embedding issue (#31019)
40ec29ad289 is described below

commit 40ec29ad289c8ac621a77e40814cb019a37e09df
Author: Svetak Sundhar 
AuthorDate: Fri Apr 19 13:34:04 2024 -0400

fix website embedding issue (#31019)

* fix embedding issue

* include slide links

* fix url for content security
---
 .../site/content/en/get-started/resources/videos-and-podcasts.md  | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git 
a/website/www/site/content/en/get-started/resources/videos-and-podcasts.md 
b/website/www/site/content/en/get-started/resources/videos-and-podcasts.md
index 61f83123d49..8bed7f9b4a9 100644
--- a/website/www/site/content/en/get-started/resources/videos-and-podcasts.md
+++ b/website/www/site/content/en/get-started/resources/videos-and-podcasts.md
@@ -57,7 +57,7 @@ Presented by Davor Bonaci, *Apache Beam PPMC member*
 
 Presented by Dan Halperin, *Apache Beam PPMC member*
 
-https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fatscaleevents%2Fvideos%2F1775945569345206%2F_text=0=560;
 width="560" height="315" style="border:none;overflow:hidden" scrolling="no" 
frameborder="0" allowTransparency="true" allowFullScreen="true">
+[Link to 
Video](https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fatscaleevents%2Fvideos%2F1775945569345206%2F_text=0=560)
 
 
 ### Software Engineering Radio Podcast Episode 272: Apache Beam
@@ -141,7 +141,7 @@ ApacheCon, Miami, 2017
 
 Presented by Etienne Chauchot and Ismaël Mejia, *Apache Beam PMC members*
 
-Nexmark with Beam  by https://www.slid [...]
+[Link to Slides](www.slideshare.net/slideshow/embed_code/key/auWXjEK7GTkiUK)
 
   https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3; 
type="audio/mpeg">
 Your browser does not support the audio element.
@@ -152,7 +152,7 @@ ApacheCon, Montreal, 2018
 
 Presented by Etienne Chauchot, *Apache Beam PMC member*
 
-Universal metrics 
with Apache Beam [...]
+[Link to Slides](www.slideshare.net/slideshow/embed_code/key/kKJRzR8HxkxLsR)
 
   
 Your browser does not support the audio element.
@@ -160,4 +160,4 @@ Your browser does not support the audio element.
 
 ## Next Steps
 
-* Take a self-paced tour through our [Learning 
Resources](/documentation/resources/learning-resources).
+* Take a self-paced tour through our [Learning 
Resources](/documentation/resources/learning-resources).
\ No newline at end of file

(beam) branch release-2.56.0 updated: [release-2.56.0] Exclude broken versions of GRPCIO and upgrade the base image requirements (#31045)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch release-2.56.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.56.0 by this push:
 new 99688da2ba7 [release-2.56.0] Exclude broken versions of GRPCIO and 
upgrade the base image requirements (#31045)
99688da2ba7 is described below

commit 99688da2ba75e239603f0cd031e2d83d0ab761d4
Author: tvalentyn 
AuthorDate: Thu Apr 18 15:31:06 2024 -0700

[release-2.56.0] Exclude broken versions of GRPCIO and upgrade the base 
image requirements (#31045)

* Exclude broken versions of GRPCIO

* Upgrade requirements.
---
 sdks/python/container/py310/base_image_requirements.txt | 8 
 sdks/python/container/py311/base_image_requirements.txt | 8 
 sdks/python/container/py38/base_image_requirements.txt  | 8 
 sdks/python/container/py39/base_image_requirements.txt  | 8 
 sdks/python/setup.py| 6 ++
 5 files changed, 18 insertions(+), 20 deletions(-)

diff --git a/sdks/python/container/py310/base_image_requirements.txt 
b/sdks/python/container/py310/base_image_requirements.txt
index 32bdfa95bc8..980dd99d5b7 100644
--- a/sdks/python/container/py310/base_image_requirements.txt
+++ b/sdks/python/container/py310/base_image_requirements.txt
@@ -50,7 +50,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -78,8 +78,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -137,7 +137,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py311/base_image_requirements.txt 
b/sdks/python/container/py311/base_image_requirements.txt
index 6db63f234ef..f0615b45b29 100644
--- a/sdks/python/container/py311/base_image_requirements.txt
+++ b/sdks/python/container/py311/base_image_requirements.txt
@@ -48,7 +48,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -76,8 +76,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -135,7 +135,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py38/base_image_requirements.txt 
b/sdks/python/container/py38/base_image_requirements.txt
index f59c4004078..c87b4fac4b2 100644
--- a/sdks/python/container/py38/base_image_requirements.txt
+++ b/sdks/python/container/py38/base_image_requirements.txt
@@ -51,7 +51,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -79,8 +79,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -141,7 +141,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.3.2
 scipy==1.10.1
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py39/base_image_requirements.txt 
b/sdks/python/container/py39/base_image_requirements.txt
index 0b4a933e788..bd63ce55de0 100644
--- a/sdks/python/container/py39/base_image_requirements.txt
+++ b/sdks/python/container/py39/base_image_requirements.txt
@@ -50,7 +50,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -78,8 +78,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -138,7 +138,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index ad8923fcc8d..13799dca942 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -365,7 +365,7 @@ if __name__ == '__main__

(beam) branch master updated: Upgrade the version of GRPC to pick up a fix for #30867 (#31044)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new d05196dea8a Upgrade the version of GRPC to pick up a fix for #30867 
(#31044)
d05196dea8a is described below

commit d05196dea8a393ef2afec8e95c49a309375a42d9
Author: tvalentyn 
AuthorDate: Thu Apr 18 15:31:01 2024 -0700

Upgrade the version of GRPC to pick up a fix for #30867 (#31044)

* Exclude broken versions of GRPCIO

* Upgrade requirements.
---
 sdks/python/container/py310/base_image_requirements.txt | 8 
 sdks/python/container/py311/base_image_requirements.txt | 8 
 sdks/python/container/py38/base_image_requirements.txt  | 8 
 sdks/python/container/py39/base_image_requirements.txt  | 8 
 sdks/python/setup.py| 6 ++
 5 files changed, 18 insertions(+), 20 deletions(-)

diff --git a/sdks/python/container/py310/base_image_requirements.txt 
b/sdks/python/container/py310/base_image_requirements.txt
index 32bdfa95bc8..980dd99d5b7 100644
--- a/sdks/python/container/py310/base_image_requirements.txt
+++ b/sdks/python/container/py310/base_image_requirements.txt
@@ -50,7 +50,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -78,8 +78,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -137,7 +137,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py311/base_image_requirements.txt 
b/sdks/python/container/py311/base_image_requirements.txt
index 6db63f234ef..f0615b45b29 100644
--- a/sdks/python/container/py311/base_image_requirements.txt
+++ b/sdks/python/container/py311/base_image_requirements.txt
@@ -48,7 +48,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -76,8 +76,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -135,7 +135,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py38/base_image_requirements.txt 
b/sdks/python/container/py38/base_image_requirements.txt
index f59c4004078..c87b4fac4b2 100644
--- a/sdks/python/container/py38/base_image_requirements.txt
+++ b/sdks/python/container/py38/base_image_requirements.txt
@@ -51,7 +51,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -79,8 +79,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -141,7 +141,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.3.2
 scipy==1.10.1
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/container/py39/base_image_requirements.txt 
b/sdks/python/container/py39/base_image_requirements.txt
index 0b4a933e788..bd63ce55de0 100644
--- a/sdks/python/container/py39/base_image_requirements.txt
+++ b/sdks/python/container/py39/base_image_requirements.txt
@@ -50,7 +50,7 @@ fastavro==1.9.4
 fasteners==0.19
 freezegun==1.4.0
 future==1.0.0
-google-api-core==2.16.2
+google-api-core==2.18.0
 google-api-python-client==2.126.0
 google-apitools==0.5.31
 google-auth==2.29.0
@@ -78,8 +78,8 @@ googleapis-common-protos==1.63.0
 greenlet==3.0.3
 grpc-google-iam-v1==0.13.0
 grpc-interceptor==0.15.4
-grpcio==1.62.1
-grpcio-status==1.62.1
+grpcio==1.62.2
+grpcio-status==1.62.2
 guppy3==3.1.4.post1
 hdfs==2.7.3
 httplib2==0.22.0
@@ -138,7 +138,7 @@ rpds-py==0.18.0
 rsa==4.9
 scikit-learn==1.4.2
 scipy==1.13.0
-shapely==2.0.3
+shapely==2.0.4
 six==1.16.0
 sortedcontainers==2.4.0
 soupsieve==2.5
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index ad8923fcc8d..13799dca942 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -365,7 +365,7 @@ if __name__ == '__main__':
   'cloudpickle~=2.2.1',
   'fastavro>=0.23.6

(beam) branch master updated (bb310e7e907 -> b69e8c615af)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from bb310e7e907 Change type for UnboundedReaderMaxReadTimeSec (#31037)
 add b69e8c615af Updates Python Dev container used by Dataflow (#31029)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch master updated: fix url for content security (#31043)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 70e067e1fde fix url for content security (#31043)
70e067e1fde is described below

commit 70e067e1fdec5c9a3d8914dd7501a784026961f8
Author: Svetak Sundhar 
AuthorDate: Thu Apr 18 14:57:02 2024 -0400

fix url for content security (#31043)
---
 website/www/site/static/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/website/www/site/static/.htaccess 
b/website/www/site/static/.htaccess
index a93e707fdb3..51e28c9a274 100644
--- a/website/www/site/static/.htaccess
+++ b/website/www/site/static/.htaccess
@@ -27,4 +27,4 @@ RedirectMatch "/contribute/release-guide" 
"https://github.com/apache/beam/blob/m
 
 RedirectMatch "/contribute/committer-guide" 
"https://github.com/apache/beam/blob/master/contributor-docs/committer-guide.md;
 
-Header set Content-Security-Policy "frame-src 'self' 
https://play.beam.apache.org/ https://youtube.com/ ;"
+Header set Content-Security-Policy "frame-src 'self' 
https://play.beam.apache.org/ https://www.youtube.com/ ;"

(beam) branch master updated (3822f551dfc -> d6eb576d037)

2024-04-17 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 3822f551dfc remove CLOUDSDK (#31016)
 add d6eb576d037 Allow IFrames to youtube. (#31021)

No new revisions were added by this update.

Summary of changes:
 website/www/site/static/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch master updated (6bca71070e9 -> 6366bd4c890)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 6bca71070e9 Add some metrics for CoGBK profiling. (#30979)
 add 6366bd4c890 Disable profiling in cythonized code by default. (#30938)

No new revisions were added by this update.

Summary of changes:
 CHANGES.md  | 2 +-
 sdks/python/apache_beam/coders/coder_impl.pxd   | 1 -
 sdks/python/apache_beam/metrics/monitoring_infos.py | 1 -
 sdks/python/apache_beam/runners/common.py   | 1 -
 sdks/python/apache_beam/runners/worker/opcounters.py| 1 -
 sdks/python/apache_beam/runners/worker/operations.py| 1 -
 sdks/python/apache_beam/runners/worker/statesampler_fast.pyx| 1 -
 sdks/python/apache_beam/testing/fast_test_utils.pxd | 1 -
 sdks/python/apache_beam/transforms/cy_dataflow_distribution_counter.pxd | 1 -
 sdks/python/apache_beam/transforms/cy_dataflow_distribution_counter.pyx | 1 -
 sdks/python/apache_beam/utils/counters.pxd  | 1 -
 sdks/python/apache_beam/utils/windowed_value.py | 1 -
 12 files changed, 1 insertion(+), 12 deletions(-)

(beam-swift) branch dependabot/swift/github.com/apple/swift-nio-http2-1.28.1 deleted (was 96e2d54)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch 
dependabot/swift/github.com/apple/swift-nio-http2-1.28.1
in repository https://gitbox.apache.org/repos/asf/beam-swift.git


 was 96e2d54  Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam-swift) branch main updated (9032090 -> ab5ed91)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/beam-swift.git


from 9032090  Merge pull request #3 from byronellis/docc-integration
 add 96e2d54  Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1
 new ab5ed91  Merge pull request #4 from 
apache/dependabot/swift/github.com/apple/swift-nio-http2-1.28.1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 Package.resolved | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(beam-swift) 01/01: Merge pull request #4 from apache/dependabot/swift/github.com/apple/swift-nio-http2-1.28.1

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/beam-swift.git

commit ab5ed919d8bbf1cf8f203a1973954c4d39d67cdd
Merge: 9032090 96e2d54
Author: tvalentyn 
AuthorDate: Tue Apr 16 12:56:19 2024 -0700

Merge pull request #4 from 
apache/dependabot/swift/github.com/apple/swift-nio-http2-1.28.1

Bump github.com/apple/swift-nio-http2 from 1.27.0 to 1.28.1

 Package.resolved | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(beam) branch fixsdist deleted (was e1f5000fb47)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch fixsdist
in repository https://gitbox.apache.org/repos/asf/beam.git


 was e1f5000fb47 Fix grep

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated (a6f3ddf898c -> 995d1bf8a5a)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from a6f3ddf898c Improvements to GetWorkTimingInfosTracker when there is 
clock skew between the worker and service. (#30990)
 add 995d1bf8a5a Update the upper bound for google-auth-httplib2 to  
'google-auth-httplib2>=0.1.0,<0.3.0' (#30962)

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/py310/base_image_requirements.txt | 14 +++---
 sdks/python/container/py311/base_image_requirements.txt | 14 +++---
 sdks/python/container/py38/base_image_requirements.txt  | 14 +++---
 sdks/python/container/py39/base_image_requirements.txt  | 14 +++---
 sdks/python/setup.py|  2 +-
 sdks/python/tox.ini |  1 -
 6 files changed, 29 insertions(+), 30 deletions(-)

(beam) branch master updated (63ebda005d0 -> fb6bfc3ef11)

2024-04-15 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 63ebda005d0 Merge pull request #30959 Automatically execute unbounded 
pipelines in streaming mode.
 add fb6bfc3ef11 Skip generating docs for yaml/integration_tests.py. 
(#30986)

No new revisions were added by this update.

Summary of changes:
 sdks/python/scripts/generate_pydoc.sh | 1 +
 1 file changed, 1 insertion(+)

(beam) branch master updated (a00e947f781 -> 4b808b0d4f5)

2024-04-15 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from a00e947f781 Upload PerWorkerMetrics every 30 second instead of every 
10 seconds (#30795)
 add 4b808b0d4f5 Fix build wheels due to setuptools changed naming (#30972)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build_wheels.yml | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

(beam) branch master updated (f8e4330b6ba -> 285b20de1ce)

2024-04-15 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from f8e4330b6ba change grafana creds to use readonly account for 
beammetrics cloudsql db (#30968)
 add 285b20de1ce Restore pip logs also for requirements.txt case (#30936)

No new revisions were added by this update.

Summary of changes:
 CHANGES.md   | 1 +
 sdks/python/container/piputil.go | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

(beam) branch revert-30956-fixsdist deleted (was 63da57a1087)

2024-04-13 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch revert-30956-fixsdist
in repository https://gitbox.apache.org/repos/asf/beam.git


 was 63da57a1087 Revert "Fix sdist (#30956)"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated: Revert setuptools upperbound since buggy releases are yanked. (#30958)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new cd253fd8922 Revert setuptools upperbound since buggy releases are 
yanked. (#30958)
cd253fd8922 is described below

commit cd253fd89220e24345a3921924ae069a63b049af
Author: tvalentyn 
AuthorDate: Fri Apr 12 20:28:48 2024 -0700

Revert setuptools upperbound since buggy releases are yanked. (#30958)

This reverts commit 72a45e4fbd427f0d7ae599777c67eb17ee9636a7.
---
 sdks/python/pyproject.toml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sdks/python/pyproject.toml b/sdks/python/pyproject.toml
index c5f57279c88..32924a92977 100644
--- a/sdks/python/pyproject.toml
+++ b/sdks/python/pyproject.toml
@@ -19,8 +19,7 @@
 # this is the minimum requirements for the build system to execute.
 [build-system]
 requires = [
-# TODO: remove after setuptools bug resolved. 
https://github.com/apache/beam/issues/30955
-"setuptools<69.3.0",
+"setuptools",
 "wheel>=0.36.0",
 "grpcio-tools==1.62.1",
 "mypy-protobuf==3.5.0",

(beam) branch revert-30956-fixsdist created (now 63da57a1087)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch revert-30956-fixsdist
in repository https://gitbox.apache.org/repos/asf/beam.git


  at 63da57a1087 Revert "Fix sdist (#30956)"

This branch includes the following new commits:

 new 63da57a1087 Revert "Fix sdist (#30956)"

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) 01/01: Revert "Fix sdist (#30956)"

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch revert-30956-fixsdist
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 63da57a1087cdbd672b56071e343b9d3c7f7ceb7
Author: tvalentyn 
AuthorDate: Fri Apr 12 16:18:23 2024 -0700

Revert "Fix sdist (#30956)"

This reverts commit 72a45e4fbd427f0d7ae599777c67eb17ee9636a7.
---
 sdks/python/pyproject.toml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sdks/python/pyproject.toml b/sdks/python/pyproject.toml
index c5f57279c88..32924a92977 100644
--- a/sdks/python/pyproject.toml
+++ b/sdks/python/pyproject.toml
@@ -19,8 +19,7 @@
 # this is the minimum requirements for the build system to execute.
 [build-system]
 requires = [
-# TODO: remove after setuptools bug resolved. 
https://github.com/apache/beam/issues/30955
-"setuptools<69.3.0",
+"setuptools",
 "wheel>=0.36.0",
 "grpcio-tools==1.62.1",
 "mypy-protobuf==3.5.0",

(beam) branch master updated: Document how to trigger test suites. (#30953)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new a764632db32 Document how to trigger test suites. (#30953)
a764632db32 is described below

commit a764632db3234784259198dbb3a180c7cc184b80
Author: tvalentyn 
AuthorDate: Fri Apr 12 15:05:30 2024 -0700

Document how to trigger test suites. (#30953)
---
 .github/workflows/README.md | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 2a36fea9ab0..143c287086a 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -21,7 +21,10 @@ Most workflows will get kicked off automatically when you 
open a PR, push code,
 
 If you would like to manually trigger a job, you have 2 options:
 
-1) Trigger Phrases: Many jobs have trigger phrases associated with them (e.g. 
`Run XYZ PreCommit`). These will appear in statuses of previous PR runs of that 
check. You can trigger the job on any PR by commenting that trigger phrase in 
the PR.
+1) Trigger Phrases: Some jobs have trigger phrases associated with them (e.g. 
`Run XYZ PreCommit`). These will appear in statuses of previous PR runs of that 
check. You can trigger the job on any PR by commenting that trigger phrase in 
the PR.
+
+   **Note:** this approach is found not scalable 
([#28909](https://github.com/apache/beam/issues/28909)) and currently only 
enabled for PreCommit workflows. For PostCommit jobs, it is currently replaced 
by a temporary approach: test suites are configured to trigger whenever a 
particular trigger file is modified. Test 
[workflows](https://github.com/apache/beam/tree/master/.github/workflows) have 
[pull_request_target 
paths](https://github.com/apache/beam/blob/e33dec69c7cfd01c0b827538e1dad8 [...]
+
 2) **Committers only** - Manual triggering: Any committer can start any job 
with a 
[workflow_dispatch](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)
 trigger defined (all jobs should have these). To do so, navigate to the 
[Actions tab](https://github.com/apache/beam/actions), click on your desired 
workflow in the left navigation bar, and then click `Run Workflow`.
 
 # Guidelines for Adding or Modifying Workflows
@@ -139,7 +142,7 @@ In order to make it easier for non-committers to interact 
with workflows, workfl
 2) Each job should have the rerun action immediately after its checkout step. 
You can add a step that uses the `setup-action` action in your workflow, which 
encapsulates the checkout and rerun logic in one place. This should be gated on 
the comment trigger (example: 
https://github.com/apache/beam/blob/0ee2dc73ec6f555a5bf1a643dffd37f4927be67e/.github/workflows/beam_PreCommit_Go.yml#L65-L70)
 3) Each job should have a descriptive name that includes the comment trigger 
(example: 
https://github.com/apache/beam/blob/ba8fc935222aeb070668fbafd588bc58e7a21289/.github/workflows/beam_PreCommit_CommunityMetrics.yml#L48)
 
-**Note:** this approach is found not scalable 
([#28909](https://github.com/apache/beam/issues/28909)) and currently only 
enabled for PreCommit workflows. For PostCommit jobs, it is currently replaced 
by a temporary approach of `pull_request_target` trigger with specific path 
`.github/trigger_files/`.
+**Note:** Comment triggering is found not scalable 
([#28909](https://github.com/apache/beam/issues/28909)) and is currently 
limited to a subset of suites. For more information see the [Running Workflows 
Manually](#running-workflows-manually) section.
 
 # Testing new workflows or workflow updates

(beam) branch master updated: Upgrade grpcio-tools. (#30912)

2024-04-11 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 67b3e111ea9 Upgrade grpcio-tools. (#30912)
67b3e111ea9 is described below

commit 67b3e111ea94092fa46fb002650b2687602e2099
Author: tvalentyn 
AuthorDate: Thu Apr 11 15:59:18 2024 -0700

Upgrade grpcio-tools. (#30912)
---
 playground/infrastructure/requirements.txt | 2 +-
 sdks/python/pyproject.toml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/playground/infrastructure/requirements.txt 
b/playground/infrastructure/requirements.txt
index a05ab1d0792..cfbd67145b1 100644
--- a/playground/infrastructure/requirements.txt
+++ b/playground/infrastructure/requirements.txt
@@ -23,7 +23,7 @@ PyYAML==6.0
 tqdm~=4.62.3
 sonora==0.2.2
 pydantic==1.10.2
-grpcio-tools==1.51.1
+grpcio-tools==1.62.1
 protobuf==4.21.12
 google-cloud-datastore==2.11.0
 
diff --git a/sdks/python/pyproject.toml b/sdks/python/pyproject.toml
index 9829671ccb7..32924a92977 100644
--- a/sdks/python/pyproject.toml
+++ b/sdks/python/pyproject.toml
@@ -21,7 +21,7 @@
 requires = [
 "setuptools",
 "wheel>=0.36.0",
-"grpcio-tools==1.53.0",
+"grpcio-tools==1.62.1",
 "mypy-protobuf==3.5.0",
 # Avoid https://github.com/pypa/virtualenv/issues/2006
 "distlib==0.3.7",

(beam) branch dependabot/pip/sdks/python/apache_beam/examples/inference/runinference_metrics/transformers-4.38.0 deleted (was 6b24806314e)

2024-04-11 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch 
dependabot/pip/sdks/python/apache_beam/examples/inference/runinference_metrics/transformers-4.38.0
in repository https://gitbox.apache.org/repos/asf/beam.git


 was 6b24806314e Bump transformers

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated: Bump transformers from 4.36.0 to 4.38.0 in /sdks/python/apache_beam/examples/inference/runinference_metrics (#30924)

2024-04-11 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 067ab8d70d6 Bump transformers from 4.36.0 to 4.38.0 in 
/sdks/python/apache_beam/examples/inference/runinference_metrics (#30924)
067ab8d70d6 is described below

commit 067ab8d70d62f4a205e460a84fe04abe983bf556
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Thu Apr 11 11:31:08 2024 -0700

Bump transformers from 4.36.0 to 4.38.0 in 
/sdks/python/apache_beam/examples/inference/runinference_metrics (#30924)

Bumps [transformers](https://github.com/huggingface/transformers) from 
4.36.0 to 4.38.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- 
[Commits](https://github.com/huggingface/transformers/compare/v4.36.0...v4.38.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] 
<49699333+dependabot[bot]@users.noreply.github.com>
---
 .../python/apache_beam/examples/inference/runinference_metrics/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py 
b/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py
index d1bc0a06c4b..fd2c07cc4e1 100644
--- a/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py
+++ b/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py
@@ -29,7 +29,7 @@ import setuptools
 from setuptools import find_packages
 
 REQUIREMENTS = [
-"apache-beam[gcp]==2.41.0", "transformers==4.36.0", "torch==1.13.1"
+"apache-beam[gcp]==2.41.0", "transformers==4.38.0", "torch==1.13.1"
 ]
 
 setuptools.setup(

(beam) branch master updated: [yaml] disable javascript mapping for python >=3.12 (#30843)

2024-04-09 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 06588745182 [yaml] disable javascript mapping for python >=3.12 
(#30843)
06588745182 is described below

commit 065887451827f5e33d63644fe222dc933fba1147
Author: Jeff Kinard 
AuthorDate: Tue Apr 9 19:54:15 2024 -0400

[yaml] disable javascript mapping for python >=3.12 (#30843)

* [yaml] disable javascript mapping for python >=3.12

Signed-off-by: Jeffrey Kinard 

* use environmental marker for js2py

Signed-off-by: Jeffrey Kinard 

* Update sdks/python/setup.py

-

Signed-off-by: Jeffrey Kinard 
Co-authored-by: tvalentyn 
---
 sdks/python/apache_beam/yaml/yaml_mapping.py  | 26 --
 sdks/python/apache_beam/yaml/yaml_udf_test.py | 10 ++
 sdks/python/setup.py  |  3 ++-
 3 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/sdks/python/apache_beam/yaml/yaml_mapping.py 
b/sdks/python/apache_beam/yaml/yaml_mapping.py
index 4839728dd88..32095fe39f2 100644
--- a/sdks/python/apache_beam/yaml/yaml_mapping.py
+++ b/sdks/python/apache_beam/yaml/yaml_mapping.py
@@ -31,11 +31,6 @@ from typing import Optional
 from typing import TypeVar
 from typing import Union
 
-import js2py
-from js2py import base
-from js2py.constructors import jsdate
-from js2py.internals import simplex
-
 import apache_beam as beam
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.portability.api import schema_pb2
@@ -52,6 +47,14 @@ from apache_beam.yaml import options
 from apache_beam.yaml import yaml_provider
 from apache_beam.yaml.yaml_provider import dicts_to_rows
 
+# Import js2py package if it exists
+try:
+  import js2py
+  from js2py.base import JsObjectWrapper
+except ImportError:
+  js2py = None
+  JsObjectWrapper = object
+
 
 def normalize_mapping(spec):
   """
@@ -87,7 +90,7 @@ def _check_mapping_arguments(
 # js2py's JsObjectWrapper object has a self-referencing __dict__ property
 # that cannot be pickled without implementing the __getstate__ and
 # __setstate__ methods.
-class _CustomJsObjectWrapper(js2py.base.JsObjectWrapper):
+class _CustomJsObjectWrapper(JsObjectWrapper):
   def __init__(self, js_obj):
 super().__init__(js_obj.__dict__['_obj'])
 
@@ -116,6 +119,17 @@ def py_value_to_js_dict(py_value):
 def _expand_javascript_mapping_func(
 original_fields, expression=None, callable=None, path=None, name=None):
 
+  # Check for installed js2py package
+  if js2py is None:
+raise ValueError(
+"Javascript mapping functions are not supported on"
+" Python 3.12 or later.")
+
+  # import remaining js2py objects
+  from js2py import base
+  from js2py.constructors import jsdate
+  from js2py.internals import simplex
+
   js_array_type = (
   base.PyJsArray,
   base.PyJsArrayBuffer,
diff --git a/sdks/python/apache_beam/yaml/yaml_udf_test.py 
b/sdks/python/apache_beam/yaml/yaml_udf_test.py
index 5f5ee1147de..c26d8ec92dd 100644
--- a/sdks/python/apache_beam/yaml/yaml_udf_test.py
+++ b/sdks/python/apache_beam/yaml/yaml_udf_test.py
@@ -29,6 +29,12 @@ from apache_beam.yaml.yaml_mapping import py_value_to_js_dict
 from apache_beam.yaml.yaml_provider import dicts_to_rows
 from apache_beam.yaml.yaml_transform import YamlTransform
 
+try:
+  import js2py
+except ImportError:
+  js2py = None
+  logging.warning('js2py is not installed; some tests will be skipped.')
+
 
 def AsRows():
   return beam.Map(
@@ -55,6 +61,7 @@ class YamlUDFMappingTest(unittest.TestCase):
   def tearDown(self):
 shutil.rmtree(self.tmpdir)
 
+  @unittest.skipIf(js2py is None, 'js2py not installed.')
   def test_map_to_fields_filter_inline_js(self):
 with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
 pickle_library='cloudpickle', yaml_experimental_features=['javascript'
@@ -125,6 +132,7 @@ class YamlUDFMappingTest(unittest.TestCase):
   beam.Row(label='389ax', conductor=390, sum=24),
   ]))
 
+  @unittest.skipIf(js2py is None, 'js2py not installed.')
   def test_filter_inline_js(self):
 with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
 pickle_library='cloudpickle', yaml_experimental_features=['javascript'
@@ -179,6 +187,7 @@ class YamlUDFMappingTest(unittest.TestCase):
   row=beam.Row(rank=2, values=[7, 8, 9])),
   ]))
 
+  @unittest.skipIf(js2py is None, 'js2py not installed.')
   def test_filter_expression_js(self):
 with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
 pickle_library='cloudpickle', yaml_experimental_features=['javascript'
@@ -222,6 +231,7 @@ class YamlUDFMappingTest(unittest.TestCase):

(beam) branch master updated (4452a6c8d97 -> 3c9e9c80e3b)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 4452a6c8d97 Adds a bound on pydantic to exclude incompatible versions 
in the compat test. (#30863)
 add 3c9e9c80e3b Add identity function as default to FlatMap (#30744)

No new revisions were added by this update.

Summary of changes:
 .../{flatmap_simple.py => flatmap_nofunction.py}   | 22 ++
 .../transforms/elementwise/flatmap_test.py |  7 +++
 sdks/python/apache_beam/transforms/core.py | 10 --
 sdks/python/apache_beam/transforms/core_test.py| 11 +++
 .../transforms/python/elementwise/flatmap.md   | 22 +++---
 5 files changed, 51 insertions(+), 21 deletions(-)
 copy 
sdks/python/apache_beam/examples/snippets/transforms/elementwise/{flatmap_simple.py
 => flatmap_nofunction.py} (74%)

(beam) branch master updated: Adds a bound on pydantic to exclude incompatible versions in the compat test. (#30863)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 4452a6c8d97 Adds a bound on pydantic to exclude incompatible versions 
in the compat test. (#30863)
4452a6c8d97 is described below

commit 4452a6c8d9758c9cd3ea39d37aaf149927c633ce
Author: tvalentyn 
AuthorDate: Fri Apr 5 16:43:54 2024 -0700

Adds a bound on pydantic to exclude incompatible versions in the compat 
test. (#30863)

* Add a lower bound on pydantic

* Add an upper bound on pydantic in the respective test suite.

* Restrict deps some more.
---
 sdks/python/tox.ini | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 86d12f31a18..153b6acde8e 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -102,7 +102,7 @@ passenv = 
GIT_*,BUILD_*,ghprb*,CHANGE_ID,BRANCH_NAME,JENKINS_*,CODECOV_*,GITHUB_
 extras = test,gcp,interactive,dataframe,aws
 commands =
   bash {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}" 
"--cov-report=xml --cov=. --cov-append"
-  
+
 [testenv:py38-lint]
 # Don't set TMPDIR to avoid "AF_UNIX path too long" errors in pylint.
 setenv =
@@ -314,6 +314,8 @@ commands =
 [testenv:py{38,39}-tft-{113,114}]
 deps =
   113: tensorflow_transform>=1.13.0,<1.14.0
+  # Help pip resolve conflict with typing-extensions due to an old version of 
tensorflow https://github.com/apache/beam/issues/30852
+  113: pydantic<2.0
   114: tensorflow_transform>=1.14.0,<1.15.0
 commands =
   bash {toxinidir}/scripts/run_pytest.sh {envname} 'apache_beam/ml/transforms 
apache_beam/examples/snippets/transforms/elementwise/mltransform_test.py'
@@ -373,6 +375,8 @@ commands =
 [testenv:py{38,39,310}-tensorflow-212]
 deps =
   212: tensorflow>=2.12rc1,<2.13
+  # Help pip resolve conflict with typing-extensions for old version of TF 
https://github.com/apache/beam/issues/30852
+  212: pydantic<2.7
 extras = test,gcp
 commands =
   # Log tensorflow version for debugging
@@ -444,6 +448,8 @@ commands =
 deps =
   014: tensorflow-hub>=0.14.0,<0.15.0
   015: tensorflow-hub>=0.15.0,<0.16.0
+  # Help pip resolve conflict with typing-extensions due to an old version of 
tensorboard https://github.com/apache/beam/issues/30852
+  015: pydantic<2.7
   tensorflow-text # required to register ops for text embedding models.
 
 extras = test,gcp

(beam) branch master updated (55f5a43eb37 -> 2e630ac3675)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 55f5a43eb37 Remove outdated comment about docker. (#30871)
 add 2e630ac3675 Lower various logging statement levels to clean up example 
printing (#30782)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/internal/gcp/auth.py  | 4 ++--
 .../apache_beam/runners/portability/fn_api_runner/translations.py | 3 ---
 .../apache_beam/runners/portability/fn_api_runner/worker_handlers.py  | 2 +-
 3 files changed, 3 insertions(+), 6 deletions(-)

(beam) branch master updated (a475fdeeb55 -> 6f7c6875368)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from a475fdeeb55 Merge pull request #27284 Yaml API: Day Zero tutorial 
notebook
 add 6f7c6875368 Correct the version spec (#30856)

No new revisions were added by this update.

Summary of changes:
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) branch tvalentyn-patch-3 created (now d7cfc7825d4)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-3
in repository https://gitbox.apache.org/repos/asf/beam.git


  at d7cfc7825d4 remove --pre for testing purposes

This branch includes the following new commits:

 new d7cfc7825d4 remove --pre for testing purposes

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) 01/01: remove --pre for testing purposes

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-3
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d7cfc7825d4c8574a9d1a0048d9f789b03f77c7c
Author: tvalentyn 
AuthorDate: Thu Apr 4 14:03:38 2024 -0700

remove --pre for testing purposes
---
 .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 8be8d73fbcb..e498abbec7e 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -2975,8 +2975,8 @@ class BeamModulePlugin implements Plugin {
 // pip 23.3 is failing due to Hash mismatch between expected SHA 
of the packaged and actual SHA.
 // until it is resolved on pip's side, don't use pip's cache.
 args '-c', ". ${project.ext.envdir}/bin/activate && " +
-"pip install --pre --retries 10 --upgrade pip --no-cache-dir 
&& " +
-"pip install --pre --retries 10 --upgrade tox --no-cache-dir"
+"pip install --retries 10 --upgrade pip --no-cache-dir && " +
+"pip install --retries 10 --upgrade tox --no-cache-dir"
   }
 }
 // Gradle will delete outputs whenever it thinks they are stale. 
Putting a

(beam) 01/01: Correct the version spec

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d0f0359f87c6618d906087ebb8fb83bdbb99f784
Author: tvalentyn 
AuthorDate: Thu Apr 4 12:58:05 2024 -0700

Correct the version spec
---
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 0e49f913fb4..2975f16d40d 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -431,7 +431,7 @@ if __name__ == '__main__':
   'psycopg2-binary>=2.8.5,<3.0.0',
   'testcontainers[mysql]>=3.0.3,<4.0.0',
   'cryptography>=41.0.2',
-  'hypothesis>5.0.0,<=7.0.0',
+  'hypothesis>5.0.0,<7.0.0',
   ],
   'gcp': [
   'cachetools>=3.1.0,<6',

(beam) branch tvalentyn-patch-1 created (now d0f0359f87c)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git


  at d0f0359f87c Correct the version spec

This branch includes the following new commits:

 new d0f0359f87c Correct the version spec

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(beam) branch master updated: Update Beam SDK dependency page and linked material. (#30712)

2024-03-22 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 71a6c029ec7 Update Beam SDK dependency page and linked material. 
(#30712)
71a6c029ec7 is described below

commit 71a6c029ec75c54153c6466f28d50a439bdc5ff4
Author: tvalentyn 
AuthorDate: Fri Mar 22 13:35:33 2024 -0700

Update Beam SDK dependency page and linked material. (#30712)

* Update Beam SDK dependency page and linked material.

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

* Update 
website/www/site/content/en/documentation/sdks/python-dependencies.md

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

-

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>
---
 .../en/documentation/sdks/python-dependencies.md   | 24 ++
 .../site/content/en/get-started/quickstart-py.md   | 13 ++--
 2 files changed, 26 insertions(+), 11 deletions(-)

diff --git 
a/website/www/site/content/en/documentation/sdks/python-dependencies.md 
b/website/www/site/content/en/documentation/sdks/python-dependencies.md
index 09c56adac43..bd834a67941 100644
--- a/website/www/site/content/en/documentation/sdks/python-dependencies.md
+++ b/website/www/site/content/en/documentation/sdks/python-dependencies.md
@@ -18,26 +18,24 @@ limitations under the License.
 
 # Beam SDK for Python dependencies
 
-The Beam SDKs depend on common third-party components which then
-import additional dependencies. Version collisions can result in unexpected
-behavior in the service. If you are using any of these packages in your code, 
be
-aware that some libraries are not forward-compatible and you may need to pin to
-the listed versions that will be in scope during execution.
+This page provides the information about the Apache Beam Python SDK 
dependencies.
 
-Dependencies for your Beam SDK version are listed in `setup.py` in the Beam 
repository. To view them, perform the following steps:
+If your pipeline requires additional dependencies, see [Managing Python 
Pipeline 
Dependencies](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/).
+
+Dependencies of the Apache Beam Python SDK are defined in the `setup.py` file 
in the Beam repository. To view them, take the following steps:
 
 1. Open `setup.py`.
 
 ```
-
https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py
+
https://github.com/apache/beam/blob/release-/sdks/python/setup.py
 ```
 
-Replace `VERSION_NUMBER` with the 
major.minor.patch version of the SDK. For example, https://raw.githubusercontent.com/apache/beam/v{{< param release_latest 
>}}/sdks/python/setup.py" target="_blank" rel="noopener 
noreferrer">https://raw.githubusercontent.com/apache/beam/v{{< param 
release_latest >}}/sdks/python/setup.py will provide the dependencies for 
the {{< param release_latest >}} release.
+Replace `VERSION_NUMBER` with the 
major.minor.patch version of the SDK. For example, https://github.com/apache/beam/blob/release-{{< param release_latest 
>}}/sdks/python/setup.py" target="_blank" rel="noopener 
noreferrer">https://github.com/apache/beam/blob/release-{{< param 
release_latest >}}/sdks/python/setup.py provides the dependencies for the 
{{< param release_latest >}} release.
 
 
 2. Review the core dependency list under `REQUIRED_PACKAGES`.
 
-**Note:** If you require [extra 
features](/get-started/quickstart-py#extra-requirements) such as `gcp` or 
`test`, you should review the lists under `REQUIRED_TEST_PACKAGES`, 
`GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies.
+**Note:** If you need [extra 
features](/get-started/quickstart-py#extra-requirements), such as `gcp` or 
`dataframe`, review the lists in `extras_require` for additional dependencies.
 
 You can also retrieve the dependency list from the command lin

(beam) branch master updated: add flag for direct path that reads from system properties (#30588)

2024-03-14 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 49c7864f0ad add flag for direct path that reads from system properties 
(#30588)
49c7864f0ad is described below

commit 49c7864f0adff1b1ba5019f1811a295e019da878
Author: martin trieu 
AuthorDate: Thu Mar 14 14:20:35 2024 -0700

add flag for direct path that reads from system properties (#30588)

* add flag for direct path that reads from system properties

* remove unused imports

* change flag name to something more unique

* remove instance factory and just use boolean

* fix option name format
---
 .../dataflow/options/DataflowStreamingPipelineOptions.java| 8 
 1 file changed, 8 insertions(+)

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowStreamingPipelineOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowStreamingPipelineOptions.java
index e8396c02726..d9c6c5f1676 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowStreamingPipelineOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowStreamingPipelineOptions.java
@@ -211,6 +211,14 @@ public interface DataflowStreamingPipelineOptions extends 
PipelineOptions {
 
   void setWindmillServiceStreamMaxBackoffMillis(int value);
 
+  @Description(
+  "If true, Dataflow streaming pipeline will be running in direct path 
mode."
+  + " VMs must have IPv6 enabled for this to work.")
+  @Default.Boolean(false)
+  boolean getIsWindmillServiceDirectPathEnabled();
+
+  void setIsWindmillServiceDirectPathEnabled(boolean 
isWindmillServiceDirectPathEnabled);
+
   /**
* Factory for creating local Windmill address. Reads from system propery 
'windmill.hostport' for
* backwards compatibility.

(beam) branch master updated: Document requirements cache behavior differences. (#30493)

2024-03-08 Thread tvalentyn

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 00526dd3fa2 Document requirements cache behavior differences. (#30493)
00526dd3fa2 is described below

commit 00526dd3fa2a8fa9168cdb7f94c86af519f5c678
Author: tvalentyn 
AuthorDate: Fri Mar 8 20:15:04 2024 -0800

Document requirements cache behavior differences. (#30493)

* Revise documentation on managing dependencies.

* Describe the staging of requirements cache behavior.

* Document that setup_file option doesn't stage dependencies.

* Apply suggestions from code review

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>

-

Co-authored-by: Rebecca Szper <98840847+rsz...@users.noreply.github.com>
---
 .../sdks/python-pipeline-dependencies.md| 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git 
a/website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
 
b/website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
index 2261c92eef0..286e90a3705 100644
--- 
a/website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
+++ 
b/website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md
@@ -46,11 +46,21 @@ To supply a requirements.txt file:
 
 The runner will use the `requirements.txt` file to install your additional 
dependencies onto the remote workers.
 
-> **NOTE**: An alternative to `pip freeze` is to use a library like 
[pip-tools](https://github.com/jazzband/pip-tools) to compile all the 
dependencies required for the pipeline from a `--requirements_file`, where only 
top-level dependencies are mentioned.
+> **NOTE**: As an alternative to `pip freeze`, use a library like 
[pip-tools](https://github.com/jazzband/pip-tools) to compile all of the 
dependencies required for the pipeline from a `requirements.in` file. In the 
`requirements.in` file, only the top-level dependencies are mentioned.
+
+When you supply the `--requirements_file` pipeline option,  during pipeline 
submission, Beam downloads
+the specified packages locally into a requirements cache directory,
+and then stages the requirements cache directory to the runner.
+At runtime, when available, Beam installs packages from the requirements cache.
+This mechanism makes it possible to stage the dependency packages to the runner
+at submission. At runtime, the runner workers might be able to install the
+packages from the cache without needing a connection to PyPI. To disable 
staging the
+requirements, use the `--requirements_cache=skip` pipeline option.
+For more information, see the [help descriptions of these pipeline 
options](https://beam.apache.org/releases/pydoc/current/_modules/apache_beam/options/pipeline_options.html#SetupOptions).
 
 ## Custom Containers {#custom-containers}
 
-You can pass a 
[container](https://hub.docker.com/search?q=apache%2Fbeam=image) image 
with all the dependencies that are needed for the pipeline instead of 
`requirements.txt`. [Follow the instructions on how to run pipeline with Custom 
Container images](/documentation/runtime/environments/#running-pipelines).
+You can pass a 
[container](https://hub.docker.com/search?q=apache%2Fbeam=image) image 
with all the dependencies that are needed for the pipeline. [Follow the 
instructions the show how to run the pipeline with custom container 
images](/documentation/runtime/environments/#running-pipelines).
 
 1. If you are using a custom container image, we recommend that you install 
the dependencies from the `--requirements_file` directly into your image at 
build time. In this case, you do not need to pass `--requirements_file` option 
at runtime, which will reduce the pipeline startup time.
 
@@ -118,7 +128,10 @@ Often, your pipeline code spans multiple files. To run 
your project remotely, yo
 
 --setup_file /path/to/setup.py
 
-**Note:** If you [created a requirements.txt file](#pypi-dependencies) and 
your project spans multiple files, you can get rid of the `requirements.txt` 
file and instead, add all packages contained in `requirements.txt` to the 
`install_requires` field of the setup call (in step 1).
+**Note:** It is not necessary to supply the `--requirements_file` 
[option](#pypi-dependencies) if the dependencies of your package are defined in 
the `install_requires` field of the `setup.py` file (see step 1).
+However unlike with the `--requirements_file` option, when you use the 
`--setup_file` option, Beam doesn't stage the dependent packages to the runner.
+Only the pipeline package is staged. If they aren't already provided in the 
runtime environment,
+the package dependencies are installed from PyPI at runtime.
 
 
 ## Non-Python Dependencies or

(beam) branch master updated (a18c6088457 -> 5342a960af2)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from a18c6088457 Do not pre-install beam in tensorrt container (#30552)
 add 5342a960af2 Attempt to install staged Beam SDK if one is available, 
before checking if Beam is installed. (#30565)

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/boot.go | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

(beam) branch master updated: Do not pre-install beam in tensorrt container (#30552)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new a18c6088457 Do not pre-install beam in tensorrt container (#30552)
a18c6088457 is described below

commit a18c608845776416f2e002a3d50c693bd4ab5109
Author: Yi Hu 
AuthorDate: Thu Mar 7 18:07:52 2024 -0500

Do not pre-install beam in tensorrt container (#30552)
---
 .../test-suites/containers/tensorrt_runinference/tensor_rt.dockerfile  | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git 
a/sdks/python/test-suites/containers/tensorrt_runinference/tensor_rt.dockerfile 
b/sdks/python/test-suites/containers/tensorrt_runinference/tensor_rt.dockerfile
index 04638de5077..6d99a6393fa 100644
--- 
a/sdks/python/test-suites/containers/tensorrt_runinference/tensor_rt.dockerfile
+++ 
b/sdks/python/test-suites/containers/tensorrt_runinference/tensor_rt.dockerfile
@@ -22,8 +22,7 @@ ENV PATH="/usr/src/tensorrt/bin:${PATH}"
 
 WORKDIR /workspace
 
-RUN pip install --no-cache-dir apache-beam[gcp]==2.48.0
-COPY --from=apache/beam_python3.8_sdk:2.48.0 /opt/apache/beam /opt/apache/beam
+COPY --from=apache/beam_python3.8_sdk:latest /opt/apache/beam /opt/apache/beam
 
 RUN pip install --upgrade pip \
 && pip install torch>=1.7.1 \

(beam) branch tvalentyn-patch-1 deleted (was b3f9c624dda)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git


 was b3f9c624dda Update image name to the one used by the test.

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(beam) branch master updated (7497495bcf7 -> 9b71fadd57b)

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 7497495bcf7 [Python] Log dependencies installed in submission 
environment (#28564)
 add 9b71fadd57b Update image name to the one used by the test. (#30566)

No new revisions were added by this update.

Summary of changes:
 sdks/python/test-suites/containers/tensorrt_runinference/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(beam) 01/01: Update image name to the one used by the test.

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b3f9c624ddadd3d969a3be768be8a0c54fc3b00d
Author: tvalentyn 
AuthorDate: Thu Mar 7 14:18:07 2024 -0800

Update image name to the one used by the test.
---
 sdks/python/test-suites/containers/tensorrt_runinference/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sdks/python/test-suites/containers/tensorrt_runinference/README.md 
b/sdks/python/test-suites/containers/tensorrt_runinference/README.md
index 18e03ffa5aa..a9dd8d8d71e 100644
--- a/sdks/python/test-suites/containers/tensorrt_runinference/README.md
+++ b/sdks/python/test-suites/containers/tensorrt_runinference/README.md
@@ -21,4 +21,4 @@
 
 This directory contains the Dockerfiles required to run Beam piplines that use 
TensorRT.
 
-To build the image, run `docker build -f tensor_rt.dockerfile -t tensor_rt .`
+To build the image, run `docker build -f tensor_rt.dockerfile -t 
us.gcr.io/apache-beam-testing/python-postcommit-it/tensor_rt:latest .`

(beam) branch tvalentyn-patch-1 created (now b3f9c624dda)