dependabot[bot] opened a new pull request, #26: URL: https://github.com/apache/beam-starter-python/pull/26
Bumps [apache-beam](https://github.com/apache/beam) from 2.51.0 to 2.52.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/apache/beam/releases";>apache-beam's releases</a>.</em></p> <blockquote> <h2>Beam 2.52.0 release</h2> <p>We are happy to present the new 2.52.0 release of Beam. This release includes both improvements and new functionality. See the <a href="https://github.com/apache/beam/blob/HEAD/get-started/downloads/#2520-2023-11-17";>download page</a> for this release.</p> <!-- raw HTML omitted --> <p>For more information on changes in 2.52.0, check out the <a href="https://github.com/apache/beam/milestone/16";>detailed release notes</a>.</p> <h2>Highlights</h2> <ul> <li>Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK "core" package. Please, use <code>beam-sdks-java-extensions-avro</code> instead. This will allow to easily update Avro version in user code without potential breaking changes in Beam "core" since the Beam Avro extension already supports the latest Avro versions and should handle this. (<a href="https://redirect.github.com/apache/beam/issues/25252";>#25252</a>).</li> <li>Publishing Java 21 SDK container images now supported as part of Apache Beam release process. (<a href="https://redirect.github.com/apache/beam/issues/28120";>#28120</a>) <ul> <li>Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects.</li> </ul> </li> </ul> <h2>New Features / Improvements</h2> <ul> <li>Add <code>UseDataStreamForBatch</code> pipeline option to the Flink runner. When it is set to true, Flink runner will run batch jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed using the DataSet API.</li> <li><code>upload_graph</code> as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (<a href="https://redirect.github.com/apache/beam/pull/28621";>PR#28621</a>.</li> <li>state amd side input cache has been enabled to a default of 100 MB. Use <code>--max_cache_memory_usage_mb=X</code> to provide cache size for the user state API and side inputs. (Python) (<a href="https://redirect.github.com/apache/beam/issues/28770";>#28770</a>).</li> <li>Beam YAML stable release. Beam pipelines can now be written using YAML and leverage the Beam YAML framework which includes a preliminary set of IO's and turnkey transforms. More information can be found in the YAML root folder and in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/README.md";>README</a>.</li> </ul> <h2>Breaking Changes</h2> <ul> <li><code>org.apache.beam.sdk.io.CountingSource.CounterMark</code> uses custom <code>CounterMarkCoder</code> as a default coder since all Avro-dependent classes finally moved to <code>extensions/avro</code>. In case if it's still required to use <code>AvroCoder</code> for <code>CounterMark</code>, then, as a workaround, a copy of "old" <code>CountingSource</code> class should be placed into a project code and used directly (<a href="https://redirect.github.com/apache/beam/issues/25252";>#25252</a>).</li> <li>Renamed <code>host</code> to <code>firestoreHost</code> in <code>FirestoreOptions</code> to avoid potential conflict of command line arguments (Java) (<a href="https://redirect.github.com/apache/beam/pull/29201";>#29201</a>).</li> </ul> <h2>Bugfixes</h2> <ul> <li>Fixed "Desired bundle size 0 bytes must be greater than 0" in Java SDK's BigtableIO.BigtableSource when you have more cores than bytes to read (Java) <a href="https://redirect.github.com/apache/beam/issues/28793";>#28793</a>.</li> <li><code>watch_file_pattern</code> arg of the <a href="https://github.com/apache/beam/blob/104c10b3ee536a9a3ea52b4dbf62d86b669da5d9/sdks/python/apache_beam/ml/inference/base.py#L997";>RunInference</a> arg had no effect prior to 2.52.0. To use the behavior of arg <code>watch_file_pattern</code> prior to 2.52.0, follow the documentation at <a href="https://beam.apache.org/documentation/ml/side-input-updates/";>https://beam.apache.org/documentation/ml/side-input-updates/</a> and use <code>WatchFilePattern</code> PTransform as a SideInput. (<a href="https://redirect.github.com/apache/beam/pulls/28948";>#28948</a>)</li> <li><code>MLTransform</code> doesn't output artifacts such as min, max and quantiles. Instead, <code>MLTransform</code> will add a feature to output these artifacts as human readable format - <a href="https://redirect.github.com/apache/beam/issues/29017";>#29017</a>. For now, to use the artifacts such as min and max that were produced by the eariler <code>MLTransform</code>, use <code>read_artifact_location</code> of <code>MLTransform</code>, which reads artifacts that were produced earlier in a different <code>MLTransform</code> (<a href="https://redirect.github.com/apache/beam/pull/29016/";>#29016</a>)</li> <li>Fixed a memory leak, which affected some long-running Python pipelines: <a href="https://redirect.github.com/apache/beam/issues/28246";>#28246</a>.</li> </ul> <h2>Security Fixes</h2> <ul> <li>Fixed <a href="https://www.cve.org/CVERecord?id=CVE-2023-39325";>CVE-2023-39325</a> (Java/Python/Go) (<a href="https://redirect.github.com/apache/beam/issues/29118";>#29118</a>).</li> <li>Mitigated <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-47248";>CVE-2023-47248</a> (Python) <a href="https://redirect.github.com/apache/beam/issues/29392";>#29392</a>.</li> </ul> <h2>List of Contributors</h2> <p>According to git shortlog, the following people contributed to the 2.52.0 release. Thank you to all contributors!</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/beam/blob/master/CHANGES.md";>apache-beam's changelog</a>.</em></p> <blockquote> <h1>[2.52.0] - 2023-11-17</h1> <h2>Highlights</h2> <ul> <li>Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK "core" package. Please, use <code>beam-sdks-java-extensions-avro</code> instead. This will allow to easily update Avro version in user code without potential breaking changes in Beam "core" since the Beam Avro extension already supports the latest Avro versions and should handle this. (<a href="https://redirect.github.com/apache/beam/issues/25252";>#25252</a>).</li> <li>Publishing Java 21 SDK container images now supported as part of Apache Beam release process. (<a href="https://redirect.github.com/apache/beam/issues/28120";>#28120</a>) <ul> <li>Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects.</li> </ul> </li> </ul> <h2>New Features / Improvements</h2> <ul> <li>Add <code>UseDataStreamForBatch</code> pipeline option to the Flink runner. When it is set to true, Flink runner will run batch jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed using the DataSet API.</li> <li><code>upload_graph</code> as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (<a href="https://redirect.github.com/apache/beam/pull/28621";>PR#28621</a>.</li> <li>state amd side input cache has been enabled to a default of 100 MB. Use <code>--max_cache_memory_usage_mb=X</code> to provide cache size for the user state API and side inputs. (Python) (<a href="https://redirect.github.com/apache/beam/issues/28770";>#28770</a>).</li> <li>Beam YAML stable release. Beam pipelines can now be written using YAML and leverage the Beam YAML framework which includes a preliminary set of IO's and turnkey transforms. More information can be found in the YAML root folder and in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/README.md";>README</a>.</li> </ul> <h2>Breaking Changes</h2> <ul> <li><code>org.apache.beam.sdk.io.CountingSource.CounterMark</code> uses custom <code>CounterMarkCoder</code> as a default coder since all Avro-dependent classes finally moved to <code>extensions/avro</code>. In case if it's still required to use <code>AvroCoder</code> for <code>CounterMark</code>, then, as a workaround, a copy of "old" <code>CountingSource</code> class should be placed into a project code and used directly (<a href="https://redirect.github.com/apache/beam/issues/25252";>#25252</a>).</li> <li>Renamed <code>host</code> to <code>firestoreHost</code> in <code>FirestoreOptions</code> to avoid potential conflict of command line arguments (Java) (<a href="https://redirect.github.com/apache/beam/pull/29201";>#29201</a>).</li> </ul> <h2>Bugfixes</h2> <ul> <li>Fixed "Desired bundle size 0 bytes must be greater than 0" in Java SDK's BigtableIO.BigtableSource when you have more cores than bytes to read (Java) <a href="https://redirect.github.com/apache/beam/issues/28793";>#28793</a>.</li> <li><code>watch_file_pattern</code> arg of the <a href="https://github.com/apache/beam/blob/104c10b3ee536a9a3ea52b4dbf62d86b669da5d9/sdks/python/apache_beam/ml/inference/base.py#L997";>RunInference</a> arg had no effect prior to 2.52.0. To use the behavior of arg <code>watch_file_pattern</code> prior to 2.52.0, follow the documentation at <a href="https://beam.apache.org/documentation/ml/side-input-updates/";>https://beam.apache.org/documentation/ml/side-input-updates/</a> and use <code>WatchFilePattern</code> PTransform as a SideInput. (<a href="https://redirect.github.com/apache/beam/pulls/28948";>#28948</a>)</li> <li><code>MLTransform</code> doesn't output artifacts such as min, max and quantiles. Instead, <code>MLTransform</code> will add a feature to output these artifacts as human readable format - <a href="https://redirect.github.com/apache/beam/issues/29017";>#29017</a>. For now, to use the artifacts such as min and max that were produced by the eariler <code>MLTransform</code>, use <code>read_artifact_location</code> of <code>MLTransform</code>, which reads artifacts that were produced earlier in a different <code>MLTransform</code> (<a href="https://redirect.github.com/apache/beam/pull/29016/";>#29016</a>)</li> <li>Fixed a memory leak, which affected some long-running Python pipelines: <a href="https://redirect.github.com/apache/beam/issues/28246";>#28246</a>.</li> </ul> <h2>Security Fixes</h2> <ul> <li>Fixed <a href="https://www.cve.org/CVERecord?id=CVE-2023-39325";>CVE-2023-39325</a> (Java/Python/Go) (<a href="https://redirect.github.com/apache/beam/issues/29118";>#29118</a>).</li> <li>Mitigated <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-47248";>CVE-2023-47248</a> (Python) <a href="https://redirect.github.com/apache/beam/issues/29392";>#29392</a>.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/beam/commit/7c8a99779bac68dfadbd7077fe6cbb76d26d04b1";><code>7c8a997</code></a> Set version for 2.52.0 RC5</li> <li><a href="https://github.com/apache/beam/commit/55e7d199a6e6c8475f13da9dc65f7c3c7dcb6441";><code>55e7d19</code></a> Mark Avro as provided in harness JAR (<a href="https://redirect.github.com/apache/beam/issues/29412";>#29412</a>) (<a href="https://redirect.github.com/apache/beam/issues/29415";>#29415</a>)</li> <li><a href="https://github.com/apache/beam/commit/d9248c8ddd1588acfe2c643b8170cd72b63b0bbd";><code>d9248c8</code></a> Fix MongoDBIO ignore SSL KeyStore not initialized (<a href="https://redirect.github.com/apache/beam/issues/29403";>#29403</a>)</li> <li><a href="https://github.com/apache/beam/commit/8b6ae4a4c351666deb4b49ecda8822c3e2f578b8";><code>8b6ae4a</code></a> Invoke pyarrow_hotfix to alleviate concerns due to CVE-2023-47248. (<a href="https://redirect.github.com/apache/beam/issues/29402";>#29402</a>)</li> <li><a href="https://github.com/apache/beam/commit/0fdf404873636d24be50ae8360a08e4dddfae679";><code>0fdf404</code></a> Revert "Support DatabaseID in Datastore beam connector (<a href="https://redirect.github.com/apache/beam/issues/27987";>#27987</a>)" (<a href="https://redirect.github.com/apache/beam/issues/29318";>#29318</a>)</li> <li><a href="https://github.com/apache/beam/commit/91238ee34f0e94cffff372c233a2b5b63fbbd97c";><code>91238ee</code></a> fix typo in workflow (<a href="https://redirect.github.com/apache/beam/issues/29300";>#29300</a>) (<a href="https://redirect.github.com/apache/beam/issues/29301";>#29301</a>)</li> <li><a href="https://github.com/apache/beam/commit/5b5289c23b0020f191f9573e7c417d6d505f05c6";><code>5b5289c</code></a> Declare signMavenJavaPublication dependency on copyPom (<a href="https://redirect.github.com/apache/beam/issues/29298";>#29298</a>)</li> <li><a href="https://github.com/apache/beam/commit/e008971bb8b512fb39f04ef568c2c8f4951adcd2";><code>e008971</code></a> [YAML] Fix PyPi caching for non-dev beam (<a href="https://redirect.github.com/apache/beam/issues/29292";>#29292</a>)</li> <li><a href="https://github.com/apache/beam/commit/058c94738b5657d92e0f93ecba55077624e9cda9";><code>058c947</code></a> [YAML] fix renaming provider caching and YAML schema validation (<a href="https://redirect.github.com/apache/beam/issues/29290";>#29290</a>)</li> <li><a href="https://github.com/apache/beam/commit/0e8e54c29c4b4eef0220d88c2b198dbb5e748b72";><code>0e8e54c</code></a> [YAML] Fix error handling for KafkaSchemaTransforms (<a href="https://redirect.github.com/apache/beam/issues/29261";>#29261</a>) (<a href="https://redirect.github.com/apache/beam/issues/29289";>#29289</a>)</li> <li>Additional commits viewable in <a href="https://github.com/apache/beam/compare/v2.51.0...v2.52.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
