Re: [RFC] Throttle Time Counters
Thanks for writing this. +1 for standardizing (and documenting) these metrics. - Cham On Thu, Jul 27, 2023 at 1:51 PM Yasha Ravindra via dev wrote: > Hello everyone, > > Throttle time counters were introduced to give clients the option to self > regulate when the service is overwhelmed by requests. > Currently, we have different namespaces for each IO and languages. We > would instead like to have a dedicated namespace for this counter. > We have put together a proposal for the same and would appreciate any > feedback or comments. > > > https://docs.google.com/document/d/1hUufb3L5jURGeFLaQKeQbPlYK-B2wbaLHtqNVETSOOk/edit?usp=sharing > > > Thank you > > Warm Regards, > Yasha Ravindra >
Re: [DISCUSS] Upgrade vended guava version
It's not designed to be merged, but there is a PR with this change in it--- https://github.com/apache/beam/pull/27695/files#diff-0435a83a413ec063bf7e682cadcd56776cd18fc878f197cc99a65fc231ef2047 On Mon, Jul 31, 2023 at 10:07 AM Ahmet Altay via dev wrote: > Hi Hong, > > Thank you for reaching out and thank you for offering to help. If you can > start the PR and do the testing, one of the committers could help with the > process. > > Thank you! > Ahmet > > On Mon, Jul 31, 2023 at 9:13 AM Hong Teoh wrote: > >> Hi all, >> >> The current version of guava that is vended in Beam is >> com.google.guava:guava:26.0-jre. >> >> This version is really old, and has active vulnerabilities [1] [2] >> [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre >> [2] CVE-2023-2976 >> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 >> [3] CVE-2020-8908 >> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 >> >> Is there anyone else keen on upgrading the vended guava version to match >> the guava version of 32.1.1-jre ? [4] >> [4] >> https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 >> >> I am happy to contribute the PR to upgrade the guava dependencies in the >> Beam repository, but I would need a committer to drive the release of the >> vended version first! [5] >> [5] >> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog >> >> >> Side question: Does anyone know why we have libraries that use the >> non-vended guava version? [6] >> [6] >> https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code >> > > @Kenneth Knowles - might know. > > >> >> >> >> Regards, >> Hong >> >
Re: [DISCUSS] Upgrade vended guava version
Hi Hong, Thank you for reaching out and thank you for offering to help. If you can start the PR and do the testing, one of the committers could help with the process. Thank you! Ahmet On Mon, Jul 31, 2023 at 9:13 AM Hong Teoh wrote: > Hi all, > > The current version of guava that is vended in Beam is > com.google.guava:guava:26.0-jre. > > This version is really old, and has active vulnerabilities [1] [2] > [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre > [2] CVE-2023-2976 > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 > [3] CVE-2020-8908 > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 > > Is there anyone else keen on upgrading the vended guava version to match > the guava version of 32.1.1-jre ? [4] > [4] > https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 > > I am happy to contribute the PR to upgrade the guava dependencies in the > Beam repository, but I would need a committer to drive the release of the > vended version first! [5] > [5] > https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog > > > Side question: Does anyone know why we have libraries that use the > non-vended guava version? [6] > [6] > https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code > @Kenneth Knowles - might know. > > > > Regards, > Hong >
[DISCUSS] Upgrade vended guava version
Hi all, The current version of guava that is vended in Beam is com.google.guava:guava:26.0-jre. This version is really old, and has active vulnerabilities [1] [2] [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre [2] CVE-2023-2976 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 [3] CVE-2020-8908 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 Is there anyone else keen on upgrading the vended guava version to match the guava version of 32.1.1-jre ? [4] [4] https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 I am happy to contribute the PR to upgrade the guava dependencies in the Beam repository, but I would need a committer to drive the release of the vended version first! [5] [5] https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog Side question: Does anyone know why we have libraries that use the non-vended guava version? [6] [6] https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code Regards, Hong
[DISCUSS] Upgrade vended guava version
Hi all, The current version of guava that is vended in Beam is com.google.guava:guava:26.0-jre. This version is really old, and has active vulnerabilities [1] [2] [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre [2] CVE-2023-2976 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 [3] CVE-2020-8908 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 Is there anyone else keen on upgrading the vended guava version to match the guava version of 32.1.1-jre ? [4] [4] https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 I am happy to contribute the PR to upgrade the guava dependencies in the Beam repository, but I would need a committer to drive the release of the vended version first! [5] [5] https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog Side question: Does anyone know why we have libraries that use the non-vended guava version? [6] [6] https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code Regards, Hong
Upgrade vended guava version
Hi all, The current version of guava that is vended in Beam is com.google.guava:guava:26.0-jre. This version is really old, and has active vulnerabilities [1] [2] [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre [2] CVE-2023-2976 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 [3] CVE-2020-8908 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 Is there anyone else keen on upgrading the vended guava version to match the guava version of 32.1.1-jre ? [4] [4] https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 I am happy to contribute the PR to upgrade the guava dependencies in the Beam repository, but I would need a committer to drive the release of the vended version first! [5] [5] https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog Side question: Does anyone know why we have libraries that use the non-vended guava version? [6] [6] https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code Regards, Hong
[DISCUSS] Upgrade vended guava version
Hi all, The current version of guava that is vended in Beam is com.google.guava:guava:26.0-jre. This version is really old, and has active vulnerabilities [1] [2] [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre [2] CVE-2023-2976 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 [3] CVE-2020-8908 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 Is there anyone else keen on upgrading the vended guava version to match the guava version of 32.1.1-jre ? [4] [4] https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542 I am happy to contribute the PR to upgrade the guava dependencies in the Beam repository, but I would need a committer to drive the release of the vended version first! [5] [5] https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog Side question: Does anyone know why we have libraries that use the non-vended guava version? [6] [6] https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code Regards, Hong
Beam High Priority Issue Report (38)
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/27648 [Bug]: Python SDFs (e.g. PeriodicImpulse) running in Flink and polling using tracker.defer_remainder have checkpoint size growing indefinitely https://github.com/apache/beam/issues/27616 [Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore with inequality filters https://github.com/apache/beam/issues/27314 [Failing Test]: bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1] https://github.com/apache/beam/issues/27312 [Bug]: JmsIO create connection based on the number of threads https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag when using Kafka and GroupByKey on Dataflow Runner https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to SchemaCoder after upgrading to 2.48 https://github.com/apache/beam/issues/26969 [Failing Test]: Python PostCommit is failing due to exceeded rate limits https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested ROW (described below) https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read not reading all rows when set --setEnableBundling=true https://github.com/apache/beam/issues/26343 [Bug]: apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is flaky https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not propagate a Coder to AvroSource https://github.com/apache/beam/issues/26041 [Bug]: Unable to create exactly-once Flink pipeline with stream source and file sink https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in FlinkRunner leads to a data loss https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK Harness ProcessBundleProgress https://github.com/apache/beam/issues/24389 [Failing Test]: HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError ContainerFetchException https://github.com/apache/beam/issues/24313 [Flaky]: apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder https://github.com/apache/beam/issues/23944 beam_PreCommit_Python_Cron regularily failing - test_pardo_large_input flaky https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder will drop message id and orderingKey https://github.com/apache/beam/issues/22913 [Bug]: beam_PostCommit_Java_ValidatesRunner_Flink is flakes in org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it https://github.com/apache/beam/issues/21714 PulsarIOTest.testReadFromSimpleTopic is very flaky https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing consistently https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit test action StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial (order 1000 elements) numpy input flakes in non-cython environment https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table destinations returns wrong tableId https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: Connection refused https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) failing: ParDoTest$TimestampTests/OnWindowExpirationTests https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not follow spec https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit data at GC time https://github.com/apache/beam/issues/21121 apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it flakey https://github.com/apache/beam/issues/21104 Flaky: apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers https://github.com/apache/beam/issues/20976 apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics is flaky https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit empty pane when it should https://github.com/apache/beam/issues/19814 Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
