[DISCUSS] Drop support for Flink 1.10
Hello, With Beam support for Flink 1.13 just merged it is the time to discuss the end of support for Flink 1.10 following the agreed policy on supporting only the latest three Flink releases [1]. I would like to propose that for Beam 2.31.0 we stop supporting Flink 1.10 [2]. I prepared a PR for this [3] but of course I wanted to bring the subject here (and to user@) for your attention and in case someone has a different opinion or reason to still support the older version. WDYT? Regards, Ismael [1] https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E [2] https://issues.apache.org/jira/browse/BEAM-12281 [3] https://github.com/apache/beam/pull/14906
Flaky test issue report (40)
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. https://issues.apache.org/jira/browse/BEAM-12322: FnApiRunnerTestWithGrpcAndMultiWorkers flaky (py precommit) (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12309: PubSubIntegrationTest.test_streaming_data_only flake (created 2021-05-07) https://issues.apache.org/jira/browse/BEAM-12307: PubSubBigQueryIT.test_file_loads flake (created 2021-05-07) https://issues.apache.org/jira/browse/BEAM-12303: Flake in PubSubIntegrationTest.test_streaming_with_attributes (created 2021-05-06) https://issues.apache.org/jira/browse/BEAM-12293: FlinkSavepointTest.testSavepointRestoreLegacy flakes due to FlinkJobNotFoundException (created 2021-05-05) https://issues.apache.org/jira/browse/BEAM-12291: org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky (created 2021-05-05) https://issues.apache.org/jira/browse/BEAM-12200: SamzaStoreStateInternalsTest is flaky (created 2021-04-20) https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13) https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27) https://issues.apache.org/jira/browse/BEAM-12019: apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics is flaky (created 2021-03-18) https://issues.apache.org/jira/browse/BEAM-11792: Python precommit failed (flaked?) installing package (created 2021-02-10) https://issues.apache.org/jira/browse/BEAM-11666: apache_beam.runners.interactive.recording_manager_test.RecordingManagerTest.test_basic_execution is flaky (created 2021-01-20) https://issues.apache.org/jira/browse/BEAM-11662: elasticsearch tests failing (created 2021-01-19) https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest flake: network not found (py38 postcommit) (created 2021-01-19) https://issues.apache.org/jira/browse/BEAM-11645: beam_PostCommit_XVR_Flink failing (created 2021-01-15) https://issues.apache.org/jira/browse/BEAM-11541: testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. (created 2020-12-30) https://issues.apache.org/jira/browse/BEAM-11540: Linter sometimes flakes on apache_beam.dataframe.frames_test (created 2020-12-30) https://issues.apache.org/jira/browse/BEAM-10995: Java + Universal Local Runner: WindowingTest.testWindowPreservation fails (created 2020-09-30) https://issues.apache.org/jira/browse/BEAM-10987: stager_test.py::StagerTest::test_with_main_session flaky on windows py3.6,3.7 (created 2020-09-29) https://issues.apache.org/jira/browse/BEAM-10968: flaky test: org.apache.beam.sdk.metrics.MetricsTest$AttemptedMetricTests.testAttemptedDistributionMetrics (created 2020-09-25) https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test flake: Could not find Flink job (created 2020-09-23) https://issues.apache.org/jira/browse/BEAM-10866: PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS (created 2020-09-09) https://issues.apache.org/jira/browse/BEAM-10504: Failure / flake in ElasticSearchIOTest > testWriteFullAddressing and testWriteWithIndexFn (created 2020-07-15) https://issues.apache.org/jira/browse/BEAM-10501: CheckGrafanaStalenessAlerts and PingGrafanaHttpApi fail with Connection refused (created 2020-07-15) https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14) https://issues.apache.org/jira/browse/BEAM-9649: beam_python_mongoio_load_test started failing due to mismatched results (created 2020-03-31) https://issues.apache.org/jira/browse/BEAM-9392: TestStream tests are all flaky (created 2020-02-27) https://issues.apache.org/jira/browse/BEAM-9232: BigQueryWriteIntegrationTests is flaky coercing to Unicode (created 2020-01-31) https://issues.apache.org/jira/browse/BEAM-9119: apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky (created 2020-01-14) https://issues.apache.org/jira/browse/BEAM-8101: Flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for Direct, Spark, Flink (created 2019-08-27) https://issues.apache.org/jira/browse/BEAM-8035: [beam_PreCommit_Java_Phrase] [WatchTest.testMultiplePollsWithManyResults] Flake: Outputs must be in timestamp order (created 2019-08-22) https://issues.apache.org/jira/browse/BEAM-7992: Unhandled type_constraint in
P1 issues report (40)
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. https://issues.apache.org/jira/browse/BEAM-12422: Vendored gRPC 1.36.0 is using a log4j version with security issues (created 2021-05-28) https://issues.apache.org/jira/browse/BEAM-12417: Go Flink and Spark postcommits permared (created 2021-05-27) https://issues.apache.org/jira/browse/BEAM-12416: Python Kafka transforms are failing due to "No Runner was specified" (created 2021-05-27) https://issues.apache.org/jira/browse/BEAM-12396: beam_PostCommit_XVR_Direct failed (flaked?) (created 2021-05-24) https://issues.apache.org/jira/browse/BEAM-12389: beam_PostCommit_XVR_Dataflow flaky: Expand method not found (created 2021-05-21) https://issues.apache.org/jira/browse/BEAM-12387: beam_PostCommit_Python* timing out (created 2021-05-21) https://issues.apache.org/jira/browse/BEAM-12386: beam_PostCommit_Py_VR_Dataflow(_V2) failing metrics tests (created 2021-05-21) https://issues.apache.org/jira/browse/BEAM-12380: Go SDK Kafka IO Transform implemented via XLang (created 2021-05-21) https://issues.apache.org/jira/browse/BEAM-12374: Spark postcommit failing ResumeFromCheckpointStreamingTest (created 2021-05-20) https://issues.apache.org/jira/browse/BEAM-12337: Replace invalid UW container name for Java SDK (created 2021-05-14) https://issues.apache.org/jira/browse/BEAM-12320: PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL PostCommit (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12310: beam_PostCommit_Java_DataflowV2 failing (created 2021-05-07) https://issues.apache.org/jira/browse/BEAM-12279: Implement destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04) https://issues.apache.org/jira/browse/BEAM-12256: PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some Avro logical types (created 2021-04-29) https://issues.apache.org/jira/browse/BEAM-12231: beam_PostRelease_NightlySnapshot failing (created 2021-04-27) https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness hangs when installing pip packages (created 2021-03-11) https://issues.apache.org/jira/browse/BEAM-11906: No trigger early repeatedly for session windows (created 2021-03-01) https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not handle XML encoding per spec (created 2021-02-26) https://issues.apache.org/jira/browse/BEAM-11828: JmsIO is not acknowledging messages correctly (created 2021-02-17) https://issues.apache.org/jira/browse/BEAM-11755: Cross-language consistency (RequiresStableInputs) is quietly broken (at least on portable flink runner) (created 2021-02-05) https://issues.apache.org/jira/browse/BEAM-11578: `dataflow_metrics` (python) fails with TypeError (when int overflowing?) (created 2021-01-06) https://issues.apache.org/jira/browse/BEAM-11434: Expose Spanner admin/batch clients in Spanner Accessor (created 2020-12-10) https://issues.apache.org/jira/browse/BEAM-11148: Kafka commitOffsetsInFinalize OOM on Flink (created 2020-10-28) https://issues.apache.org/jira/browse/BEAM-11017: Timer with dataflow runner can be set multiple times (dataflow runner) (created 2020-10-05) https://issues.apache.org/jira/browse/BEAM-10670: Make non-portable Splittable DoFn the only option when executing Java "Read" transforms (created 2020-08-10) https://issues.apache.org/jira/browse/BEAM-10617: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows (created 2020-07-31) https://issues.apache.org/jira/browse/BEAM-10569: SpannerIO tests don't actually assert anything. (created 2020-07-23) https://issues.apache.org/jira/browse/BEAM-10288: Quickstart documents are out of date (created 2020-06-19) https://issues.apache.org/jira/browse/BEAM-10244: Populate requirements cache fails on poetry-based packages (created 2020-06-11) https://issues.apache.org/jira/browse/BEAM-10100: FileIO writeDynamic with AvroIO.sink not writing all data (created 2020-05-27) https://issues.apache.org/jira/browse/BEAM-9564: Remove insecure ssl options from MongoDBIO (created 2020-03-20) https://issues.apache.org/jira/browse/BEAM-9455: Environment-sensitive provisioning for Dataflow (created 2020-03-05) https://issues.apache.org/jira/browse/BEAM-9293: Python direct runner doesn't emit empty pane when it should (created 2020-02-11) https://issues.apache.org/jira/browse/BEAM-8986: SortValues may not work correct for numerical types (created 2019-12-17)
Re: contributor permission for Beam Jira tickets
Following up on this again.
Thanks in advance
Paresh
On Tue, 25 May, 2021, 3:32 PM Paresh Saraf,
wrote:
> Hi,
>
> This is Paresh Saraf from MongoDB. I'm working on MongoDBIO Connector for
> apache beam. Can someone add me as a contributor for Beam's Jira issue
> tracker? I would like to create/assign tickets for my work.
>
> Thanks,
> Paresh
>
>
>
> --
>
>
> { Name : "Paresh Saraf",
> Title: "Senior Solutions Architect",
> Phone: "+ 91 8095554631",
> Location : "Bengaluru, India"}
>
