Re: [DISCUSS] Migrate Jira to GitHub Issues?
Hey all, I summarized the discussion in this document[1]. IMO a lot of the concerns raised can be worked around (multiple milestones, components, tags, sub-issues), while the biggest benefit will be decreasing the barrier for new users to contribute and have better discoverability and linkage between code, issues and PRs. Please assign your priority levels for the various features in the comparison table. I left it out because I have a clear bias here : ) Next steps would be to decide whether (1) to move, and (2) to copy over JIRA issues. IMO, Airflow's approach to not copy everything will be the right choice. [1] https://docs.google.com/document/d/1_n7gboVbSKPs-CVcHzADgg8qpNL9igiHqUPCmiOslf0/edit# On Fri, Jan 28, 2022 at 2:30 PM Brian Hulette wrote: > Thanks for volunteering to pick this up Aizhamal, while I'm interested in > this change happening I don't have the bandwidth to push it along. > > I think there was another point where we're missing consensus: how would > we deal with existing jiras. Do we write some automation to port > everything, or just flip the switch and encourage users/devs to port active > jiras over to GitHub? > > Manual porting pros: > - Ambiguous situations get human attention. > - Tickets with no interested parties will be implicitly cleared out of the > backlog. > - No need to write automation for porting tools. > Manual porting cons: > - Unambiguous situations get (unnecessary) human attention. > > A compromise might be to build a simple tool for porting jiras, but don't > automatically run it on everything. > > On Tue, Jan 18, 2022 at 6:04 AM Kenneth Knowles wrote: > >> I also think that we are at the point where a document describing them >> side-by-side is needed. I would very much like to help. I strongly support >> moving to GitHub Issues. >> >> I'm less concerned about pros/cons (I think the one big pro of "everyone >> knows it and already has an account" outweighs almost any con) but I want >> to build a very clear plan of how we will map Jira features to GitHub >> features. I use quite a lot of Jira's features. In particular, a lot of >> things seem like they'll become conventions around labels, which I expect >> to often be low enough data quality that we would just not bother, unless >> we can control it a bit. >> >> I eagerly await the link! Feel free to share very early :-) >> >> Kenn >> >> On Thu, Jan 13, 2022 at 1:48 PM Aizhamal Nurmamat kyzy < >> aizha...@apache.org> wrote: >> >>> I think I am enthusiastic enough to help with the doc :) will share the >>> link soon. >>> >>> On Thu, Jan 13, 2022 at 10:12 AM Robert Bradshaw >>> wrote: >>> I don't know if we have consensus, but it seems that some people are quite supportive (myself included), and some are ambivalent. The only major con I can see is that github doesn't support tagging an issue to multiple milestones (but it's unclear how important that is). I would suggest that someone enthusiastic about this proposal put together a doc where we can enumerate the pros and cons and once the list seems complete we can bring it back to the list for further discussion and/or a vote (if needed, likely not). On Thu, Jan 13, 2022 at 9:27 AM Alexey Romanenko wrote: > > I’m not sure that we have a consensus on this. Since this thread initially was started to discuss and gather some feedback then I think it would be great to have a summary with pros and cons of this migration. > > — > Alexey > > On 13 Jan 2022, at 00:11, Aizhamal Nurmamat kyzy wrote: > > Hi all, > > Is there a consensus to migrate to GitHub? > > On Wed, Dec 15, 2021 at 9:17 AM Brian Hulette wrote: >> >> >> >> On Tue, Dec 14, 2021 at 1:14 PM Kenneth Knowles wrote: >>> >>> >>> >>> On Thu, Dec 9, 2021 at 11:50 PM Jean-Baptiste Onofre < j...@nanthrax.net> wrote: Hi, No problem for me. The only thing I don’t like with GitHub issues is that fact that it’s not possible to “assign” several milestones to an issue. When we maintain several active branch/version, it sucks (one issue == one milestone), as we have to create several issue. >>> >>> >>> This is a good point to consider. In Beam we often create multiple issues anyhow when we intend to backport/cherrypick a fix. One issue for the original fix and one each targeted cherrypick. This way their resolution status can be tracked separately. But it is nice for users to be able to go back and edit the original bug report to say which versions are affected and which are not. >> >> >> I looked into this a little bit. It looks like milestones don't have to represent a release (e.g. they could represent some abstract goal), but they are often associated with releases. This seems like a reas
Flaky test issue report (43)
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. https://issues.apache.org/jira/browse/BEAM-13741: :sdks:java:extensions:sql:hcatalog:compileJava failing in beam_Release_NightlySnapshot (created 2022-01-25) https://issues.apache.org/jira/browse/BEAM-13708: flake: FlinkRunnerTest.testEnsureStdoutStdErrIsRestored (created 2022-01-20) https://issues.apache.org/jira/browse/BEAM-13693: beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timing out at 9 hours (created 2022-01-19) https://issues.apache.org/jira/browse/BEAM-13575: Flink testParDoRequiresStableInput flaky (created 2021-12-28) https://issues.apache.org/jira/browse/BEAM-13525: Java VR (Dataflow, V2, Streaming) failing: ParDoTest$TimestampTests/OnWindowExpirationTests (created 2021-12-22) https://issues.apache.org/jira/browse/BEAM-13519: Java precommit flaky (timing out) (created 2021-12-22) https://issues.apache.org/jira/browse/BEAM-13500: NPE in Flink Portable ValidatesRunner streaming suite (created 2021-12-21) https://issues.apache.org/jira/browse/BEAM-13453: Flake in org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use (created 2021-12-13) https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is failing (created 2021-12-07) https://issues.apache.org/jira/browse/BEAM-13367: [beam_PostCommit_Python36] [ apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary (created 2021-12-01) https://issues.apache.org/jira/browse/BEAM-13312: org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle is flaky in Java Spark ValidatesRunner suite (created 2021-11-23) https://issues.apache.org/jira/browse/BEAM-13311: org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful is flaky in Java ValidatesRunner Flink suite. (created 2021-11-23) https://issues.apache.org/jira/browse/BEAM-13234: Flake in StreamingWordCountIT.test_streaming_wordcount_it (created 2021-11-12) https://issues.apache.org/jira/browse/BEAM-13025: pubsublite.ReadWriteIT flaky in beam_PostCommit_Java_DataflowV2 (created 2021-10-08) https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 - CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21) https://issues.apache.org/jira/browse/BEAM-12859: org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer is flaky (created 2021-09-08) https://issues.apache.org/jira/browse/BEAM-12858: org.apache.beam.sdk.io.gcp.datastore.RampupThrottlingFnTest.testRampupThrottler is flaky (created 2021-09-08) https://issues.apache.org/jira/browse/BEAM-12809: testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26) https://issues.apache.org/jira/browse/BEAM-12794: PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12793: beam_PostRelease_NightlySnapshot failed (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16) https://issues.apache.org/jira/browse/BEAM-12673: apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it flakey (created 2021-07-28) https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking in PipelineOptionsTest.test_display_data (created 2021-06-18) https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: Failed to read inputs in the data plane (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12320: PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL PostCommit (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12291: org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky (created 2021-05-05) https://issues.apache.org/jira/browse/BEAM-12200: SamzaStoreStateInternalsTest is flaky (created 2021-04-20) https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13) https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27) https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: "Memory constraints are impeding performance" (created 2021-02-18) https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest flake: network not found (py38 p
P1 issues report (67)
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. https://issues.apache.org/jira/browse/BEAM-13769: beam_PreCommit_Python_Cron failing on test_create_uses_coder_for_pickling (created 2022-01-28) https://issues.apache.org/jira/browse/BEAM-13763: Rotate credentials for 'io-datastores' Kubernetes cluster (created 2022-01-28) https://issues.apache.org/jira/browse/BEAM-13741: :sdks:java:extensions:sql:hcatalog:compileJava failing in beam_Release_NightlySnapshot (created 2022-01-25) https://issues.apache.org/jira/browse/BEAM-13715: Kafka commit offset drop data on failure for runners that have non-checkpointing shuffle (created 2022-01-21) https://issues.apache.org/jira/browse/BEAM-13694: beam_PostCommit_Java_Hadoop_Versions failing with ClassDefNotFoundError (created 2022-01-19) https://issues.apache.org/jira/browse/BEAM-13693: beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timing out at 9 hours (created 2022-01-19) https://issues.apache.org/jira/browse/BEAM-13686: OOM while logging a large pipeline even when logging level is higher (created 2022-01-19) https://issues.apache.org/jira/browse/BEAM-13668: Java Spanner IO Request Count metrics broke backwards compatibility (created 2022-01-15) https://issues.apache.org/jira/browse/BEAM-13615: Bumping up FnApi environment version to 9 in Java, Python SDK (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13606: bigtable io doesn't handle non-ok row mutations (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13598: Install Java 17 on Jenkins VM (created 2022-01-04) https://issues.apache.org/jira/browse/BEAM-13582: Beam website precommit mentions broken links, but passes. (created 2021-12-30) https://issues.apache.org/jira/browse/BEAM-13579: Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0 (created 2021-12-29) https://issues.apache.org/jira/browse/BEAM-13487: WriteToBigQuery Dynamic table destinations returns wrong tableId (created 2021-12-17) https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is failing (created 2021-12-07) https://issues.apache.org/jira/browse/BEAM-13376: Missing error for nonexistent column family BigTable (created 2021-12-03) https://issues.apache.org/jira/browse/BEAM-13237: org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView flaky on Dataflow Runner V2 (created 2021-11-12) https://issues.apache.org/jira/browse/BEAM-13203: Potential data loss when using SnsIO.writeAsync (created 2021-11-08) https://issues.apache.org/jira/browse/BEAM-13164: Race between member variable being accessed due to leaking uninitialized state via OutboundObserverFactory (created 2021-11-01) https://issues.apache.org/jira/browse/BEAM-13132: WriteToBigQuery submits a duplicate BQ load job if a 503 error code is returned from googleapi (created 2021-10-27) https://issues.apache.org/jira/browse/BEAM-13087: apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible (created 2021-10-20) https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does not emit data at GC time (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll do not follow spec (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files (created 2021-10-06) https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with random prefix (created 2021-10-04) https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in CombinePerKey operation (created 2021-09-26) https://issues.apache.org/jira/browse/BEAM-12867: Either Create or DirectRunner fails to produce all elements to the following transform (created 2021-09-09) https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) Bricked Dataflow Pipeline (created 2021-09-06) https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect pipeline proto when core-construction-java jar is not in the CLASSPATH (created 2021-08-26) https://issues.apache.org/jira/browse/BEAM-12799: "Java IO IT Tests" - missing data in grafana (created 2021-08-25) https://issues.apache.org/jira/browse/BEAM-12792: Multiple jobs running on Flink session cluster reuse the persistent Python environment. (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12621: Update Jenkins VMs to modern Ubun
Re: [VOTE] Release 2.36.0, release candidate #1
+1 (binding) I tried a couple more pipelines and flag combinations I particularly care about :-) Kenn On Fri, Jan 28, 2022 at 10:03 PM Valentyn Tymofieiev wrote: > +1 (non-binding). Verified that Dataflow containers have necessary Beam > requirements. > > On Fri, Jan 28, 2022 at 3:16 PM Robert Bradshaw > wrote: > >> +1 (binding) The artifacts and signatures look good to me. >> >> On Fri, Jan 28, 2022 at 12:08 PM Jack McCluskey >> wrote: >> > >> > +1 (non-binding) - validated Go quickstart for direct and flink >> > >> > On Fri, Jan 28, 2022 at 3:07 PM Andy Ye wrote: >> >> >> >> +1 (non-binding) - Validated batch Python quickstart for direct and >> dataflow >> >> >> >> On Fri, Jan 28, 2022 at 1:03 PM Ahmet Altay wrote: >> >>> >> >>> +1 (binding) >> >>> >> >>> I validated the python quickstarts. Thank you Emily! >> >>> >> >>> On Wed, Jan 26, 2022 at 6:07 PM Emily Ye wrote: >> >> From: emil...@apache.org >> To: dev@beam.apache.org >> Subject: [VOTE] Release 2.36.0, release candidate #1 >> >> Hi everyone, >> Please review and vote on the release candidate #1 for the version >> 2.36.0 as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> >> Reviewers are encouraged to test their own use cases with the >> release candidate, and vote +1 if no issues are found. >> >> The complete staging area is available for your review, which >> includes: >> * JIRA release notes [1], >> * the official Apache source release to be deployed to >> dist.apache.org [2], which is signed with the key with fingerprint >> 730D5E8D4CA810C1F01D304B610B38AA1BE11656 [3], >> * all artifacts to be deployed to the Maven Central Repository [4], >> * source code tag "v2.36.0-RC1" [5], >> * website pull request listing the release [6], the blog post [6], >> and publishing the API reference manual [7]. >> * Java artifacts were built with Gradle 7.3.2 and OpenJDK 1.8.0_181 >> * Python artifacts are deployed along with the source release to the >> dist.apache.org [2] and PyPI[8]. >> * Validation sheet with a tab for 2.36.0 release to help with >> validation [9]. >> * Docker images published to Docker Hub [10]. >> >> The vote will be open for at least 72 hours. It is adopted by >> majority approval, with at least 3 PMC affirmative votes. >> >> For guidelines on how to try the release in your projects, check out >> our blog post at https://beam.apache.org/blog/validate-beam-release/. >> >> Thanks, >> Release Manager >> >> [1] >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350407 >> [2] https://dist.apache.org/repos/dist/dev/beam/2.36.0/ >> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >> [4] >> https://repository.apache.org/content/repositories/orgapachebeam-1248/ >> [5] https://github.com/apache/beam/tree/v2.36.0-RC1 >> [6] https://github.com/apache/beam/pull/16627 >> [7] https://github.com/apache/beam-site/pull/622 >> [8] https://pypi.org/project/apache-beam/2.36.0rc1/ >> [9] >> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit >> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image >> >
