Beam Dependency Check Report (2022-09-15)

[Apache Jenkins Server]

<<< text/html; charset=UTF-8: Unrecognized >>>

Re: [Infrastructure] Periodically run Java microbenchmarks on Jenkins

[Brian Hulette via dev]

Is there somewhere we could document this?

On Thu, Sep 15, 2022 at 6:45 AM Moritz Mack  wrote:

> Thank you, Andrew!
>
> Exactly what I was looking for, that’s awesome!
>
>
>
> On 15.09.22, 06:37, "Alexey Romanenko"  wrote:
>
>
>
>
>
> Ahh, great! I didn’t know that 'beam-perf’ label is used for that.
> Thanks!
>
> > On 14 Sep 2022, at 17:47, Andrew Pilloud  wrote:
> >
> > We do have a dedicated machine for benchmarks. This is a single
> > machine limited to running one test at a time. Set the
> > jenkinsExecutorLabel for the job to 'beam-perf' to use it. For
> > example:
> >
> https://urldefense.com/v3/__https://github.com/apache/beam/blob/66bbee84ed477d86008905646e68b100591b6f78/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy*L36__;Iw!!CiXD_PY!Qat2J4NAyHVo4Cc32PKMn50yw8LgWHmEOm4Ltb7aRV-7KCfNamu3tGOiSYKDUZhLHKu3zlqbBXzJNiX_f_Qteg$
> 
>
> >
> > Andrew
> >
> > On Wed, Sep 14, 2022 at 8:28 AM Alexey Romanenko
> >  wrote:
> >>
> >> I think it depends on the goal why to run that benchmarks. In ideal
> case, we need to run them on the same dedicated machine(s) and with the
> same configuration all the time but I’m not sure that it can be achieved in
> current infrastructure reality.
> >>
> >> On the other hand, IIRC, the initial goal of benchmarks, like Nexmark,
> was to detect fast any major regressions, especially between releases, that
> are not so sensitive to ideal conditions. And here we a field for
> improvements.
> >>
> >> —
> >> Alexey
> >>
> >> On 13 Sep 2022, at 22:57, Kenneth Knowles  wrote:
> >>
> >> Good idea. I'm curious about our current benchmarks. Some of them run
> on clusters, but I think some of them are running locally and just being
> noisy. Perhaps this could improve that. (or if they are running on local
> Spark/Flink then maybe the results are not really meaningful anyhow)
> >>
> >> On Tue, Sep 13, 2022 at 2:54 AM Moritz Mack  wrote:
> >>>
> >>> Hi team,
> >>>
> >>>
> >>>
> >>> I’m looking for some help to setup infrastructure to periodically run
> Java microbenchmarks (JMH).
> >>>
> >>> Results of these runs will be added to our community metrics
> (InfluxDB) to help us track performance, see [1].
> >>>
> >>>
> >>>
> >>> To prevent noisy runs this would require a dedicated Jenkins machine
> that runs at most one job (benchmark) at a time. Benchmark runs take quite
> some time, but on the other hand they don’t have to run very frequently
> (once a week should be fine initially).
> >>>
> >>>
> >>>
> >>> Thanks so much,
> >>>
> >>> Moritz
> >>>
> >>>
> >>>
> >>> [1]
> https://urldefense.com/v3/__https://github.com/apache/beam/pull/23041__;!!CiXD_PY!Qat2J4NAyHVo4Cc32PKMn50yw8LgWHmEOm4Ltb7aRV-7KCfNamu3tGOiSYKDUZhLHKu3zlqbBXzJNiUkaqlEKQ$
> 
>
> >>>
> >>> As a recipient of an email from Talend, your contact personal data
> will be on our systems. Please see our privacy notice.
> >>>
> >>>
> >>>
> >>
>
> *As a recipient of an email from Talend, your contact personal data will
> be on our systems. Please see our privacy notice.
> *
>
>
>

Re: What to do about issues that track flaky tests?

[Brian Hulette via dev]

I agree with Austin on this one, it makes sense to be realistic, but I'm
concerned about just blanket reducing the priority on all flakes. Two
classes of issues that could certainly be dropped to P2:
- Issues tracking flakes that have not been sickbayed yet (e.g.
https://github.com/apache/beam/issues/21266). These tests are still
providing signal (we should notice if it goes perma-red), and clearly the
flakes aren't so painful that someone felt the need to sickbay it.
- A sickbayed test, iff a breakage in the functionality it's testing would
be P2. This is admittedly difficult to identify.

It looks like we don't have a way to label sickbayed tests (or the inverse,
currently-failing), maybe we should have one?

Another thing to note: this email is reporting _unassigned_ P1 issues,
another way to remove issues from the search results would be to ensure
each flake has an owner (somehow). Maybe that's just shifting the problem,
but it could avoid the tragedy of the commons. To Manu's point, maybe those
new owners will happily discover their flake is no longer a problem.

Brian

On Wed, Sep 14, 2022 at 5:58 PM Manu Zhang  wrote:

> Agreed. I also mentioned in a previous email that some issues have been
> open for a long time (before being migrated to GitHub) and it's possible
> that those tests can pass constantly now.
> We may double check and close them since reopening is just one click.
>
> Manu
>
> On Thu, Sep 15, 2022 at 6:58 AM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> +1 to being realistic -- proper labels are worthwhile.  Though, some
>> flaky tests probably should be P1, and just because isn't addressed in a
>> timely manner doesn't mean it isn't a P1 - though, it does mean it wasn't
>> addressed.
>>
>>
>>
>> On Wed, Sep 14, 2022 at 1:19 PM Kenneth Knowles  wrote:
>>
>>> I would like to make this alert email actionable.
>>>
>>> I went through most of these issues. About half are P1 "flake" issues. I
>>> don't think magically expecting them to be deflaked is helpful. So I have a
>>> couple ideas:
>>>
>>> 1. Exclude "flake" P1s from this email. This is what we used to do. But
>>> then... are they really P1s?
>>> 2. Make "flake" bugs P2 if they are not currently impacting our test
>>> signal. But then... we may have a gap in test coverage that could cause
>>> severe problems. But anyhow something that is P1 for a long time is not
>>> *really* P1, so it is just being realistic.
>>>
>>> What do you all think?
>>>
>>> Kenn
>>>
>>> On Wed, Sep 14, 2022 at 3:03 AM  wrote:
>>>
 This is your daily summary of Beam's current high priority issues that
 may need attention.

 See https://beam.apache.org/contribute/issue-priorities for the
 meaning and expectations around issue priorities.

 Unassigned P1 Issues:

 https://github.com/apache/beam/issues/23227 [Bug]: Python SDK
 installation cannot generate proto with protobuf 3.20.2
 https://github.com/apache/beam/issues/23179 [Bug]: Parquet size
 exploded for no apparent reason
 https://github.com/apache/beam/issues/22913 [Bug]:
 beam_PostCommit_Java_ValidatesRunner_Flink is flakey
 https://github.com/apache/beam/issues/22303 [Task]: Add tests to Kafka
 SDF and fix known and discovered issues
 https://github.com/apache/beam/issues/22299 [Bug]: JDBCIO Write freeze
 at getConnection() in WriteFn
 https://github.com/apache/beam/issues/21794 Dataflow runner creates a
 new timer whenever the output timestamp is change
 https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't
 get output to Failed Inserts PCollection
 https://github.com/apache/beam/issues/21704
 beam_PostCommit_Java_DataflowV2 failures parent bug
 https://github.com/apache/beam/issues/21701
 beam_PostCommit_Java_DataflowV1 failing with a variety of flakes and errors
 https://github.com/apache/beam/issues/21700
 --dataflowServiceOptions=use_runner_v2 is broken
 https://github.com/apache/beam/issues/21696 Flink Tests failure :
 java.lang.NoClassDefFoundError: Could not initialize class
 org.apache.beam.runners.core.construction.SerializablePipelineOptions
 https://github.com/apache/beam/issues/21695 DataflowPipelineResult
 does not raise exception for unsuccessful states.
 https://github.com/apache/beam/issues/21694 BigQuery Storage API
 insert with writeResult retry and write to error table
 https://github.com/apache/beam/issues/21480 flake:
 FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
 https://github.com/apache/beam/issues/21472 Dataflow streaming tests
 failing new AfterSynchronizedProcessingTime test
 https://github.com/apache/beam/issues/21471 Flakes: Failed to load
 cache entry
 https://github.com/apache/beam/issues/21470 Test flake:
 test_split_half_sdf
 https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink
 flaky: Connection refused
 

Re: [Infrastructure] Periodically run Java microbenchmarks on Jenkins

[Moritz Mack]

Thank you, Andrew!
Exactly what I was looking for, that’s awesome!

On 15.09.22, 06:37, "Alexey Romanenko"  wrote:

Ahh, great! I didn’t know that 'beam-perf’ label is used for that.
Thanks!

> On 14 Sep 2022, at 17:47, Andrew Pilloud  wrote:
>
> We do have a dedicated machine for benchmarks. This is a single
> machine limited to running one test at a time. Set the
> jenkinsExecutorLabel for the job to 'beam-perf' to use it. For
> example:
> https://urldefense.com/v3/__https://github.com/apache/beam/blob/66bbee84ed477d86008905646e68b100591b6f78/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy*L36__;Iw!!CiXD_PY!Qat2J4NAyHVo4Cc32PKMn50yw8LgWHmEOm4Ltb7aRV-7KCfNamu3tGOiSYKDUZhLHKu3zlqbBXzJNiX_f_Qteg$
>
> Andrew
>
> On Wed, Sep 14, 2022 at 8:28 AM Alexey Romanenko
>  wrote:
>>
>> I think it depends on the goal why to run that benchmarks. In ideal case, we 
>> need to run them on the same dedicated machine(s) and with the same 
>> configuration all the time but I’m not sure that it can be achieved in 
>> current infrastructure reality.
>>
>> On the other hand, IIRC, the initial goal of benchmarks, like Nexmark, was 
>> to detect fast any major regressions, especially between releases, that are 
>> not so sensitive to ideal conditions. And here we a field for improvements.
>>
>> —
>> Alexey
>>
>> On 13 Sep 2022, at 22:57, Kenneth Knowles  wrote:
>>
>> Good idea. I'm curious about our current benchmarks. Some of them run on 
>> clusters, but I think some of them are running locally and just being noisy. 
>> Perhaps this could improve that. (or if they are running on local 
>> Spark/Flink then maybe the results are not really meaningful anyhow)
>>
>> On Tue, Sep 13, 2022 at 2:54 AM Moritz Mack  wrote:
>>>
>>> Hi team,
>>>
>>>
>>>
>>> I’m looking for some help to setup infrastructure to periodically run Java 
>>> microbenchmarks (JMH).
>>>
>>> Results of these runs will be added to our community metrics (InfluxDB) to 
>>> help us track performance, see [1].
>>>
>>>
>>>
>>> To prevent noisy runs this would require a dedicated Jenkins machine that 
>>> runs at most one job (benchmark) at a time. Benchmark runs take quite some 
>>> time, but on the other hand they don’t have to run very frequently (once a 
>>> week should be fine initially).
>>>
>>>
>>>
>>> Thanks so much,
>>>
>>> Moritz
>>>
>>>
>>>
>>> [1] 
>>> https://urldefense.com/v3/__https://github.com/apache/beam/pull/23041__;!!CiXD_PY!Qat2J4NAyHVo4Cc32PKMn50yw8LgWHmEOm4Ltb7aRV-7KCfNamu3tGOiSYKDUZhLHKu3zlqbBXzJNiUkaqlEKQ$
>>>
>>> As a recipient of an email from Talend, your contact personal data will be 
>>> on our systems. Please see our privacy notice.
>>>
>>>
>>>
>>

As a recipient of an email from Talend, your contact personal data will be on 
our systems. Please see our privacy notice. 

Beam High Priority Issue Report (75)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/23227 [Bug]: Python SDK installation 
cannot generate proto with protobuf 3.20.2
https://github.com/apache/beam/issues/23179 [Bug]: Parquet size exploded for no 
apparent reason
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakey
https://github.com/apache/beam/issues/22303 [Task]: Add tests to Kafka SDF and 
fix known and discovered issues
https://github.com/apache/beam/issues/22299 [Bug]: JDBCIO Write freeze at 
getConnection() in WriteFn
https://github.com/apache/beam/issues/21794 Dataflow runner creates a new timer 
whenever the output timestamp is change
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21704 beam_PostCommit_Java_DataflowV2 
failures parent bug
https://github.com/apache/beam/issues/21701 beam_PostCommit_Java_DataflowV1 
failing with a variety of flakes and errors
https://github.com/apache/beam/issues/21700 
--dataflowServiceOptions=use_runner_v2 is broken
https://github.com/apache/beam/issues/21696 Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions 
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not 
raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21480 flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21472 Dataflow streaming tests failing 
new AfterSynchronizedProcessingTime test
https://github.com/apache/beam/issues/21471 Flakes: Failed to load cache entry
https://github.com/apache/beam/issues/21470 Test flake: test_split_half_sdf
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21468 
beam_PostCommit_Python_Examples_Dataflow failing
https://github.com/apache/beam/issues/21467 GBK and CoGBK streaming Java load 
tests failing
https://github.com/apache/beam/issues/21463 NPE in Flink Portable 
ValidatesRunner streaming suite
https://github.com/apache/beam/issues/21462 Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21271 pubsublite.ReadWriteIT flaky in 
beam_PostCommit_Java_DataflowV2  
https://github.com/apache/beam/issues/21270 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2
https://github.com/apache/beam/issues/21267 WriteToBigQuery submits a duplicate 
BQ load job if a 503 error code is returned from googleapi
https://github.com/apache/beam/issues/21266 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite.
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21261 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21257 Either Create or DirectRunner fails 
to produce all elements to the following transform
https://github.com/apache/beam/issues/21123 Multiple jobs running on Flink 
session cluster reuse the persistent Python environment.
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21118 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky
https://github.com/apache/beam/issues/21114 Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN
https://github.com/apache/beam/issues/21113 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky
https://github.com/apache/beam/issues/2 Java creates an incorrect pipeline 
proto when core-construction-java jar is not in the CLASSPATH
https://github.com/apache/beam/issues/20981 Python precommit flaky: Failed to 
read inputs in the data plane
https://github.com/apache/beam/issues/20977 SamzaStoreStateInternalsTest is 
flaky
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20975 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky
https://github.com/apache/beam/issues/20974 Python GHA PreCommits flake with 
grpc.FutureTimeoutError on SDK harness