Credentials Rotation Failure on IO-Datastores cluster

[Apache Jenkins Server]

Something went wrong during the automatic credentials rotation for 
IO-Datastores Cluster, performed at Thu Dec 01 00:53:08 UTC 2022. It may be 
necessary to check the state of the cluster certificates. For further details 
refer to the following links:
 * https://ci-beam.apache.org/job/beam_SeedJob/ 
 * https://ci-beam.apache.org/.

Re: Issue with website update and Jenkins

[Alexey Romanenko]

Thanks, Kenn!

> On 30 Nov 2022, at 18:59, Kenneth Knowles  wrote:
> 
> I filed https://issues.apache.org/jira/browse/INFRA-23967
> 
> On Tue, Nov 29, 2022 at 7:53 AM Alexey Romanenko  > wrote:
>> It looks that there is again the same issue with 
>> beam_PostCommit_Website_Publish job - the last successful build was 6 days 
>> ago.
>> 
>> Could someone take a look on this?
>> 
>> Thanks,
>> Alexey
>> 
>>> On 11 Oct 2022, at 19:23, David Huntsperger via dev >> > wrote:
>>> 
>>> Thank you!
>>> 
>>> On Mon, Oct 10, 2022 at 8:18 AM Alexey Romanenko >> > wrote:
 The issue is resolved.
 Thank you to everybody who made this working again!
 
 ---
 Alexey
 
> On 7 Oct 2022, at 15:26, Alexey Romanenko  > wrote:
> 
> Hi everybody,
> 
> The is an issue with updating a content on Beam website. I believe it’s 
> caused by not running the Jenkins job that publishes the Beam website 
> into the asf-site branch since 30 Sep 2022 [1] with an error message 
> “There are no nodes with the label ‘git-websites’”:
> - https://ci-beam.apache.org/job/beam_PostCommit_Website_Publish/
> 
> In it’s order, iinm, it’s caused by the fact that several 
> “apache-beam-jenkins-*” nodes are offline.
> 
> Does anyone aware of this problem and work on this? To whom we need to 
> address such problems, INFRA?
> 
> How we can prevent this silent behaviour in the future? Beam website 
> publishing job (“beam_PostCommit_Website_Publish”) just got stuck and 
> hangs for a while.
> 
> —
> Alexey
> 
 
>> 

Re: Issue with website update and Jenkins

[Kenneth Knowles]

I filed https://issues.apache.org/jira/browse/INFRA-23967

On Tue, Nov 29, 2022 at 7:53 AM Alexey Romanenko 
wrote:

> It looks that there is again the same issue
> with beam_PostCommit_Website_Publish job - the last successful build was 6
> days ago.
>
> Could someone take a look on this?
>
> Thanks,
> Alexey
>
> On 11 Oct 2022, at 19:23, David Huntsperger via dev 
> wrote:
>
> Thank you!
>
> On Mon, Oct 10, 2022 at 8:18 AM Alexey Romanenko 
> wrote:
>
>> The issue is resolved.
>> Thank you to everybody who made this working again!
>>
>> ---
>> Alexey
>>
>> On 7 Oct 2022, at 15:26, Alexey Romanenko 
>> wrote:
>>
>> Hi everybody,
>>
>> The is an issue with updating a content on Beam website. I believe it’s
>> caused by not running the Jenkins job that publishes the Beam website into
>> the asf-site branch since 30 Sep 2022 [1] with an error message “There are
>> no nodes with the label ‘git-websites’”:
>> - https://ci-beam.apache.org/job/beam_PostCommit_Website_Publish/
>>
>> In it’s order, iinm, it’s caused by the fact that several
>> “apache-beam-jenkins-*” nodes are offline.
>>
>> Does anyone aware of this problem and work on this? To whom we need to
>> address such problems, INFRA?
>>
>> How we can prevent this silent behaviour in the future? Beam website
>> publishing job (“beam_PostCommit_Website_Publish”) just got stuck and hangs
>> for a while.
>>
>> —
>> Alexey
>>
>>
>>
>

Re: [DISCUSSION][JAVA] Current state of Java 17 support

[Kenneth Knowles]

An important thing is to ensure that we do not accidentally depend on
something that would break Java 8 support.

Currently our Java 11 and 17 tests build the code with Java 8 (just like
our released artifacts) and then compile and run the test code with the
newer JDK. This roughly matches the user scenario, I think. So it is a
little more complex than just having separate test runs for different JDK
versions. But it would be good to make this more symmetrical between JDK
versions to develop the mindset that JDK is always explicit.

Kenn

On Wed, Nov 30, 2022 at 9:48 AM Alexey Romanenko 
wrote:

>
> On 30 Nov 2022, at 03:56, Tomo Suzuki via dev  wrote:
>
> > Do we still need to support Java 8 SDK?
>
> Yes, for Google Cloud customers who still use Java 8, I want Apache Beam
> to support Java 8. Do you observe any special burden maintaining Java 8?
>
>
> I can only think of the additional resources costs if we will test all
> supported JDKs, as Austin mentioned above. Imho, we should do that for all
> JDK that are officially supported.
> Another less-costly way is to run the Java tests for all JDKs only during
> the release preparation stage.
>
> I agree that it would make sense to continue to support Java 8 until a
> significant number of users are using it.
>
> —
> Alexey
>
>
>
> Regards,
> Tomo
>
> On Tue, Nov 29, 2022 at 21:48 Austin Bennett  wrote:
>
>> -1 for ongoing Java8 support [ or, said another way, +1 for dropping
>> support of Java8 ]
>>
>> +1 for having tests that run for ANY JDK that we say we support.  Is
>> there any reason the resources to support are too costly [ or outweigh the
>> benefits of additional confidence in ensuring we support what we say we do
>> ]?  I am not certain on whether this would only be critical for releases,
>> or should be done as part of regular CI.
>>
>> On Tue, Nov 29, 2022 at 8:51 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I’m sorry if it’s already discussed somewhere but I find myself a little
>>> bit lost in the subject.
>>> So, I’d like to clarify this - what is a current official state of Java
>>> 17 support at Beam?
>>>
>>> I recall that a great job was done to make Beam compatible with Java 17
>>> [1] and Beam already provides “beam_java17_sdk” Docker image [2] but, iiuc,
>>> Java 8 is still the default JVM to run all Java tests on Jenkins ("Java
>>> PreCommit" in the first order) and there are only limited number of tests
>>> that are running with JDK 11 and 17 on Jenkins by dedicated jobs.
>>>
>>> So, my question would sound like if Beam officially supports Java 17
>>> (and 11), do we need to run all Beam Java SDK related tests (VR and IT test
>>> including) against all supported Java SDKs?
>>>
>>> Do we still need to support Java 8 SDK?
>>>
>>> In the same time, as we are heading to move everything from Jenkins to
>>> GitHub actions, what would be the default JDK there or we will run all
>>> Java-related actions against all supported JDKs?
>>>
>>> —
>>> Alexey
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-12240
>>> [2] https://hub.docker.com/r/apache/beam_java17_sdk
>>>
>>>
>>>
>>> --
> Regards,
> Tomo
>
>
>

Re: [DISCUSSION][JAVA] Current state of Java 17 support

[Alexey Romanenko]

> On 30 Nov 2022, at 03:56, Tomo Suzuki via dev  wrote:
> 
> > Do we still need to support Java 8 SDK?
> 
> Yes, for Google Cloud customers who still use Java 8, I want Apache Beam to 
> support Java 8. Do you observe any special burden maintaining Java 8?

I can only think of the additional resources costs if we will test all 
supported JDKs, as Austin mentioned above. Imho, we should do that for all JDK 
that are officially supported.
Another less-costly way is to run the Java tests for all JDKs only during the 
release preparation stage.

I agree that it would make sense to continue to support Java 8 until a 
significant number of users are using it.

—
Alexey


> 
> Regards,
> Tomo
> 
> On Tue, Nov 29, 2022 at 21:48 Austin Bennett  > wrote:
>> -1 for ongoing Java8 support [ or, said another way, +1 for dropping support 
>> of Java8 ]
>> 
>> +1 for having tests that run for ANY JDK that we say we support.  Is there 
>> any reason the resources to support are too costly [ or outweigh the 
>> benefits of additional confidence in ensuring we support what we say we do 
>> ]?  I am not certain on whether this would only be critical for releases, or 
>> should be done as part of regular CI.  
>> 
>> On Tue, Nov 29, 2022 at 8:51 AM Alexey Romanenko > > wrote:
>>> Hello,
>>> 
>>> I’m sorry if it’s already discussed somewhere but I find myself a little 
>>> bit lost in the subject. 
>>> So, I’d like to clarify this - what is a current official state of Java 17 
>>> support at Beam?
>>> 
>>> I recall that a great job was done to make Beam compatible with Java 17 [1] 
>>> and Beam already provides “beam_java17_sdk” Docker image [2] but, iiuc, 
>>> Java 8 is still the default JVM to run all Java tests on Jenkins ("Java 
>>> PreCommit" in the first order) and there are only limited number of tests 
>>> that are running with JDK 11 and 17 on Jenkins by dedicated jobs.
>>> 
>>> So, my question would sound like if Beam officially supports Java 17 (and 
>>> 11), do we need to run all Beam Java SDK related tests (VR and IT test 
>>> including) against all supported Java SDKs? 
>>> 
>>> Do we still need to support Java 8 SDK?
>>> 
>>> In the same time, as we are heading to move everything from Jenkins to 
>>> GitHub actions, what would be the default JDK there or we will run all 
>>> Java-related actions against all supported JDKs?
>>> 
>>> —
>>> Alexey
>>> 
>>> [1] https://issues.apache.org/jira/browse/BEAM-12240
>>> [2] https://hub.docker.com/r/apache/beam_java17_sdk
>>> 
>>> 
>>> 
> -- 
> Regards,
> Tomo

Beam High Priority Issue Report (63)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/24415 [Bug]: Cannot find a matching 
Calcite SqlTypeName for Beam type: LOGICAL_TYPE seen in 2.44.0 SNAPSHOT
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24384 [Bug]: 
RampupThrottlingFnTest.testRampupThrottler TooManyActualInvocations
https://github.com/apache/beam/issues/24383 [Bug]: Daemon will be stopped at 
the end of the build after the daemon was no longer found in the daemon registry
https://github.com/apache/beam/issues/24374 [Bug]: Fail to retrieve rowcount 
for first arrow chunk: null.
https://github.com/apache/beam/issues/24367 [Bug]: workflow.tar.gz cannot be 
passed to flink runner
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/24267 [Failing Test]: Timeout waiting to 
lock gradle
https://github.com/apache/beam/issues/24263 [Bug]: Remote call on 
apache-beam-jenkins-3 failed. The channel is closing down or has closed down
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23745 [Bug]: Samza 
AsyncDoFnRunnerTest.testSimplePipeline is flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22969 Discrepancy in behavior of 
`DoFn.process()` when `yield` is combined with `return` statement, or vice versa
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22321 
PortableRunnerTestWithExternalEnv.test_pardo_large_input is regularly failing 
on jenkins
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21561 
ExternalPythonTransformTest.trivialPythonTransform flaky
https://github.com/apache/beam/issues/21480 flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21462 Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21261 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21113 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20975 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky
https://github.com/apache/beam/issues/20974 Python GHA PreCommits flake with 
grpc.FutureTimeoutError on SDK harness startup
https://github.com/apache/beam/issues/20689 Kafka commitOffsetsInFinalize OOM 
on Flink
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19734 
WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp 
order (sickbayed)
https://github.com/apache/beam/issues/19465 Explore possibilities to lower 
in-use IP address quota footprint.
https://github.com/apache/beam/issues/19241 Python Dataflow integration tests 
should export the pipeline Job ID and console output to Jenkins Test Result 
section


P1 Issues with no update in the last week:

https://github.com/apache/beam/issues/24100 [Bug]: `Filter.whereFieldName` 
appears in docs but not available
https://github.com/apache/beam/issues/23906 [Bug]: Dataflow jpms tests fail on 
the 2.43.0 release branch