[GitHub] [beam-site] y1chi closed pull request #627: Publish 2.39.0 release

[GitBox]

y1chi closed pull request #627: Publish 2.39.0 release
URL: https://github.com/apache/beam-site/pull/627


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

P1 issues report (78)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-14467: Failed test annotated 
with no_xdist passes github action python test (created 2022-05-12)
https://issues.apache.org/jira/browse/BEAM-14459: Docker Snapshots failing 
to be published since April 14th (created 2022-05-11)
https://issues.apache.org/jira/browse/BEAM-14434: 
beam_LoadTests_Python_GBK_reiterate_Dataflow_Streaming failure (created 
2022-05-06)
https://issues.apache.org/jira/browse/BEAM-14431: Handle nulls using 
SnowflakeIO (created 2022-05-06)
https://issues.apache.org/jira/browse/BEAM-14421: 
--dataflowServiceOptions=use_runner_v2 is broken (created 2022-05-05)
https://issues.apache.org/jira/browse/BEAM-14411: TypeCodersTest is never 
executed (created 2022-05-04)
https://issues.apache.org/jira/browse/BEAM-14390: Java license check is 
broken (created 2022-05-02)
https://issues.apache.org/jira/browse/BEAM-14364: 404s in BigQueryIO don't 
get output to Failed Inserts PCollection (created 2022-04-25)
https://issues.apache.org/jira/browse/BEAM-14356: Java PostCommits: 
BigQueryIO.Read needs a GCS temp location (created 2022-04-22)
https://issues.apache.org/jira/browse/BEAM-14298: Can't resolve 
org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde (created 2022-04-12)
https://issues.apache.org/jira/browse/BEAM-14291: DataflowPipelineResult 
does not raise exception for unsuccessful states. (created 2022-04-11)
https://issues.apache.org/jira/browse/BEAM-14276: 
beam_PostCommit_Java_DataflowV2 failures parent bug (created 2022-04-07)
https://issues.apache.org/jira/browse/BEAM-14275: SpannerWriteIT failing in 
beam PostCommit Java V1 (created 2022-04-07)
https://issues.apache.org/jira/browse/BEAM-14265: Flink should hold the 
watermark at the output timestamp for processing time timers (created 
2022-04-06)
https://issues.apache.org/jira/browse/BEAM-14263: 
beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing 
consistently (created 2022-04-05)
https://issues.apache.org/jira/browse/BEAM-14253: pubsublite.ReadWriteIT 
failing in beam_PostCommit_Java_DataflowV1 and V2 (created 2022-04-05)
https://issues.apache.org/jira/browse/BEAM-14239: Changing the output 
timestamp of a timer does not clear the previously set timer (created 
2022-04-04)
https://issues.apache.org/jira/browse/BEAM-14174: Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions  (created 
2022-03-24)
https://issues.apache.org/jira/browse/BEAM-14135: BigQuery Storage API 
insert with writeResult retry and write to error table (created 2022-03-20)
https://issues.apache.org/jira/browse/BEAM-13952: Dataflow streaming tests 
failing new AfterSynchronizedProcessingTime test (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13950: PVR_Spark2_Streaming 
perma-red (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13920: Beam x-lang Dataflow 
tests failing due to _InactiveRpcError (created 2022-02-10)
https://issues.apache.org/jira/browse/BEAM-13852: 
KafkaIO.read.withDynamicRead() doesn't pick up new TopicPartitions (created 
2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13850: 
beam_PostCommit_Python_Examples_Dataflow failing (created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13822: GBK and CoGBK streaming 
Java load tests failing (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13805: Simplify version override 
for Dev versions of the Go SDK. (created 2022-02-02)
https://issues.apache.org/jira/browse/BEAM-13747: Add integration testing 
for BQ Storage API  write modes (created 2022-01-26)
https://issues.apache.org/jira/browse/BEAM-13715: Kafka commit offset drop 
data on failure for runners that have non-checkpointing shuffle (created 
2022-01-21)
https://issues.apache.org/jira/browse/BEAM-13487: WriteToBigQuery Dynamic 
table destinations returns wrong tableId (created 2021-12-17)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13164: Race between member 
variable being accessed due to leaking uninitialized state via 
OutboundObserverFactory (created 2021-11-01)
https://issues.apache.org/jira/browse/BEAM-13132: WriteToBigQuery submits a 
duplicate BQ load job if a 503 error code is returned from googleapi (created 
2021-10-27)
https://issues.apache.org/jira/browse/BEAM-13087: 
apache_beam.runners.p

Flaky test issue report (57)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-14459: Docker Snapshots failing 
to be published since April 14th (created 2022-05-11)
https://issues.apache.org/jira/browse/BEAM-14410: FnRunnerTest with 
non-trivial (order 1000 elements) numpy input flakes in non-cython environment 
(created 2022-05-04)
https://issues.apache.org/jira/browse/BEAM-14407: Jenkins worker sometimes 
crashes while running Python Flink pipeline (created 2022-05-04)
https://issues.apache.org/jira/browse/BEAM-14367: Flaky timeout in github 
Python unit test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer (created 
2022-04-26)
https://issues.apache.org/jira/browse/BEAM-14349: GroupByKeyTest BasicTests 
testLargeKeys100MB flake (on ULR) (created 2022-04-21)
https://issues.apache.org/jira/browse/BEAM-14276: 
beam_PostCommit_Java_DataflowV2 failures parent bug (created 2022-04-07)
https://issues.apache.org/jira/browse/BEAM-14269: 
PulsarIOTest.testReadFromSimpleTopic is very flaky (created 2022-04-06)
https://issues.apache.org/jira/browse/BEAM-14263: 
beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing 
consistently (created 2022-04-05)
https://issues.apache.org/jira/browse/BEAM-14252: 
beam_PostCommit_Java_DataflowV1 failing with a variety of flakes and errors 
(created 2022-04-05)
https://issues.apache.org/jira/browse/BEAM-14216: Multiple XVR Suites 
having similar flakes simultaneously (created 2022-03-31)
https://issues.apache.org/jira/browse/BEAM-14174: Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions  (created 
2022-03-24)
https://issues.apache.org/jira/browse/BEAM-14172: beam_PreCommit_PythonDocs 
failing (jinja2) (created 2022-03-24)
https://issues.apache.org/jira/browse/BEAM-13952: Dataflow streaming tests 
failing new AfterSynchronizedProcessingTime test (created 2022-02-15)
https://issues.apache.org/jira/browse/BEAM-13859: Test flake: 
test_split_half_sdf (created 2022-02-09)
https://issues.apache.org/jira/browse/BEAM-13850: 
beam_PostCommit_Python_Examples_Dataflow failing (created 2022-02-08)
https://issues.apache.org/jira/browse/BEAM-13822: GBK and CoGBK streaming 
Java load tests failing (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13810: Flaky tests: Gradle build 
daemon disappeared unexpectedly (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13809: beam_PostCommit_XVR_Flink 
flaky: Connection refused (created 2022-02-03)
https://issues.apache.org/jira/browse/BEAM-13797: Flakes: Failed to load 
cache entry (created 2022-02-01)
https://issues.apache.org/jira/browse/BEAM-13708: flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored (created 2022-01-20)
https://issues.apache.org/jira/browse/BEAM-13575: Flink 
testParDoRequiresStableInput flaky (created 2021-12-28)
https://issues.apache.org/jira/browse/BEAM-13500: NPE in Flink Portable 
ValidatesRunner streaming suite (created 2021-12-21)
https://issues.apache.org/jira/browse/BEAM-13453: Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use 
(created 2021-12-13)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13367: 
[beam_PostCommit_Python36] [ 
apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary 
(created 2021-12-01)
https://issues.apache.org/jira/browse/BEAM-13312: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
 is flaky in Java Spark ValidatesRunner suite  (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13311: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite. (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13237: 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2 (created 2021-11-12)
https://issues.apache.org/jira/browse/BEAM-13025: pubsublite.ReadWriteIT 
flaky in beam_PostCommit_Java_DataflowV2   (created 2021-10-08)
https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefu

Re: [VOTE] Release 2.39.0, candidate #1

[Jean-Baptiste Onofré]

+1 (binding)

Smoke quick tests with the direct runner.

Regards
JB

On Tue, May 17, 2022 at 5:05 PM Ahmet Altay  wrote:
>
> +1 (binding). I validated python quick starts on Direct runner and on 
> Dataflow.
>
> Thank you Yichi!
>
> On Mon, May 16, 2022 at 11:58 PM Yichi Zhang  wrote:
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version 2.39.0, 
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release 
>> candidate, and vote +1 if no issues are found.
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2], 
>> which is signed with the key with fingerprint 
>> 0DB9FF04B5D967903A28427021DE16602971EB7C [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.39.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and 
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK 1.8.0_332.
>> * Python artifacts are deployed along with the source release to the 
>> dist.apache.org [2] and PyPI[8].
>> * Validation sheet with a tab for 2.39.0 release to help with validation [9].
>> * Docker images published to Docker Hub [10].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority 
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our 
>> blog post at https://beam.apache.org/blog/validate-beam-release/.
>>
>> Thanks,
>> Yichi Zhang
>>
>> [1] 
>> https://issues.apache.org/jira/secure/ConfigureReleaseNote.jspa?projectId=12319527&version=12351170
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.39.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1271/
>> [5] https://github.com/apache/beam/tree/v2.39.0-RC1
>> [6] https://github.com/apache/beam/pull/17690
>> [7] https://github.com/apache/beam-site/pull/627
>> [8] https://pypi.org/project/apache-beam/2.39.0rc1/
>> [9] 
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=584711835
>> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image

SerializablePipelineOptions / FileSystems.setDefaultPipelineOptions

[Moritz Mack]

Does anybody here have some insights on this? Really wondering about the 
numbers, initializing all filesystems ~80k times for a pipeline run doesn’t 
seem right.

On 13.05.22, 09:10, "Moritz Mack"  wrote:

Hi Jack, Silencing info logs for that class during IT tests would be a quick 
fix, but also removing logging there entirely shouldn’t hurt. If the S3 
filesystem is used it’ll fail on first usage and the issue should be fairly 
obvious… ‍ ‍ ‍ ‍ ‍ ‍
ZjQcmQRYFpfptBannerStart
ZjQcmQRYFpfptBannerEnd
Hi Jack,

Silencing info logs for that class during IT tests would be a quick fix, but 
also removing logging there entirely shouldn’t hurt.
If the S3 filesystem is used it’ll fail on first usage and the issue should be 
fairly obvious…

Though wondering, this is logged once when file systems are initialized… seeing 
this ~80k times basically means all file systems including the S3 one get 
initialized so many times.
That doesn’t seem right! But I know very little about the portable runners and 
lifecycle of components / JVMs :/
I suspect this behavior is related to calling 
FileSystems.setDefaultPipelineOptions (triggering initialization of all 
Filesystems) on deserialization of SerializablePipelineOptions, see 
https://issues.apache.org/jira/browse/BEAM-2712.

Best,
Moritz (mosche)


From: Pablo Estrada 
Date: Thursday, 12. May 2022 at 20:50
To: dev , Alexey Romanenko 
Subject: Re: S3ClientBuilder Logging
Hi Jack, Frankly, I think you should feel free to make a change to reduce the 
logging levels for some of the logs, and get a review from @Alexey Romanenko 
@mosche(github), or myself. This is valuable feedback, so it would be great if 
we can
ZjQcmQRYFpfptBannerStart
ZjQcmQRYFpfptBannerEnd
Hi Jack,
Frankly, I think you should feel free to make a change to reduce the logging 
levels for some of the logs, and get a review from @Alexey 
Romanenko @mosche(github), or myself. This is 
valuable feedback, so it would be great if we can get the logging to a good 
spot : )
Best
-P.

On Thu, May 12, 2022 at 8:56 AM Jack McCluskey 
mailto:jrmcclus...@google.com>> wrote:
Hey everyone,

I've noticed that the logging in the AWS2 
DefaultS3ClientBuilderFactory
 is extremely verbose when running Beam-on-Flink. For reference, the logged 
statement "INFO: The AWS S3 Beam extension was included in this build, but the 
awsRegion flag was not specified. If you don't plan to use S3, then ignore this 
message." appears in a full run of the Beam Go Flink integration tests 79,970 
times. Is this necessary? This makes parsing output in the event of an error 
much more difficult and time consuming. I don't have the full context around 
this class and its usage, but it seems excessive at face value.

Thanks,

Jack McCluskey

--
[https://lh4.googleusercontent.com/OT9S5kjymtmtwskqZyTzlenaa8pi4IqW358dV1KN_HEP2T0KIx5VKMlWhzP9fM27_juihIYJ6-9aMRlD3uvT-ilMsbWCuPCDkk9bg60EH5Q4GiypzX00lpGthpiZTetEEGb0NBm7PDUT]

Jack McCluskey
SWE - DataPLS PLAT/ Beam Go
RDU
jrmcclus...@gmail.com


As a recipient of an email from Talend, your contact personal data will be on 
our systems. Please see our privacy notice. 


As a recipient of an email from Talend, your contact personal data will be on 
our systems. Please see our privacy notice. 

Re: [DISCUSS] Next steps for update of Avro dependency in Beam

[Moritz Mack]

Sorry, please ignore the previous empty reply …

On 17.05.22, 09:31, "Moritz Mack"  wrote:

On 16.05.22, 18:09, "Robert Bradshaw"  wrote:

On Mon, May 16, 2022 at 8:53 AM Alexey Romanenko
 wrote:
>
>> On 13 May 2022, at 18:38, Robert Bradshaw  wrote:
>>
>> We should probably remove the experimental annotations from
> SchemaCoder at this point.
>
> Is there anything that stops us from that?

It's a commitment to not change things, but I think we're at that point by now.

> Having that in mind, here’s my 2 cents on the options mentioned:
>
> Bump Avro in core:
>
> This will seamlessly work for users that just need a coder for complex types, 
> but don’t use Avro themselves.
> Anyone using Avro APIs from user code might get into trouble.
>
> Support different Avro versions & making the dependency provided
>
> In terms of user impact, making Avro provided will be very troublesome. 
> Unless Avro remains available on the classpath through different ways, this 
> would break things in a really bad way. Worst of all, it might not even break 
> at compile time but at runtime only.
> I don’t think this option requires Avro being a provided dependency. If the 
> default is kept as is, anyone can (safely) bump Avro to a higher (supported) 
> version on their classpath. In that case it would be good to warn users about 
> CVEs in logs if they remain on an old version of Avro. But I’m not sure how 
> feasible (reliable) it is to detect the Avro minor version given that things 
> might be repackaged into fat jars.
>
> This seems both difficult and potentially dangerous.
>
> What do you mean by “potentially dangerous”, Robert? Any specific point 
> mentioned above or just having another version of Avro jars in the classpath?

The most dangerous is if a change in avro version had slightly
[1] different semantics when it was swapped out. I think the chances of
that are not too high, but it'll be hard to detect. Mostly I'm
concerned about [2] breaking at runtime in the case of a missing or
incompatible avro.

[1] Don’t you think these can be mitigated by testing relevant parts of Beam 
against a number of supported Avro versions the same way parts of Beam are 
tested against various different Hadoop versions.

[2] I agree, this is a huge concern and for that reason I believe Avro cannot 
be turned into a provided dependency. It would be a dangerous trap for users. 
But if the default Avro dependency remains as is and Beam is tested to work 
with a well-defined set of Avro versions (swapping out the default using 
standard dependency resolution rules), I’d consider the risk to be rather 
manageable.


>> Extract Avro as an extension
> I think a change that breaks loudly at compile time (if the features
> in question are being depended on), with a clear fix (e.g. "depend on
> this instead") is the "cleanest" way to do a breaking change.
>
> On this note, would it be possible to do explicit "without avro" and
> "with avro" targets/variants in the core module, with plain old "core"
> being an alias first to the latter, and then to the former, according
> to some deprecation timeline? (I'm not familiar enough with
> gradle/maven to know if this is possible.) This could unblock users
> needing a new avro in the short run while providing a longer-term
> migration path. (Ideally the (relatively limited) avro parts of core
> would also get moved to their own extension under their own namespace,
> and the core one would remain solely for backwards compatibility
> reasons.)
>
>
> Good point. Are you talking about keeping the default “core" artifacts with 
> old Avro version (as for now) as a main dependency and provide additionally 
> other “core” artifacts with a recent Avro version or even as provided for 
> users that need other Avro versions?

I'm thinking about keeping the existing "core" artifact, and also
shipping an [3] "avro-less" core target (whether that be via shading or
total removal, but ripping it out of the public API). We would then
ship a [4] separate "avro" target (TBD if providing the dependency or not
makes the most sense here) that the existing core would depend on and
would be a target for avro users to migrate to.

[3] It wouldn’t be too hard modelling this using feature variants. Though, I’m 
worried about the complexity this creates. In the end, every single Beam module 
depends on core, which would then depend on the new Avro target. Effectively, 
we would need to build two different versions of each module depending on 
either core or the avro-less core. If there would be a well-planned, temporary 
transition period ahead, it should be fine. But I fear with Beam 3 not even 
being on the horizon, we’d have to carry this forward for far too long.

[4] Also, what are your thoughts about the new Avro target? It could be done 
just by moving respective code into a new module (maintaining all packages / 
fully qualified names). Or “clean” by moving classes adequately but breaking 
fully qualified names / imports 

Re: [DISCUSS] Next steps for update of Avro dependency in Beam

[Moritz Mack]

On 17.05.22, 09:31, "Moritz Mack"  wrote:
On 16.05.22, 18:09, "Robert Bradshaw"  wrote: On Mon, May 
16, 2022 at 8:53 AM Alexey Romanenko  wrote: > >> On 
13 May 2022, at 18:38, Robert Bradshaw 
ZjQcmQRYFpfptBannerStar

On 16.05.22, 18:09, "Robert Bradshaw"  wrote:

On Mon, May 16, 2022 at 8:53 AM Alexey Romanenko
 wrote:
>
>> On 13 May 2022, at 18:38, Robert Bradshaw  wrote:
>>
>> We should probably remove the experimental annotations from
> SchemaCoder at this point.
>
> Is there anything that stops us from that?

It's a commitment to not change things, but I think we're at that point by now.

> Having that in mind, here’s my 2 cents on the options mentioned:
>
> Bump Avro in core:
>
> This will seamlessly work for users that just need a coder for complex types, 
> but don’t use Avro themselves.
> Anyone using Avro APIs from user code might get into trouble.
>
> Support different Avro versions & making the dependency provided
>
> In terms of user impact, making Avro provided will be very troublesome. 
> Unless Avro remains available on the classpath through different ways, this 
> would break things in a really bad way. Worst of all, it might not even break 
> at compile time but at runtime only.
> I don’t think this option requires Avro being a provided dependency. If the 
> default is kept as is, anyone can (safely) bump Avro to a higher (supported) 
> version on their classpath. In that case it would be good to warn users about 
> CVEs in logs if they remain on an old version of Avro. But I’m not sure how 
> feasible (reliable) it is to detect the Avro minor version given that things 
> might be repackaged into fat jars.
>
> This seems both difficult and potentially dangerous.
>
> What do you mean by “potentially dangerous”, Robert? Any specific point 
> mentioned above or just having another version of Avro jars in the classpath?

The most dangerous is if a change in avro version had slightly
[1] different semantics when it was swapped out. I think the chances of
that are not too high, but it'll be hard to detect. Mostly I'm
concerned about [2] breaking at runtime in the case of a missing or
incompatible avro.

[1] Don’t you think these can be mitigated by testing relevant parts of Beam 
against a number of supported Avro versions the same way parts of Beam are 
tested against various different Hadoop versions.

[2] I agree, this is a huge concern and for that reason I believe Avro cannot 
be turned into a provided dependency. It would be a dangerous trap for users. 
But if the default Avro dependency remains as is and Beam is tested to work 
with a well-defined set of Avro versions (swapping out the default using 
standard dependency resolution rules), I’d consider the risk to be rather 
manageable.


>> Extract Avro as an extension
> I think a change that breaks loudly at compile time (if the features
> in question are being depended on), with a clear fix (e.g. "depend on
> this instead") is the "cleanest" way to do a breaking change.
>
> On this note, would it be possible to do explicit "without avro" and
> "with avro" targets/variants in the core module, with plain old "core"
> being an alias first to the latter, and then to the former, according
> to some deprecation timeline? (I'm not familiar enough with
> gradle/maven to know if this is possible.) This could unblock users
> needing a new avro in the short run while providing a longer-term
> migration path. (Ideally the (relatively limited) avro parts of core
> would also get moved to their own extension under their own namespace,
> and the core one would remain solely for backwards compatibility
> reasons.)
>
>
> Good point. Are you talking about keeping the default “core" artifacts with 
> old Avro version (as for now) as a main dependency and provide additionally 
> other “core” artifacts with a recent Avro version or even as provided for 
> users that need other Avro versions?

I'm thinking about keeping the existing "core" artifact, and also
shipping an [3] "avro-less" core target (whether that be via shading or
total removal, but ripping it out of the public API). We would then
ship a separate "avro" target (TBD if providing the dependency or not
makes the most sense here) that the existing core would depend on and
would be a target for avro users to migrate to.

[3] It wouldn’t be too hard modelling this using feature variants. Though, I’m 
worried about the complexity this creates. In the end, every single Beam module 
depends on core, which would then depend on the new Avro target. Effectively, 
we would need to build two different versions of each module depending on 
either core or the avro-less core. If there would be a well-planned, temporary 
transition period ahead, it should be fine. But I fear with Beam 3 not even 
being on the horizon, we’d have to carry this forward for far too long.


> To finally make progress on this topic, I’d suggest starting off with 
> supporting multiple different Avro versions in core (2) but to k

Re: [DISCUSS] Next steps for update of Avro dependency in Beam

[Moritz Mack]

On 16.05.22, 18:09, "Robert Bradshaw"  wrote:

On Mon, May 16, 2022 at 8:53 AM Alexey Romanenko
 wrote:
>
>> On 13 May 2022, at 18:38, Robert Bradshaw  wrote:
>>
>> We should probably remove the experimental annotations from
> SchemaCoder at this point.
>
> Is there anything that stops us from that?

It's a commitment to not change things, but I think we're at that point by now.

> Having that in mind, here’s my 2 cents on the options mentioned:
>
> Bump Avro in core:
>
> This will seamlessly work for users that just need a coder for complex types, 
> but don’t use Avro themselves.
> Anyone using Avro APIs from user code might get into trouble.
>
> Support different Avro versions & making the dependency provided
>
> In terms of user impact, making Avro provided will be very troublesome. 
> Unless Avro remains available on the classpath through different ways, this 
> would break things in a really bad way. Worst of all, it might not even break 
> at compile time but at runtime only.
> I don’t think this option requires Avro being a provided dependency. If the 
> default is kept as is, anyone can (safely) bump Avro to a higher (supported) 
> version on their classpath. In that case it would be good to warn users about 
> CVEs in logs if they remain on an old version of Avro. But I’m not sure how 
> feasible (reliable) it is to detect the Avro minor version given that things 
> might be repackaged into fat jars.
>
> This seems both difficult and potentially dangerous.
>
> What do you mean by “potentially dangerous”, Robert? Any specific point 
> mentioned above or just having another version of Avro jars in the classpath?

The most dangerous is if a change in avro version had slightly
[1] different semantics when it was swapped out. I think the chances of
that are not too high, but it'll be hard to detect. Mostly I'm
concerned about [2] breaking at runtime in the case of a missing or
incompatible avro.

[1] Don’t you think these can be mitigated by testing relevant parts of Beam 
against a number of supported Avro versions the same way parts of Beam are 
tested against various different Hadoop versions.

[2] I agree, this is a huge concern and for that reason I believe Avro cannot 
be turned into a provided dependency. It would be a dangerous trap for users. 
But if the default Avro dependency remains as is and Beam is tested to work 
with a well-defined set of Avro versions (swapping out the default using 
standard dependency resolution rules), I’d consider the risk to be rather 
manageable.


>> Extract Avro as an extension
> I think a change that breaks loudly at compile time (if the features
> in question are being depended on), with a clear fix (e.g. "depend on
> this instead") is the "cleanest" way to do a breaking change.
>
> On this note, would it be possible to do explicit "without avro" and
> "with avro" targets/variants in the core module, with plain old "core"
> being an alias first to the latter, and then to the former, according
> to some deprecation timeline? (I'm not familiar enough with
> gradle/maven to know if this is possible.) This could unblock users
> needing a new avro in the short run while providing a longer-term
> migration path. (Ideally the (relatively limited) avro parts of core
> would also get moved to their own extension under their own namespace,
> and the core one would remain solely for backwards compatibility
> reasons.)
>
>
> Good point. Are you talking about keeping the default “core" artifacts with 
> old Avro version (as for now) as a main dependency and provide additionally 
> other “core” artifacts with a recent Avro version or even as provided for 
> users that need other Avro versions?

I'm thinking about keeping the existing "core" artifact, and also
shipping an [3] "avro-less" core target (whether that be via shading or
total removal, but ripping it out of the public API). We would then
ship a separate "avro" target (TBD if providing the dependency or not
makes the most sense here) that the existing core would depend on and
would be a target for avro users to migrate to.

[3] It wouldn’t be too hard modelling this using feature variants. Though, I’m 
worried about the complexity this creates. In the end, every single Beam module 
depends on core, which would then depend on the new Avro target. Effectively, 
we would need to build two different versions of each module depending on 
either core or the avro-less core. If there would be a well-planned, temporary 
transition period ahead, it should be fine. But I fear with Beam 3 not even 
being on the horizon, we’d have to carry this forward for far too long.


> To finally make progress on this topic, I’d suggest starting off with 
> supporting multiple different Avro versions in core (2) but to keep the 
> default Avro version as is.

As stated above, I think it's safer to make the avroless version a new
target initially than change core.

> At the same time, extracting Avro as extension in a backwards compatibl