[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131203#comment-17131203 ] Beam JIRA Bot commented on BEAM-8388: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Priority: P2 > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121853#comment-17121853 ] Kenneth Knowles commented on BEAM-8388: --- This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: P2 > Labels: stale-assigned > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018074#comment-17018074 ] Tomo Suzuki commented on BEAM-8388: --- > just to implement such conversions on Beam side in the meantime. I was thinking the same. Let me take that. https://issues.apache.org/jira/browse/BEAM-9144 > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018030#comment-17018030 ] Ismaël Mejía commented on BEAM-8388: So the solution for this specific case would be just to implement such conversions on Beam side in the meantime. Maybe worth to do a PR (+ different issue for that). Anyone willing to do this? I don't have the context but from a quicklook seems like a renaming in the Avro side to do more precise time conversions. Notice however that JodaTime is removed starting with Avro 1.10 so that's something that we also have to be aware of. > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017595#comment-17017595 ] Tomo Suzuki commented on BEAM-8388: --- (Answering my question) bq. Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? Yes, this change was added since 2.17. https://github.com/apache/beam/commit/13cea03cb6e8f8f48e0ac0ae64d60928df1fd9d5 > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519 ] Aaron Dixon commented on BEAM-8388: --- Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): ``` java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # ## at org.apache.beam.sdk.coders.AvroCoder. ([AvroCoder.java:269|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=269=flow-p1-9c38=1]) ## at org.apache.beam.sdk.coders.AvroCoder.of ([AvroCoder.java:121|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=121=flow-p1-9c38=1]) ## at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder ([KafkaUnboundedSource.java:131|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.io.kafka%2FKafkaUnboundedSource.java=131=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator ([WorkerCustomSources.java:419|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FWorkerCustomSources.java=419=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop ([ReadOperation.java:178|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=178=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start ([ReadOperation.java:159|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=159=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute ([MapTaskExecutor.java:77|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java=77=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process ([StreamingDataflowWorker.java:1320|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1320=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000 ([StreamingDataflowWorker.java:151|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=151=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run ([StreamingDataflowWorker.java:1053|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1053=flow-p1-9c38=1]) # ## at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142) ## at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) ## at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.avro.data.TimeConversions$TimestampConversion # ## at java.net.URLClassLoader.findClass ([URLClassLoader.java:381|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Fnet%2FURLClassLoader.java=381=flow-p1-9c38=1]) ## at java.lang.ClassLoader.loadClass ([ClassLoader.java:424|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Flang%2FClassLoader.java=424=flow-p1-9c38=1]) ## at sun.misc.Launcher$AppClassLoader.loadClass ([Launcher.java:335|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=sun%2Fmisc%2FLauncher.java=335=flow-p1-9c38=1]) ## at java.lang.ClassLoader.loadClass ([ClassLoader.java:357|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Flang%2FClassLoader.java=357=flow-p1-9c38=1]) ## ``` > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL:
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016551#comment-17016551 ] Tomo Suzuki commented on BEAM-8388: --- Note for myself: Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016525#comment-17016525 ] Aaron Dixon commented on BEAM-8388: --- More context just in case it is helpful: I didn't discover this first through the PR. I upgraded to Beam 2.17.0 and ran my job (in Dataflow) and got the slew of ClassNotFoundExceptions for TimestampConversion coming from AvroCoder.. > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016514#comment-17016514 ] Tomo Suzuki commented on BEAM-8388: --- Thanks > TimestampConversion > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016510#comment-17016510 ] Aaron Dixon commented on BEAM-8388: --- [~suztomo] 2.17.0 references a class `TimestampConversion` that is only in Avro 1.8.x. It appears that the PR you referenced (that didn't get merged) shows the required change away from using `TimestampConversion`: [https://github.com/apache/beam/pull/9779/files] Regarding Ken's comment – wouldn't shading Avro be an apropos solution? This is a very common approach in dealing with situations like this. (See [https://maven.apache.org/plugins/maven-shade-plugin/]) In this way Beam could (internally, for AvroCoder, etc) use whichever Avro version it prefers while clients could then have direct dependencies on any Avro version and not conflict with Beam's? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016493#comment-17016493 ] Tomo Suzuki commented on BEAM-8388: --- [~atdixon] I don't have a solution but where did you find these exactly? bq. 2.17.0 links to Java classes in Avro 1.8.x that are not available in 1.9.x > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016490#comment-17016490
]
Tomo Suzuki commented on BEAM-8388:
---
The [PR#9779|https://github.com/apache/beam/pull/9779] did not get merged.
Comments on with Avro version incompatibility with other components. The last
comment from Ken:
{quote}Seems like including Avro in the core SDK might be a mistake / legacy
problem. We could potentially create new artifacts for particular versions so
users can choose compatible pieces of Beam.
{quote}
> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
> Issue Type: Improvement
> Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
> Original Estimate: 24h
> Time Spent: 3h 20m
> Remaining Estimate: 20h 40m
>
> Update build dependency to 1.9.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016476#comment-17016476 ] Aaron Dixon commented on BEAM-8388: --- I'd like to add a vote to this ticket with some context: My org must use Avro 1.9.x (due to some Avro schema resolution issues resolved in 1.9.x) so downgrading Avro is not possible for us. Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are not available in 1.9.x. So we are stuck on 2.16.0 until Beam can eliminate the hard dependency on 1.8.x. I see that there is [this now closed PR|[https://github.com/apache/beam/pull/9779]] that hoped to address this. Would shading Avro (maven-shade-plugin) be an option for Beam here - so that clients could use their own version of Avro. (For context, for us, we use Avro to parse events from a KafkaIO source; it would be perfectly fine if Beam internally used a shaded 1.8.x version to do its own coding/serde.) > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
