[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519
]
Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM:
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
#
##
## ```
was (Author: atdixon):
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
```
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
## at org.apache.beam.sdk.coders.AvroCoder.
([AvroCoder.java:269|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=269=flow-p1-9c38=1])
## at org.apache.beam.sdk.coders.AvroCoder.of
([AvroCoder.java:121|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=121=flow-p1-9c38=1])
## at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
([KafkaUnboundedSource.java:131|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.io.kafka%2FKafkaUnboundedSource.java=131=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
([WorkerCustomSources.java:419|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FWorkerCustomSources.java=419=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
([ReadOperation.java:178|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=178=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
([ReadOperation.java:159|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=159=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
([MapTaskExecutor.java:77|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java=77=flow-p1-9c38=1])
## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
([StreamingDataflowWorker.java:1320|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1320=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519
]
Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM:
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
was (Author: atdixon):
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
#
##
## ```
> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
> Issue Type: Improvement
> Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
> Original Estimate: 24h
> Time Spent: 3h 20m
> Remaining Estimate: 20h 40m
>
> Update build dependency to 1.9.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016551#comment-17016551 ] Tomo Suzuki edited comment on BEAM-8388 at 1/16/20 5:25 AM: Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect from which line of code? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? was (Author: suztomo): Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016551#comment-17016551 ] Tomo Suzuki edited comment on BEAM-8388 at 1/16/20 5:25 AM: Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? was (Author: suztomo): Note for myself: Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
