[ 
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020344#comment-17020344
 ] 

Aaron Dixon commented on BEAM-9144:
-----------------------------------

[~suztomo] This appears to be a runner issue:

The exception (class load) is not occurring from my code but from the Dataflow 
runner system, from `StreamingDataflowWorker` class.

* beam-2.16.0 transitively includes `org.apache.beam.vendor.grpc.v1p21p0.*` 
packages.
   * The Dataflow runtime attempts to load v1p21p0 classes and finds them 
present.
* beam-2.20.0-SNAPS transitively includes 
`org.apache.beam.vendor.grpc.v1p26p0.*` packages (_not_ v1p21p0 classes).
   * Dataflow's runtime system tries to load `*.v1p21p0.*` but no longer finds 
them present. I expect it _should_ be attempting to load `*.v1p26p0.*` classes 
when the SDK version is 2.20.0-* .

Thoughts?

> Beam's own Avro TimeConversion class in beam-sdk-java-core 
> -----------------------------------------------------------
>
>                 Key: BEAM-9144
>                 URL: https://issues.apache.org/jira/browse/BEAM-9144
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Tomo Suzuki
>            Assignee: Tomo Suzuki
>            Priority: Major
>             Fix For: 2.19.0
>
>         Attachments: avro-beam-dependency-graph.png
>
>          Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> From Aaron's comment in 
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
>  .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues 
> resolved in 1.9.x) so downgrading Avro is not possible for us.
>  Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to 
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are 
> not available in 1.9.x.
> {quote}
> The Java class is 
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
>  It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of 
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide 
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method. 
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
>   public static class TimestampConversion extends Conversion<DateTime> {
>     @Override
>     public Class<DateTime> getConvertedType() {
>       return DateTime.class;
>     }
>     @Override
>     public String getLogicalTypeName() {
>       return "timestamp-millis";
>     }
>     @Override
>     public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType 
> type) {
>       return new DateTime(millisFromEpoch, DateTimeZone.UTC);
>     }
>     @Override
>     public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
>       return timestamp.getMillis();
>     }
>   }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
>   public static class TimestampConversion extends Conversion<DateTime> {
>     @Override
>     public Class<DateTime> getConvertedType() {
>       return DateTime.class;
>     }
>     @Override
>     public String getLogicalTypeName() {
>       return "timestamp-millis";
>     }
>     @Override
>     public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType 
> type) {
>       return new DateTime(millisFromEpoch, DateTimeZone.UTC);
>     }
>     @Override
>     public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
>       return timestamp.getMillis();
>     }
>     @Override
>     public Schema getRecommendedSchema() {
>       return 
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
>     }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to