[
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomo Suzuki updated BEAM-9144:
------------------------------
Description:
>From Aaron's comment in
>https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
{quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
resolved in 1.9.x) so downgrading Avro is not possible for us.
Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are not
available in 1.9.x.
{quote}
The Java class is {{org.apache.avro.data.TimeConversions.TimestampConversion}}
in Avro 1.8.
It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
h1. Beam Java SDK cannot upgrade Avro to 1.9
Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
Illustration of the dependecy
!avro-beam-dependency-graph.png|width=799,height=385!
h1. Short-term Solution
As illustrated above, as long as Beam Java SDK uses only the intersection of
Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
flexibility in runtime Avro versions (as it did until Beam 2.16).
h2. Difference of the TimeConversion Classes
The type parameter is different(DateTime and LocalTime). Does this work?
Avro 1.8's TimeConversions.TimestampConversion:
{code:java}
public static class TimestampConversion extends Conversion<DateTime> {
@Override
public Class<DateTime> getConvertedType() {
return DateTime.class;
}
@Override
public String getLogicalTypeName() {
return "timestamp-millis";
}
@Override
public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
type) {
return new DateTime(millisFromEpoch, DateTimeZone.UTC);
}
@Override
public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
return timestamp.getMillis();
}
}
{code}
Avro 1.9's JodaTimeConversions.TimeConversion:
{code:java}
public static class TimeConversion extends Conversion<LocalTime> {
@Override
public Class<LocalTime> getConvertedType() {
return LocalTime.class;
}
@Override
public String getLogicalTypeName() {
return "time-millis";
}
@Override
public LocalTime fromInt(Integer millisFromMidnight, Schema schema,
LogicalType type) {
return LocalTime.fromMillisOfDay(millisFromMidnight);
}
@Override
public Integer toInt(LocalTime time, Schema schema, LogicalType type) {
return time.millisOfDay().get();
}
@Override
public Schema getRecommendedSchema() {
return
LogicalTypes.timeMillis().addToSchema(Schema.create(Schema.Type.INT));
}
}
{code}
was:
>From Aaron's comment in
>https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
{quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
resolved in 1.9.x) so downgrading Avro is not possible for us.
Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are not
available in 1.9.x.
{quote}
The Java class is {{org.apache.avro.data.TimeConversions.TimestampConversion}}
in Avro 1.8.
It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
h1. Beam Java SDK cannot upgrade Avro to 1.9
Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
Illustration of the dependecy
!avro-beam-dependency-graph.png|width=799,height=385!
h1. Short-term Solution
As illustrated above, as long as Beam Java SDK uses only the intersection of
Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
flexibility in runtime Avro versions (as it did until Beam 2.16).
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> -----------------------------------------------------------
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Tomo Suzuki
> Assignee: Tomo Suzuki
> Priority: Major
> Attachments: avro-beam-dependency-graph.png
>
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependecy
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> The type parameter is different(DateTime and LocalTime). Does this work?
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion<DateTime> {
> @Override
> public Class<DateTime> getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimeConversion:
> {code:java}
> public static class TimeConversion extends Conversion<LocalTime> {
> @Override
> public Class<LocalTime> getConvertedType() {
> return LocalTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "time-millis";
> }
> @Override
> public LocalTime fromInt(Integer millisFromMidnight, Schema schema,
> LogicalType type) {
> return LocalTime.fromMillisOfDay(millisFromMidnight);
> }
> @Override
> public Integer toInt(LocalTime time, Schema schema, LogicalType type) {
> return time.millisOfDay().get();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timeMillis().addToSchema(Schema.create(Schema.Type.INT));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
