[
https://issues.apache.org/jira/browse/BEAM-9144?focusedWorklogId=373643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373643
]
ASF GitHub Bot logged work on BEAM-9144:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jan/20 15:10
Start Date: 17/Jan/20 15:10
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10628: [BEAM-9144]
Beam's own Avro TimeConversion class in beam-sdk-java-core
URL: https://github.com/apache/beam/pull/10628
https://issues.apache.org/jira/browse/BEAM-9144
This is a copy from Avro 1.8's TimestampConversion, which is renamed in Avro
1.9. Defining own copy gives flexibility for Beam Java SDK to work with Avro
1.8 and 1.9 at runtime.
CC: @iemejia
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/)
| --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 373643)
Remaining Estimate: 0h
Time Spent: 10m
> Beam's own Avro TimeConversion class in beam-sdk-java-core
> -----------------------------------------------------------
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Tomo Suzuki
> Assignee: Tomo Suzuki
> Priority: Major
> Attachments: avro-beam-dependency-graph.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From Aaron's comment in
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
> .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues
> resolved in 1.9.x) so downgrading Avro is not possible for us.
> Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are
> not available in 1.9.x.
> {quote}
> The Java class is
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
> It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method.
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion<DateTime> {
> @Override
> public Class<DateTime> getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
> public static class TimestampConversion extends Conversion<DateTime> {
> @Override
> public Class<DateTime> getConvertedType() {
> return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType
> type) {
> return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
> return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
