[
https://issues.apache.org/jira/browse/BEAM-9051?focusedWorklogId=371319&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371319
]
ASF GitHub Bot logged work on BEAM-9051:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Jan/20 03:12
Start Date: 14/Jan/20 03:12
Worklog Time Spent: 10m
Work Description: RyanBerti commented on pull request #10540: [BEAM-9051]
BigQueryUtils toBeamRow support the Avro timestamp-millis logical type
URL: https://github.com/apache/beam/pull/10540#discussion_r366135078
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
##########
@@ -390,6 +394,35 @@ public static Row toBeamRow(GenericRecord record, Schema
schema, ConversionOptio
return Row.withSchema(schema).addValues(valuesInOrder).build();
}
+ /**
+ * Convert to a Beam {@link Row} from an Avro {@link GenericRecord}, using
the avro schema to
+ * differentiate between Avro {@link org.apache.avro.LogicalType}.
+ */
+ public static Row toBeamRow(
+ GenericRecord record,
+ org.apache.avro.Schema avroSchema,
+ Schema beamSchema,
+ ConversionOptions options) {
Review comment:
Yes - the goal was to add support to the public API, as I found I couldn't
rely on the existing implementation of convertAvroFormat. I didn't realize
there was an equivalent method in AvroUtils! I'll test it out and see if I can
use that one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 371319)
Time Spent: 1h 40m (was: 1.5h)
> BigQueryUtils toBeamRow to support GenericRecords with millisecond precision
> timestamps
> ---------------------------------------------------------------------------------------
>
> Key: BEAM-9051
> URL: https://issues.apache.org/jira/browse/BEAM-9051
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.16.0
> Reporter: Ryan Berti
> Assignee: Ryan Berti
> Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently, BigQueryUtils assumes all Avro GenericRecords which utilize
> timestamp fields include timestamp data in microseconds when converting from
> GenericRecords to Beam Rows
> ([https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L568]).
> The implementation should also support timestamp data in milliseconds; this
> should be feasible by referencing the LogicalType associated with the avro
> field to determine if the underlying value is represented in millis or micros.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
