[
https://issues.apache.org/jira/browse/BEAM-9051?focusedWorklogId=371272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371272
]
ASF GitHub Bot logged work on BEAM-9051:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Jan/20 01:47
Start Date: 14/Jan/20 01:47
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10540:
[BEAM-9051] BigQueryUtils toBeamRow support the Avro timestamp-millis logical
type
URL: https://github.com/apache/beam/pull/10540#discussion_r366113970
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
##########
@@ -390,6 +394,35 @@ public static Row toBeamRow(GenericRecord record, Schema
schema, ConversionOptio
return Row.withSchema(schema).addValues(valuesInOrder).build();
}
+ /**
+ * Convert to a Beam {@link Row} from an Avro {@link GenericRecord}, using
the avro schema to
+ * differentiate between Avro {@link org.apache.avro.LogicalType}.
+ */
+ public static Row toBeamRow(
+ GenericRecord record,
+ org.apache.avro.Schema avroSchema,
+ Schema beamSchema,
+ ConversionOptions options) {
Review comment:
I don't actually see this new function used anywhere outside of tests, are
you adding it as a public API for general purpose Avro -> Row conversion? If so
I think something like this change may be better in
[`AvroUtils`](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java).
In fact, I think much of the existing Avro conversion code in this class
should be moved there, or should defer to it.
@kennknowles why do we have so much Avro to Row code here rather than
deferring to the schema conversion code, is it because this predates
org.apache.beam.sdk.schemas?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 371272)
Time Spent: 1.5h (was: 1h 20m)
> BigQueryUtils toBeamRow to support GenericRecords with millisecond precision
> timestamps
> ---------------------------------------------------------------------------------------
>
> Key: BEAM-9051
> URL: https://issues.apache.org/jira/browse/BEAM-9051
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.16.0
> Reporter: Ryan Berti
> Assignee: Ryan Berti
> Priority: Minor
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently, BigQueryUtils assumes all Avro GenericRecords which utilize
> timestamp fields include timestamp data in microseconds when converting from
> GenericRecords to Beam Rows
> ([https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L568]).
> The implementation should also support timestamp data in milliseconds; this
> should be feasible by referencing the LogicalType associated with the avro
> field to determine if the underlying value is represented in millis or micros.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
