kkdoon commented on code in PR #23620:
URL: https://github.com/apache/beam/pull/23620#discussion_r1017277186
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java:
##########
@@ -448,4 +449,32 @@ private static Field convertField(TableFieldSchema
bigQueryField) {
bigQueryField.getDescription(),
(Object) null /* Cast to avoid deprecated JsonNode constructor. */);
}
+
+ private static Schema handleAvroLogicalTypes(TableFieldSchema bigQueryField,
Type avroType) {
+ String bqType = bigQueryField.getType();
+ switch (bqType) {
+ case "NUMERIC":
+ // Default value based on
+ //
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types
+ int precision =
Optional.ofNullable(bigQueryField.getPrecision()).orElse(38L).intValue();
+ int scale =
Optional.ofNullable(bigQueryField.getScale()).orElse(9L).intValue();
+ return LogicalTypes.decimal(precision,
scale).addToSchema(Schema.create(Type.BYTES));
+ case "BIGNUMERIC":
+ // Default value based on
+ //
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types
+ int precisionBigNumeric =
+
Optional.ofNullable(bigQueryField.getPrecision()).orElse(77L).intValue();
Review Comment:
I ran a beam job and tried reading from an actual BigQuery table with
BIGNUMERIC field. The [writer
schema](https://github.com/apache/beam/blob/483a0c95734c528aa45419596e9f27e9e650c5d7/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L754),
set in the extracted Avro file, has precision value set as 77 and the scale is
set to 38, for fields that have no explicit precision/scale specified.
I also tried inserting the max supported value that has a precision of 77
and it works as long as its <=
578960446186580977117854925043439539266.34992332820282019728792003956564819967
(since 77th digit is partial it supports this value and not
999999999999999999999999999999999999999.99999999999999999999999999999999999999
instead)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
