Just wanted to clarify: BigQuery BIGNUMERIC type costs more than NUMERIC type, so if NUMERIC is sufficient, the users likely won't want to switch to BIGNUMERIC.
Does Beam DECIMAL datatype contain the precision/scale parameters in the metadata? If so, can we use those parameters to determine the mapped type? On Thu, Mar 18, 2021 at 12:08 PM Brian Hulette <bhule...@google.com> wrote: > Hi Vachan, > I don't think Beam DECIMAL is really a great mapping for either BigQuery's > NUMERIC or BIGNUMERIC type. Beam's DECIMAL represents arbitrary precision > decimals [1] to map well to Java's BigDecimal class [2]. > > Maybe we should add a fixed-precision decimal logical type [3], then have > specific instances of it with the appropriate precision that map to NUMERIC > and BIGNUMERIC. We could also shunt Beam DECIMAL to BIGNUMERIC for > compatibility. > > [1] > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L424 > [2] https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html > [3] > https://github.com/apache/beam/tree/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes > > On Thu, Mar 18, 2021 at 12:00 PM Vachan Shetty <vac...@google.com> wrote: > >> Hello, I am currently trying to add support for BigQuery's new BIGNUMERIC >> datatype [1] in Beam's BigQueryIO. I am currently following the steps that >> were used for adding the NUMERIC datatype [2]. AFAICT Beam's DECIMAL is >> the most appropriate datatype to map to BIGNUMERIC in BQ. However, the >> Beam DECIMAL datatype is already mapped to NUMERIC in BQ [2, 3]. Given >> this, should I simply map all Beam DECIMAL to BQ BIGNUMERIC? Or should this >> conversion be done based on other information? [1]: >> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types >> [2]: https://issues.apache.org/jira/browse/BEAM-4417 [3]: >> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L188 >> > -- Thanks, Mingyu
