kmjung commented on pull request #15185: URL: https://github.com/apache/beam/pull/15185#issuecomment-906643295
@satybald the downside of maintaining 100% compatibility directly in the Beam source is that it reduces the efficiency of the pipeline -- you would be paying for the VM cycles to convert every BigQuery `DATETIME` record from string to native Python when using the storage API, for example. An extreme case is on the Java side, where a non-zero number of pipelines read BigQuery data using the read API in Avro format and then [convert it to JSON](https://github.com/apache/beam/blob/3a5b52cd112ab7aad894808cf08775fe93adf7f4/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L599) for consumption by the pipeline. With that said, if your proposal is that we should provide a single, centrally-maintained compat layer that customers can make use of, that makes sense to me. The `DATETIME` issue is particularly galling as an end user -- let's start there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
