Chamikara Jayalath created BEAM-3088:
----------------------------------------
Summary: BigQuery source should consider streaming buffer when
determining estimated sizes of tables
Key: BEAM-3088
URL: https://issues.apache.org/jira/browse/BEAM-3088
Project: Beam
Issue Type: Bug
Components: sdk-java-gcp
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath
Currently BigQuery table source determines estimated size using table.numBytes
property.
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java#L100
If BigQuery table has data in the streaming buffer, size of that data will not
be reflected by table.numBytes. To better estimate size of table, data in the
streaming buffer has to be considered as well. Size of data in streaming buffer
can be determined based on property streamingBuffer.estimatedBytes according to
following.
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
