Eugene Kirpichov created BEAM-3683:
--------------------------------------

             Summary: Support BigQuery column-based time partitioning
                 Key: BEAM-3683
                 URL: https://issues.apache.org/jira/browse/BEAM-3683
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-gcp
            Reporter: Eugene Kirpichov
            Assignee: Eugene Kirpichov


BigQuery now supports tables partitioned by a DATE or TIMESTAMP column. This is 
very useful for backfilling, because now it doesn't require 1 load job per 
partition (1 load job for the whole table is fine now), and in case of 
BigQueryIO.write(), doesn't require using DynamicDestinations - one only needs 
to specify which field to partition on.

It is specified via TimePartitioning.field: 
[https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load]
 (configuration.load.timePartitioning.field).

Seems that the only thing that's needed is to update the BigQuery client - then 
users can use BigQueryIO.write().withTimePartitioning() in some cases where 
they previously needed to use write().to(DynamicDestinations).

Plus publicity (e.g. a StackOverflow answer)

CC: [~reuvenlax] [~chamikara]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to