Eugene Kirpichov created BEAM-3683:

             Summary: Support BigQuery column-based time partitioning
                 Key: BEAM-3683
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-gcp
            Reporter: Eugene Kirpichov
            Assignee: Eugene Kirpichov

BigQuery now supports tables partitioned by a DATE or TIMESTAMP column. This is 
very useful for backfilling, because now it doesn't require 1 load job per 
partition (1 load job for the whole table is fine now), and in case of 
BigQueryIO.write(), doesn't require using DynamicDestinations - one only needs 
to specify which field to partition on.

It is specified via TimePartitioning.field: 

Seems that the only thing that's needed is to update the BigQuery client - then 
users can use BigQueryIO.write().withTimePartitioning() in some cases where 
they previously needed to use write().to(DynamicDestinations).

Plus publicity (e.g. a StackOverflow answer)

CC: [~reuvenlax] [~chamikara]

This message was sent by Atlassian JIRA

Reply via email to