Thanks for the contribution. I can take a look later this week. On Wed, Nov 28, 2018 at 12:29 AM Wout Scheepers < [email protected]> wrote:
> Hey all, > > > > Almost two weeks ago, I create a PR to support BigQuery clustering [1]. > > Can someone please have a look? > > > > Thanks, > > Wout > > > > 1: https://github.com/apache/beam/pull/7061 > > > > > > *From: *Lukasz Cwik <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, 29 August 2018 at 18:32 > *To: *dev <[email protected]>, "[email protected]" < > [email protected]> > *Cc: *Bob De Schutter <[email protected]> > *Subject: *Re: BigqueryIO field clustering > > > > [email protected] > > > > Wout, I assigned this task to you since it seems like your interested in > contributing. > > The Apache Beam contribution guide[1] is a good place to start for > answering questions on how to contribute. > > > > If you need help in getting stuff reviewed or having questions, feel free > to reach out on [email protected] or on Slack. > > > > 1: https://beam.apache.org/contribute/ > > > > > > On Wed, Aug 29, 2018 at 1:28 AM Wout Scheepers < > [email protected]> wrote: > > Hey all, > > > > I’m trying to use the field clustering beta feature in bigquery [1]. > > However, the current Beam/dataflow worker bigquery api service dependency > is ‘google-api-services-bigquery: com.google.apis: v2-rev374-1.23.0’, which > does not include the clustering option in the TimePartitioning class. > > Hereby, I can’t specify the clustering field when loading/streaming into > bigquery. See [2] for the bigquery api error details. > > > > Does anyone know a workaround for this? > > > > I guess that in the worst case I’ll have to wait until Beam supports a > newer version of the bigquery api service. > > 1. After checking the Beam Jira I’ve found BEAM-5191 > <https://jira.apache.org/jira/browse/BEAM-5191>. Is there any way I can > help to push this forward and make this feature possible in the near future? > > > > Thanks in advance, > > Wout > > > > [1] https://cloud.google.com/bigquery/docs/clustered-tables > > [2] "errorResult" : { > > "message" : "Incompatible table partitioning specification. Expects > partitioning specification interval(type:day,field:publish_time) > clustering(clustering_id), but input partitioning specification is > interval(type:day,field:publish_time)", > > "reason" : "invalid" > > } > >
