Darshan Mehta created BEAM-3501:
-----------------------------------
Summary: BigQuery Partitioned table creation/write fails when
destination has partition decorator
Key: BEAM-3501
URL: https://issues.apache.org/jira/browse/BEAM-3501
Project: Beam
Issue Type: Bug
Components: beam-model
Reporter: Darshan Mehta
Assignee: Kenneth Knowles
Following is the code that writes to BigQuery:
{code:java}
BigQueryIO.writeTableRows()
.to(destination)
.withCreateDisposition(CREATE_IF_NEEDED)
.withWriteDisposition(WRITE_APPEND)
.withSchema(tableSchema)
.expand(tableRows);{code}
Here's the destination's implementation:
{code:java}
public TableDestination apply(ValueInSingleWindow<TableRow> input) {
String partition = timestampExtractor.apply(input.getValue())
.toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
TableReference tableReference = new TableReference();
tableReference.setDatasetId(dataset);
tableReference.setProjectId(projectId);
tableReference.setTableId(String.format("%s_%s", table, partition));
log.debug("Will write to BigQuery table: %s", tableReference);
return new TableDestination(tableReference, null);
}{code}
When the dataflow tries to write to this table, I see the following message:
{code:java}
"errors" : [ {
"domain" : "global",
"message" : "Cannot read partition information from a table that is not
partitioned: <project_id>:<dataset>.<table>$19730522",
"reason" : "invalid"
} ]{code}
So, it looks like it's not creating tables with partition in the first place?
Apache beam version : 2.2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
