[
https://issues.apache.org/jira/browse/BEAM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Darshan Mehta updated BEAM-3501:
--------------------------------
Description:
Following is the code that writes to BigQuery:
{code:java}
BigQueryIO.writeTableRows()
.to(destination)
.withCreateDisposition(CREATE_IF_NEEDED)
.withWriteDisposition(WRITE_APPEND)
.withSchema(tableSchema)
.expand(tableRows);{code}
Here's the destination's implementation:
{code:java}
public TableDestination apply(ValueInSingleWindow<TableRow> input) {
String partition = timestampExtractor.apply(input.getValue())
.toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
TableReference tableReference = new TableReference();
tableReference.setDatasetId(dataset);
tableReference.setProjectId(projectId);
tableReference.setTableId(String.format("%s_%s", table, partition));
log.debug("Will write to BigQuery table: %s", tableReference);
return new TableDestination(tableReference, null);
}{code}
When the dataflow tries to write to this table, I see the following message:
{code:java}
"errors" : [ {
"domain" : "global",
"message" : "Cannot read partition information from a table that is not
partitioned: <project_id>:<dataset>.<table>$19730522",
"reason" : "invalid"
} ]{code}
So, it looks like it's not creating tables with partition in the first place?
Apache beam version : 2.2.0
was:
Following is the code that writes to BigQuery:
{code:java}
BigQueryIO.writeTableRows()
.to(destination)
.withCreateDisposition(CREATE_IF_NEEDED)
.withWriteDisposition(WRITE_APPEND)
.withSchema(tableSchema)
.expand(tableRows);{code}
Here's the destination's implementation:
{code:java}
public TableDestination apply(ValueInSingleWindow<TableRow> input) {
String partition = timestampExtractor.apply(input.getValue())
.toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
TableReference tableReference = new TableReference();
tableReference.setDatasetId(dataset);
tableReference.setProjectId(projectId);
tableReference.setTableId(String.format("%s_%s", table, partition));
log.debug("Will write to BigQuery table: %s", tableReference);
return new TableDestination(tableReference, null);
}{code}
When the dataflow tries to write to this table, I see the following message:
{code:java}
"errors" : [ {
"domain" : "global",
"message" : "Cannot read partition information from a table that is not
partitioned: <project_id>:<dataset>.<table>$19730522",
"reason" : "invalid"
} ]{code}
So, it looks like it's not creating tables with partition in the first place?
Apache beam version : 2.2.0
> BigQuery Partitioned table creation/write fails when destination has
> partition decorator
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-3501
> URL: https://issues.apache.org/jira/browse/BEAM-3501
> Project: Beam
> Issue Type: Bug
> Components: beam-model
> Reporter: Darshan Mehta
> Assignee: Kenneth Knowles
> Priority: Major
>
> Following is the code that writes to BigQuery:
> {code:java}
> BigQueryIO.writeTableRows()
> .to(destination)
> .withCreateDisposition(CREATE_IF_NEEDED)
> .withWriteDisposition(WRITE_APPEND)
> .withSchema(tableSchema)
> .expand(tableRows);{code}
>
> Here's the destination's implementation:
> {code:java}
> public TableDestination apply(ValueInSingleWindow<TableRow> input) {
> String partition = timestampExtractor.apply(input.getValue())
> .toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
> TableReference tableReference = new TableReference();
> tableReference.setDatasetId(dataset);
> tableReference.setProjectId(projectId);
> tableReference.setTableId(String.format("%s_%s", table, partition));
> log.debug("Will write to BigQuery table: %s", tableReference);
> return new TableDestination(tableReference, null);
> }{code}
>
> When the dataflow tries to write to this table, I see the following message:
> {code:java}
> "errors" : [ {
> "domain" : "global",
> "message" : "Cannot read partition information from a table that is not
> partitioned: <project_id>:<dataset>.<table>$19730522",
> "reason" : "invalid"
> } ]{code}
> So, it looks like it's not creating tables with partition in the first place?
> Apache beam version : 2.2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
