[
https://issues.apache.org/jira/browse/BEAM-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci updated BEAM-2122:
-------------------------------
Fix Version/s: (was: First stable release)
> Writing to partitioned BigQuery tables from Dataflow is causing errors
> ----------------------------------------------------------------------
>
> Key: BEAM-2122
> URL: https://issues.apache.org/jira/browse/BEAM-2122
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
> Environment: Running with Beam 0.7.0-SNAPSHOT version 48 for
> beam-sdks-java-io-google-cloud-platform, 49 for beam-sdks-java-core and
> beam-runners-google-cloud-dataflow-java in Eclipse using Dataflow service.
> Reporter: Matthias Baetens
> Assignee: Reuven Lax
>
> Using the latest Beam SNAPSHOT which has a new BigQuery connector and trying
> to write to partitioned tables according to the docs (or this Stackoverflow
> question
> http://stackoverflow.com/questions/43505534/writing-different-values-to-different-bigquery-tables-in-apache-beam/43655461#43655461):
> static class PartitionedTableGeneration
> implements
> SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination> {
> @ProcessElement
> public TableDestination apply(ValueInSingleWindow<TableRow>
> value) {
> // String dayString =
> //
> DateTimeFormat.forPattern("yyyy_MM_dd").withZone(DateTimeZone.UTC)
> String dayString =
> DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC)
> .print(((IntervalWindow)
> value.getWindow()).start());
> TableDestination td = new TableDestination(
> "projecet:dataset.table + '$'
> dayString, "");
> return td;
> }
> }
> causes the following issues when running (depending on the specification of
> the dayString):
> 1. "Invalid table ID \"partitioned_sample$20150905\". Table IDs must be
> alphanumeric (plus underscores) and must be at most 1024 characters long.
> Also, Table decorators cannot be used.",
> 2. java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException:
> java.lang.RuntimeException: Failed to create load job with id prefix
> ...
> "errorResult" : {
> "message" : "Invalid date partitioned table suffix: 2015_11_26",
> "reason" : "invalid"
> }
> Writing to sharded tables (without the '$'-sign) is working fine.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
