[
https://issues.apache.org/jira/browse/BEAM-7195?focusedWorklogId=573269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573269
]
ASF GitHub Bot logged work on BEAM-7195:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Mar/21 22:42
Start Date: 28/Mar/21 22:42
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#14238:
URL: https://github.com/apache/beam/pull/14238#discussion_r602943526
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
##########
@@ -88,8 +98,35 @@
})
class WriteTables<DestinationT>
extends PTransform<
- PCollection<KV<ShardedKey<DestinationT>, List<String>>>,
- PCollection<KV<TableDestination, String>>> {
+ PCollection<KV<ShardedKey<DestinationT>, WritePartition.Result>>,
+ PCollection<KV<TableDestination, WriteTables.Result>>> {
+ @AutoValue
+ abstract static class Result {
+ abstract String getTableName();
+
+ abstract Boolean isFirstPane();
+ }
+
+ static class ResultCoder extends AtomicCoder<WriteTables.Result> {
Review comment:
Did you mean to redefine Result and ResultCoder classes here ? They are
Already defined in the WritePartition class and definitions seems to be
identical. Can we move these to new files and re-use ?
##########
File path:
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
##########
@@ -65,19 +65,24 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
Review comment:
Is it possible to add a new unit test that would break for the previous
implementation (due to pane numbers being incorrect) but would pass for the new
implementation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 573269)
Time Spent: 4h (was: 3h 50m)
> BigQuery - 404 errors for 'table not found' when using dynamic destinations -
> sometimes, new table fails to get created
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-7195
> URL: https://issues.apache.org/jira/browse/BEAM-7195
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.5.0
> Environment: Windows
> Reporter: Chris
> Priority: P1
> Time Spent: 4h
> Remaining Estimate: 0h
>
> See the following StackOverflow question, which describes the details:
>
> [https://stackoverflow.com/questions/55932291/apache-beam-for-google-cloud-dataflow-404-errors-when-using-bigqueryio-write-c]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
