[
https://issues.apache.org/jira/browse/BEAM-6076?focusedWorklogId=167328&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167328
]
ASF GitHub Bot logged work on BEAM-6076:
----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Nov/18 09:47
Start Date: 19/Nov/18 09:47
Worklog Time Spent: 10m
Work Description: lgajowy closed pull request #7066: [BEAM-6076] Fetch
only required fields for BigQuery Table
URL: https://github.com/apache/beam/pull/7066
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryClient.java
b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryClient.java
index fd8b4823d2a4..5a47afe18e49 100644
---
a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryClient.java
+++
b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryClient.java
@@ -20,6 +20,7 @@
import static java.lang.String.format;
import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.BigQuery.TableOption;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.InsertAllRequest;
@@ -40,6 +41,11 @@
*/
public class BigQueryClient {
+ // Fetching all fields (default way) may result in NullPointerException from
the client library:
+ // https://issues.apache.org/jira/browse/BEAM-6076.
+ // Specifying only TYPE field enforces getting only the required fields (not
null).
+ private static final TableOption FIELD_OPTIONS =
TableOption.fields(BigQuery.TableField.TYPE);
+
private BigQuery client;
private String projectId;
@@ -71,7 +77,7 @@ public static BigQueryClient create(String dataset) {
public void createTableIfNotExists(String tableName, Map<String, String>
schema) {
TableId tableId = TableId.of(projectId, dataset, tableName);
- if (client.getTable(tableId) == null) {
+ if (client.getTable(tableId, FIELD_OPTIONS) == null) {
List<Field> schemaFields =
schema
.entrySet()
@@ -89,7 +95,7 @@ private void createTable(TableId tableId, Schema schema) {
.setFriendlyName(tableId.getTable())
.build();
- client.create(tableInfo);
+ client.create(tableInfo, FIELD_OPTIONS);
}
/**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 167328)
Time Spent: 50m (was: 40m)
> NEXMark flakiness: NPE thrown from BigQuery client library once in a while
> --------------------------------------------------------------------------
>
> Key: BEAM-6076
> URL: https://issues.apache.org/jira/browse/BEAM-6076
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> It seems that once in a while the library that is used to connect to BigQuery
> throws exceptions like this:
> {code:java}
> Exception in thread "main"
> java.lang.NullPointerException
> at
> com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.fromPb(StandardTableDefinition.java:116)
> at
> com.google.cloud.bigquery.StandardTableDefinition.fromPb(StandardTableDefinition.java:225)
> at com.google.cloud.bigquery.TableDefinition.fromPb(TableDefinition.java:155)
> at com.google.cloud.bigquery.TableInfo$BuilderImpl.<init>(TableInfo.java:183)
> at com.google.cloud.bigquery.Table.fromPb(Table.java:593)
> at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:410)
> at
> org.apache.beam.sdk.testutils.publishing.BigQueryClient.createTableIfNotExists(BigQueryClient.java:74)
> at org.apache.beam.sdk.nexmark.Main.savePerfsToBigQuery(Main.java:184)
> at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:148)
> at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:98)
> at org.apache.beam.sdk.nexmark.Main.main(Main.java:423){code}
> Similar error found in the network:
> [https://github.com/googleapis/google-cloud-java/issues/1689]
> +Example logs:+
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Spark/1085/console]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
