[
https://issues.apache.org/jira/browse/BEAM-4835?focusedWorklogId=125350&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125350
]
ASF GitHub Bot logged work on BEAM-4835:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Jul/18 05:33
Start Date: 20/Jul/18 05:33
Worklog Time Spent: 10m
Work Description: gene-telligent opened a new pull request #6008:
[BEAM-4835] Adding in additional options for BigQueryIO insert statements
URL: https://github.com/apache/beam/pull/6008
Adds in two flags for BigQueryIO.Write, skipInvalidRows and
ignoreUnknownValues. These are passed through to the HTTP API. More details in
the [corresponding JIRA
ticket.](https://issues.apache.org/jira/browse/BEAM-4835)
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [x] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 125350)
Time Spent: 10m
Remaining Estimate: 0h
> Add more flexible options for data loading to BigQueryIO.Write
> --------------------------------------------------------------
>
> Key: BEAM-4835
> URL: https://issues.apache.org/jira/browse/BEAM-4835
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Gene Peters
> Assignee: Chamikara Jayalath
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of the BigQuery API, there are a few options exposed to end-users
> which allow for more flexible data loading.
> For both
> [streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-]
> and
> [batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-]
> inserts, the flag "ignoreUnknownValues" can be set, which indicates if
> BigQuery should accept rows that contain values that do not match the schema.
> [In
> addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-]
> streaming inserts allow for the option of accepting an inserted batch of
> rows even if some of of the rows are invalid.
> I've made the necessary code changes to make this available within
> BigQueryIO.Write and will be attaching the pull request to this ticket for
> review. Both flags are off by default.
> Let me know if you have any questions or feedback about this!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
