[
https://issues.apache.org/jira/browse/BEAM-876?focusedWorklogId=309094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309094
]
ASF GitHub Bot logged work on BEAM-876:
---------------------------------------
Author: ASF GitHub Bot
Created on: 09/Sep/19 17:28
Start Date: 09/Sep/19 17:28
Worklog Time Spent: 10m
Work Description: ziel commented on pull request #9524: [BEAM-876]
Support schemaUpdateOption in BigQueryIO
URL: https://github.com/apache/beam/pull/9524
# [BEAM-876] Support schemaUpdateOption in BigQueryIO
This adds schemaUpdateOptions to `BigQueryIO.Write` so that one can specify
these for BiqQuery when writing in batch mode.
Usage example:
```
Write<TableRow> writer = BigQueryIO.writeTableRows()
.to("name-goes-here")
.withMethod(BigQueryIO.Write.Method.FILE_LOADS)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
.withSchemaUpdateOptions(EnumSet.of(BigQueryIO.Write.SchemaUpdate.Option.ALLOW_FIELD_ADDITION));
```
**Implementation Notes**
*Hi, hello:*
Hi all. I haven't contributed to this code base before, and am not super
familiar with it. Style advice and such is super welcome.
*Load vs Query Jobs in BigQuery:*
BigQuery supports schema update side effects for load and query jobs. This
implements support for load jobs only. In the context of Apache Beam, it
doesn't seem like there's much utility to supporting queries which write to a
table directly (as opposed to reading via a query and writing later in the
pipeline). (Open to discussion/correction of course).
*Avoiding Changing WriteTables Constructor:*
I've added this to `WriteTables` with a `withSchemaUpdateOptions` method so
as not to perturb the public constructor with an optional parameter. There's
some awkwardness there -- but it seemed preferable to breaking the api or
adding a secondary constructor.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/)
| --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 309094)
Remaining Estimate: 0h
Time Spent: 10m
> Support schemaUpdateOption in BigQueryIO
> ----------------------------------------
>
> Key: BEAM-876
> URL: https://issues.apache.org/jira/browse/BEAM-876
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Eugene Kirpichov
> Assignee: canaan silberberg
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BigQuery recently added support for updating the schema as a side effect of
> the load job.
> Here is the relevant API method in JobConfigurationLoad:
> https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setSchemaUpdateOptions(java.util.List)
> BigQueryIO should support this too. See user request for this:
> http://stackoverflow.com/questions/40333245/is-it-possible-to-update-schema-while-doing-a-load-into-an-existing-bigquery-tab
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
