[
https://issues.apache.org/jira/browse/BEAM-8960?focusedWorklogId=362794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362794
]
ASF GitHub Bot logged work on BEAM-8960:
----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Dec/19 23:58
Start Date: 23/Dec/19 23:58
Worklog Time Spent: 10m
Work Description: yirutang commented on issue #10427: [BEAM-8960]: Add an
option for user to opt out of using insert id for BigQuery streaming insert.
URL: https://github.com/apache/beam/pull/10427#issuecomment-568609169
+Pavan Edara <[email protected]>
On Mon, Dec 23, 2019 at 3:41 PM Chamikara Jayalath <[email protected]>
wrote:
> *@chamikaramj* commented on this pull request.
> ------------------------------
>
> In
>
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
> <https://github.com/apache/beam/pull/10427#discussion_r361027082>:
>
> > + .to("project-id:dataset-id.table-id")
> +
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
> + .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
> + .withSchema(
> + new TableSchema()
> + .setFields(
> + ImmutableList.of(
> + new
TableFieldSchema().setName("name").setType("STRING"),
> + new
TableFieldSchema().setName("number").setType("INTEGER"))))
> + .withTestServices(fakeBqServices)
> + .ignoreInsertIds()
> + .withoutValidation());
> + p.run();
> + // Only row1 and row3 were successfully inserted.
> + assertThat(
> + fakeDatasetService.getAllIds("project-id", "dataset-id",
"table-id"), containsInAnyOrder());
>
> We should set the expected value here ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/beam/pull/10427?email_source=notifications&email_token=AHGYVEXK4AXZU5NUBHC62OTQ2FEC3A5CNFSM4J5PGDCKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCQDY5MQ#pullrequestreview-336039602>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AHGYVETZRJETGYDURXJDL43Q2FEC3ANCNFSM4J5PGDCA>
> .
>
--
Thanks.
Yiru
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 362794)
Remaining Estimate: 22h 40m (was: 22h 50m)
Time Spent: 1h 20m (was: 1h 10m)
> Add an option for user to be able to opt out of using insert id for BigQuery
> streaming insert.
> ----------------------------------------------------------------------------------------------
>
> Key: BEAM-8960
> URL: https://issues.apache.org/jira/browse/BEAM-8960
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
> Reporter: Yiru Tang
> Priority: Minor
> Original Estimate: 24h
> Time Spent: 1h 20m
> Remaining Estimate: 22h 40m
>
> BigQuery streaming insert id offers best effort insert deduplication. If user
> choose to opt out of using insert ids, they could potentially to be opt into
> using our current new streaming backend which gives higher speed and more
> quota. Insert id deduplication is best effort and doesn't have ultimate just
> once guarantees.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
