[
https://issues.apache.org/jira/browse/BEAM-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678911#comment-16678911
]
Charles Chen edited comment on BEAM-6014 at 11/7/18 10:55 PM:
--------------------------------------------------------------
I can confirm that this problem occurs for me too. For more info, I added
debugging statements and found that the InsertAll requests all return from
BigQuery without error, but the table ends up without output:
{code}
BigQueryWriteFn {'tornado_count': 16, 'month': 1}
BigQueryWriteFn {'tornado_count': 6, 'month': 3}
BigQueryWriteFn {'tornado_count': 7, 'month': 2}
BigQueryWriteFn {'tornado_count': 6, 'month': 5}
BigQueryWriteFn {'tornado_count': 5, 'month': 4}
BigQueryWriteFn {'tornado_count': 8, 'month': 7}
BigQueryWriteFn {'tornado_count': 5, 'month': 6}
BigQueryWriteFn {'tornado_count': 7, 'month': 9}
BigQueryWriteFn {'tornado_count': 4, 'month': 8}
BigQueryWriteFn {'tornado_count': 9, 'month': 11}
BigQueryWriteFn {'tornado_count': 10, 'month': 10}
BigQueryWriteFn {'tornado_count': 10, 'month': 12}
FINISH BUNDLE
FLUSH BATCH
_insert_all_rows REQUEST <BigqueryTabledataInsertAllRequest
datasetId: 'mydatasetname'
projectId: 'myproject'
tableDataInsertAllRequest: <TableDataInsertAllRequest
rows: [<RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_1'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 16>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 1>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_2'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 6>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 3>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_3'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 7>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 2>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_4'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 6>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 5>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_5'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 5>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 4>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_6'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 8>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 7>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_7'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 5>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 6>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_8'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 7>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 9>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_9'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 4>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 8>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_10'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 9>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 11>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_11'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 10>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 10>>]>>, <RowsValueListEntry
insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_12'
json: <JsonObject
additionalProperties: [<AdditionalProperty
key: 'tornado_count'
value: <JsonValue
integer_value: 10>>, <AdditionalProperty
key: 'month'
value: <JsonValue
integer_value: 12>>]>>]>
tableId: 'out'>
_insert_all_rows RESPONSE <TableDataInsertAllResponse
insertErrors: []
kind: u'bigquery#mydatasetname'>
BigQuery table: [MY_PROJECT:myproject.out]; errors: []
Successfully wrote 12 rows.
{code}
was (Author: charleschen):
I can confirm that this problem occurs for me too. For more info, I added
debugging statements and found that the InsertAll requests all return from
BigQuery without error, but the table ends up without output:
{{BigQueryWriteFn \{'tornado_count': 16, 'month': 1}}}
{{BigQueryWriteFn \{'tornado_count': 6, 'month': 3}}}
{{BigQueryWriteFn \{'tornado_count': 7, 'month': 2}}}
{{BigQueryWriteFn \{'tornado_count': 6, 'month': 5}}}
{{BigQueryWriteFn \{'tornado_count': 5, 'month': 4}}}
{{BigQueryWriteFn \{'tornado_count': 8, 'month': 7}}}
{{BigQueryWriteFn \{'tornado_count': 5, 'month': 6}}}
{{BigQueryWriteFn \{'tornado_count': 7, 'month': 9}}}
{{BigQueryWriteFn \{'tornado_count': 4, 'month': 8}}}
{{BigQueryWriteFn \{'tornado_count': 9, 'month': 11}}}
{{BigQueryWriteFn \{'tornado_count': 10, 'month': 10}}}
{{BigQueryWriteFn \{'tornado_count': 10, 'month': 12}}}
{{FINISH BUNDLE}}
{{FLUSH BATCH}}
{{_insert_all_rows REQUEST <BigqueryTabledataInsertAllRequest}}
{{ datasetId: 'mydatasetname'}}
{{ projectId: 'myproject'}}
{{ tableDataInsertAllRequest: <TableDataInsertAllRequest}}
{{ rows: [<RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_1'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 16>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 1>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_2'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 6>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 3>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_3'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 7>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 2>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_4'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 6>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 5>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_5'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 5>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 4>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_6'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 8>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 7>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_7'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 5>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 6>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_8'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 7>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 9>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_9'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 4>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 8>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_10'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 9>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 11>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_11'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 10>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 10>>]>>, <RowsValueListEntry}}
{{ insertId: '189c7935-0982-4ca2-a959-0b3de67efbd9_12'}}
{{ json: <JsonObject}}
{{ additionalProperties: [<AdditionalProperty}}
{{ key: 'tornado_count'}}
{{ value: <JsonValue}}
{{ integer_value: 10>>, <AdditionalProperty}}
{{ key: 'month'}}
{{ value: <JsonValue}}
{{ integer_value: 12>>]>>]>}}
{{ tableId: 'out'>}}
{{_insert_all_rows RESPONSE <TableDataInsertAllResponse}}
{{ insertErrors: []}}
{{ kind: u'bigquery#mydatasetname'>}}
{{BigQuery table: [MY_PROJECT:myproject.out]; errors: []}}
{{Successfully wrote 12 rows.}}
> bigquery_tornadoes example does not work on Python Direct Runner
> ----------------------------------------------------------------
>
> Key: BEAM-6014
> URL: https://issues.apache.org/jira/browse/BEAM-6014
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Ahmet Altay
> Priority: Major
>
> Steps to reproduce:
>
> {noformat}
> <Make new virtual env, install apache-beam[gcp] >
>
> PROJECT=$(gcloud config get-value project)
> BQ_DATASET=${USER}_bq_dataset
> TABLE_NAME=out
> bq rm -rf --project=$PROJECT $BQ_DATASET
> bq mk --project=$PROJECT $BQ_DATASET
> python -m apache_beam.examples.cookbook.bigquery_tornadoes --output
> $BQ_DATASET.$TABLE_NAME --project=$PROJECT
> bq show $PROJECT:$BQ_DATASET.$TABLE_NAME
> {noformat}
> The resulting Database is empty, however it has several rows if we run the
> example with Dataflow runner.
> Older versions of SDK also seem to have this problem.
> cc: [~chamikara], [~charleschen].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
