Valentyn Tymofieiev created BEAM-3824:
-----------------------------------------
Summary: BQ sink fails on Direct Runner for 2.4.0 RC2
Key: BEAM-3824
URL: https://issues.apache.org/jira/browse/BEAM-3824
Project: Beam
Issue Type: Bug
Components: runner-direct
Reporter: Valentyn Tymofieiev
Assignee: Chamikara Jayalath
python -m apache_beam.examples.complete.game.hourly_team_score
--project=$PROJECT --dataset=beam_release_2_4_0
--input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
The pipeline fails with:
INFO:root:finish <DoOperation WriteTeamScoreSums/WriteToBigQuery
output_tags=['out'],
receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0,
coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]>
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
line 276, in <
module>
run()
File
"/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
line 270, in r
un
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 389, in __exit__
self.run().wait_until_finish()
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 382, in run
return self.runner.run_pipeline(self)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
line 129, in run_pip
eline
return runner.run_pipeline(pipeline)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 215, in ru
n_pipeline
return self.run_via_runner_api(pipeline.to_runner_api())
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 218, in ru
n_via_runner_api
return self.run_stages(*self.create_stages(pipeline_proto))
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 837, in ru
n_stages
pcoll_buffers, safe_coders).process_bundle.metrics
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 938, in ru
n_stage
self._progress_frequency).process_bundle(data_input, data_output)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 1110, in p
rocess_bundle
result_future = self._controller.control_handler.push(process_bundle)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
line 1003, in p
ush
response = self.worker.do_instruction(request)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 185, in do_instruc
tion
request.instruction_id)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 202, in process_bu
ndle
processor.process_bundle(instruction_id)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 298, in proc
ess_bundle
op.finish()
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
line 389, in finish
self.dofn_runner.finish()
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 517, in finish
self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 508, in _invoke_bundle_method
self._reraise_augmented(exn)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 537, in _reraise_augmented
six.raise_from(new_exn, original_traceback)
File
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/six.py",
line 737, in raise_from
raise value
RuntimeError: Could not successfully insert rows to BigQuery table
[google.com:clouddfe:beam_release_2_4_0.leader_board]. Errors: [<InsertEr
rorsValueListEntry
errors: [<ErrorProto
debugInfo: u''
location: u'processing_time'
message: u'no such field.'
reason: u'invalid'>]
index: 0>, <InsertErrorsValueListEntry
errors: [<ErrorProto
debugInfo: u''
location: u'processing_time'
message: u'no such field.'
reason: u'invalid'>]
index: 1>, <InsertErrorsValueListEntry
errors: [<ErrorProto
debugInfo: u''
location: u'processing_time'
message: u'no such field.'
reason: u'invalid'>]
index: 2>, <InsertErrorsValueListEntry
errors: [<ErrorProto
debugInfo: u''
location: u'processing_time'
message: u'no such field.'
reason: u'invalid'>]
index: 3>, <InsertErrorsValueListEntry
errors: [<ErrorProto
debugInfo: u''
location: u'processing_time'
message: u'no such field.'
reason: u'invalid'>]
...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
