[
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393843#comment-16393843
]
Charles Chen commented on BEAM-3824:
------------------------------------
It looks like this is not an issue with the new DirectRunner implementation–the
same error occurs when we use the old BundleBasedDirectRunner. It is possible
there is some other difference between 2.3.0 and now (maybe a dependency
issue?).
> BQ sink fails on Direct Runner for 2.4.0 RC2
> --------------------------------------------
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Reporter: Valentyn Tymofieiev
> Assignee: Chamikara Jayalath
> Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score
> --project=$PROJECT --dataset=beam_release_2_4_0
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish <DoOperation WriteTeamScoreSums/WriteToBigQuery
> output_tags=['out'],
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0,
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
> line 276, in <
> module>
> run()
> File
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
> line 270, in r
> un
> write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 389, in __exit__
> self.run().wait_until_finish()
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 369, in run
> self.to_runner_api(), self.runner, self._options).run(False)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 382, in run
> return self.runner.run_pipeline(self)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pip
> eline
> return runner.run_pipeline(pipeline)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 215, in ru
> n_pipeline
> return self.run_via_runner_api(pipeline.to_runner_api())
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 218, in ru
> n_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 837, in ru
> n_stages
> pcoll_buffers, safe_coders).process_bundle.metrics
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 938, in ru
> n_stage
> self._progress_frequency).process_bundle(data_input, data_output)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1110, in p
> rocess_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1003, in p
> ush
> response = self.worker.do_instruction(request)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 185, in do_instruc
> tion
> request.instruction_id)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 202, in process_bu
> ndle
> processor.process_bundle(instruction_id)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 298, in proc
> ess_bundle
> op.finish()
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
> line 389, in finish
> self.dofn_runner.finish()
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 517, in finish
> self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 508, in _invoke_bundle_method
> self._reraise_augmented(exn)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 537, in _reraise_augmented
> six.raise_from(new_exn, original_traceback)
> File
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/six.py",
> line 737, in raise_from
> raise value
> RuntimeError: Could not successfully insert rows to BigQuery table
> [google.com:clouddfe:beam_release_2_4_0.leader_board]. Errors: [<InsertEr
> rorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 0>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 1>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 2>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 3>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
