[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2
[ https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev updated BEAM-3824: -- Affects Version/s: (was: 2.4.0) > BQ sink fails on Direct Runner for 2.4.0 RC2 > > > Key: BEAM-3824 > URL: https://issues.apache.org/jira/browse/BEAM-3824 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Valentyn Tymofieiev >Assignee: David Cavazos >Priority: Blocker > Time Spent: 10m > Remaining Estimate: 0h > > python -m apache_beam.examples.complete.game.hourly_team_score > --project=$PROJECT --dataset=beam_release_2_4_0 > --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv > The pipeline fails with: > INFO:root:finish output_tags=['out'], > receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, > coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 276, in < > module> > run() > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 270, in r > un > write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 389, in __exit__ > self.run().wait_until_finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 369, in run > self.to_runner_api(), self.runner, self._options).run(False) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 382, in run > return self.runner.run_pipeline(self) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", > line 129, in run_pip > eline > return runner.run_pipeline(pipeline) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 215, in ru > n_pipeline > return self.run_via_runner_api(pipeline.to_runner_api()) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 218, in ru > n_via_runner_api > return self.run_stages(*self.create_stages(pipeline_proto)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 837, in ru > n_stages > pcoll_buffers, safe_coders).process_bundle.metrics > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 938, in ru > n_stage > self._progress_frequency).process_bundle(data_input, data_output) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1110, in p > rocess_bundle > result_future = self._controller.control_handler.push(process_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1003, in p > ush > response = self.worker.do_instruction(request) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 185, in do_instruc > tion > request.instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 202, in process_bu > ndle > processor.process_bundle(instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 298, in proc > ess_bundle > op.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py", > line 389, in finish > self.dofn_runner.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 517, in finish > self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 508, in _invoke_bundle_method > self._reraise_augmented(exn) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", >
[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2
[ https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev updated BEAM-3824: -- Priority: Minor (was: Blocker) > BQ sink fails on Direct Runner for 2.4.0 RC2 > > > Key: BEAM-3824 > URL: https://issues.apache.org/jira/browse/BEAM-3824 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Valentyn Tymofieiev >Assignee: David Cavazos >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > python -m apache_beam.examples.complete.game.hourly_team_score > --project=$PROJECT --dataset=beam_release_2_4_0 > --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv > The pipeline fails with: > INFO:root:finish output_tags=['out'], > receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, > coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 276, in < > module> > run() > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 270, in r > un > write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 389, in __exit__ > self.run().wait_until_finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 369, in run > self.to_runner_api(), self.runner, self._options).run(False) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 382, in run > return self.runner.run_pipeline(self) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", > line 129, in run_pip > eline > return runner.run_pipeline(pipeline) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 215, in ru > n_pipeline > return self.run_via_runner_api(pipeline.to_runner_api()) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 218, in ru > n_via_runner_api > return self.run_stages(*self.create_stages(pipeline_proto)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 837, in ru > n_stages > pcoll_buffers, safe_coders).process_bundle.metrics > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 938, in ru > n_stage > self._progress_frequency).process_bundle(data_input, data_output) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1110, in p > rocess_bundle > result_future = self._controller.control_handler.push(process_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1003, in p > ush > response = self.worker.do_instruction(request) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 185, in do_instruc > tion > request.instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 202, in process_bu > ndle > processor.process_bundle(instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 298, in proc > ess_bundle > op.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py", > line 389, in finish > self.dofn_runner.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 517, in finish > self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 508, in _invoke_bundle_method > self._reraise_augmented(exn) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line
[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2
[ https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev updated BEAM-3824: -- Affects Version/s: 2.4.0 > BQ sink fails on Direct Runner for 2.4.0 RC2 > > > Key: BEAM-3824 > URL: https://issues.apache.org/jira/browse/BEAM-3824 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.4.0 >Reporter: Valentyn Tymofieiev >Assignee: David Cavazos >Priority: Blocker > > python -m apache_beam.examples.complete.game.hourly_team_score > --project=$PROJECT --dataset=beam_release_2_4_0 > --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv > The pipeline fails with: > INFO:root:finish output_tags=['out'], > receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, > coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 276, in < > module> > run() > File > "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", > line 270, in r > un > write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 389, in __exit__ > self.run().wait_until_finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 369, in run > self.to_runner_api(), self.runner, self._options).run(False) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 382, in run > return self.runner.run_pipeline(self) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", > line 129, in run_pip > eline > return runner.run_pipeline(pipeline) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 215, in ru > n_pipeline > return self.run_via_runner_api(pipeline.to_runner_api()) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 218, in ru > n_via_runner_api > return self.run_stages(*self.create_stages(pipeline_proto)) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 837, in ru > n_stages > pcoll_buffers, safe_coders).process_bundle.metrics > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 938, in ru > n_stage > self._progress_frequency).process_bundle(data_input, data_output) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1110, in p > rocess_bundle > result_future = self._controller.control_handler.push(process_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", > line 1003, in p > ush > response = self.worker.do_instruction(request) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 185, in do_instruc > tion > request.instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 202, in process_bu > ndle > processor.process_bundle(instruction_id) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 298, in proc > ess_bundle > op.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py", > line 389, in finish > self.dofn_runner.finish() > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 517, in finish > self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 508, in _invoke_bundle_method > self._reraise_augmented(exn) > File > "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 537, in _reraise_augmented >
