[ https://issues.apache.org/jira/browse/BEAM-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on BEAM-7471 started by null. ---------------------------------- > DirectRunner GameStats fails, ValueError: PCollection of size 2 with more > than one element accessed as a singleton view. First two elements encountered > are "181.611538462", "9.1". [while running > 'CalculateSpammyUsers/ProcessAndFilter'] > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: BEAM-7471 > URL: https://issues.apache.org/jira/browse/BEAM-7471 > Project: Beam > Issue Type: Improvement > Components: runner-direct, runner-gearpump > Affects Versions: 2.13.0 > Reporter: Ankur Goenka > Priority: P3 > > > GameStats example is failing with following logs > {noformat} > ***************************************************** > * Running GameStats with DirectRunner > ***************************************************** > INFO:root:Missing pipeline option (runner). Executing pipeline using the > default runner: DirectRunner. > /usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137: > BeamDeprecationWarning: options is deprecated since First stable release. > References to <pipeline>.options will not be supported > method_to_use = self._compute_method(p, p.options) > INFO:root:Running pipeline with DirectRunner. > INFO:oauth2client.transport:Attempting refresh to obtain initial access_token > INFO:oauth2client.client:Refreshing access_token > INFO:root:Created table > google.com:clouddfe.goenka_python_validations.game_stats_teams with schema > <TableSchema > fields: [<TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'processing_time' > type: u'STRING'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'total_score' > type: u'INTEGER'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'window_start' > type: u'STRING'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'team' > type: u'STRING'>]>. Result: <Table > creationTime: 1559341252301 > etag: u'tEQeN0Sr4FvttBRZuXn7zw==' > id: u'google.com:clouddfe:goenka_python_validations.game_stats_teams' > kind: u'bigquery#table' > lastModifiedTime: 1559341252336 > location: u'US' > numBytes: 0 > numLongTermBytes: 0 > numRows: 0 > schema: <TableSchema > fields: [<TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'processing_time' > type: u'STRING'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'total_score' > type: u'INTEGER'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'window_start' > type: u'STRING'>, <TableFieldSchema > fields: [] > mode: u'NULLABLE' > name: u'team' > type: u'STRING'>]> > selfLink: > u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams' > tableReference: <TableReference > datasetId: u'goenka_python_validations' > projectId: u'google.com:clouddfe' > tableId: u'game_stats_teams'> > type: u'TABLE'>. > INFO:oauth2client.transport:Attempting refresh to obtain initial access_token > INFO:oauth2client.client:Refreshing access_token > ERROR:root:Exception at bundle > <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, > due to an exception. > Traceback (most recent call last): > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 343, in call > finish_state) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 380, in attempt_call > evaluator.process_element(value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", > line 633, in process_element > self.runner.process(element) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 753, in process > self._reraise_augmented(exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 807, in _reraise_augmented > raise_with_traceback(new_exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 751, in process > return self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 564, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 571, in _invoke_process_per_window > side_inputs = [si[window] for si in self.side_inputs] > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", > line 65, in __getitem__ > _FilteringIterable(self._iterable, target_window), self._view_options) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 346, in _from_runtime_iterable > return options['data'].view_fn(it) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 323, in <lambda> > lambda iterable: from_runtime_iterable(iterable, view_options)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 433, in _from_runtime_iterable > len(head), str(head[0]), str(head[1]))) > ValueError: PCollection of size 2 with more than one element accessed as a > singleton view. First two elements encountered are "181.611538462", "9.1". > [while running 'CalculateSpammyUsers/ProcessAndFilter'] > ERROR:root:Exception at bundle > <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, > due to an exception. > Traceback (most recent call last): > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 343, in call > finish_state) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 380, in attempt_call > evaluator.process_element(value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", > line 633, in process_element > self.runner.process(element) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 753, in process > self._reraise_augmented(exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 807, in _reraise_augmented > raise_with_traceback(new_exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 751, in process > return self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 564, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 571, in _invoke_process_per_window > side_inputs = [si[window] for si in self.side_inputs] > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", > line 65, in __getitem__ > _FilteringIterable(self._iterable, target_window), self._view_options) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 346, in _from_runtime_iterable > return options['data'].view_fn(it) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 323, in <lambda> > lambda iterable: from_runtime_iterable(iterable, view_options)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 433, in _from_runtime_iterable > len(head), str(head[0]), str(head[1]))) > ValueError: PCollection of size 2 with more than one element accessed as a > singleton view. First two elements encountered are "181.611538462", "9.1". > [while running 'CalculateSpammyUsers/ProcessAndFilter'] > ERROR:root:Exception at bundle > <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, > due to an exception. > Traceback (most recent call last): > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 343, in call > finish_state) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 380, in attempt_call > evaluator.process_element(value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", > line 633, in process_element > self.runner.process(element) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 753, in process > self._reraise_augmented(exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 807, in _reraise_augmented > raise_with_traceback(new_exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 751, in process > return self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 564, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 571, in _invoke_process_per_window > side_inputs = [si[window] for si in self.side_inputs] > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", > line 65, in __getitem__ > _FilteringIterable(self._iterable, target_window), self._view_options) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 346, in _from_runtime_iterable > return options['data'].view_fn(it) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 323, in <lambda> > lambda iterable: from_runtime_iterable(iterable, view_options)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 433, in _from_runtime_iterable > len(head), str(head[0]), str(head[1]))) > ValueError: PCollection of size 2 with more than one element accessed as a > singleton view. First two elements encountered are "181.611538462", "9.1". > [while running 'CalculateSpammyUsers/ProcessAndFilter'] > ERROR:root:Exception at bundle > <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, > due to an exception. > Traceback (most recent call last): > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 343, in call > finish_state) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 380, in attempt_call > evaluator.process_element(value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", > line 633, in process_element > self.runner.process(element) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 753, in process > self._reraise_augmented(exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 807, in _reraise_augmented > raise_with_traceback(new_exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 751, in process > return self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 564, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 571, in _invoke_process_per_window > side_inputs = [si[window] for si in self.side_inputs] > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", > line 65, in __getitem__ > _FilteringIterable(self._iterable, target_window), self._view_options) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 346, in _from_runtime_iterable > return options['data'].view_fn(it) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 323, in <lambda> > lambda iterable: from_runtime_iterable(iterable, view_options)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 433, in _from_runtime_iterable > len(head), str(head[0]), str(head[1]))) > ValueError: PCollection of size 2 with more than one element accessed as a > singleton view. First two elements encountered are "181.611538462", "9.1". > [while running 'CalculateSpammyUsers/ProcessAndFilter'] > ERROR:root:Giving up after 4 attempts. > WARNING:root:A task failed with exception: PCollection of size 2 with more > than one element accessed as a singleton view. First two elements encountered > are "181.611538462", "9.1". [while running > 'CalculateSpammyUsers/ProcessAndFilter'] > INFO:oauth2client.transport:Attempting refresh to obtain initial access_token > INFO:oauth2client.client:Refreshing access_token > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", > line 399, in <module> > run() > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", > line 393, in run > }, options.view_as(GoogleCloudOptions).project)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 426, in __exit__ > self.run().wait_until_finish() > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", > line 430, in wait_until_finish > self._executor.await_completion() > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 400, in await_completion > self._executor.await_completion() > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 446, in await_completion > raise_(t, v, tb) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 343, in call > finish_state) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", > line 380, in attempt_call > evaluator.process_element(value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", > line 633, in process_element > self.runner.process(element) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 753, in process > self._reraise_augmented(exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 807, in _reraise_augmented > raise_with_traceback(new_exn) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 751, in process > return self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 564, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", > line 571, in _invoke_process_per_window > side_inputs = [si[window] for si in self.side_inputs] > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", > line 65, in __getitem__ > _FilteringIterable(self._iterable, target_window), self._view_options) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 346, in _from_runtime_iterable > return options['data'].view_fn(it) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 323, in <lambda> > lambda iterable: from_runtime_iterable(iterable, view_options)) > File > "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", > line 433, in _from_runtime_iterable > len(head), str(head[0]), str(head[1]))) > ValueError: PCollection of size 2 with more than one element accessed as a > singleton view. First two elements encountered are "181.611538462", "9.1". > [while running 'CalculateSpammyUsers/ProcessAndFilter'] > {noformat} -- This message was sent by Atlassian Jira (v8.20.7#820007)