pritamdodeja commented on issue #35867:
URL: https://github.com/apache/beam/issues/35867#issuecomment-3341737811
@shunping I'm able to produce this error when I increase the number of
records generated. Here is the behavior when ```NUMBER_OF_ROWS``` is set to
100_000
```
(venv) pdodeja@distml:prismrunner$ python debug_prism.py
Generating data for 'random_data.csv' from 2016-01-01 00:00:00 to 2016-06-01
00:00:00 (max 100000 rows)...
Data generation complete. Wrote 87437 rows.
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey5.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe0ab8b0>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey10.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe0abc70>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey15.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe17b760>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey20.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe0a9de0>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey25.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe17b610>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey30.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfdfdbd30>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey35.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfdfdb6d0>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey40.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe179030>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey45.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe007a00>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey50.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe007a60>, DefaultTrigger(), 1, 1, None)
WARNING:apache_beam.transforms.core:All input pcollections must have the
same window. Windowing for flatten set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe356ef0>, DefaultTrigger(), 1, 1, None), windowing of pcoll
PCollection[ResampleAndKey55.None] set to
Windowing(<apache_beam.transforms.window.FixedWindows object at
0x7fbcfe006b00>, DefaultTrigger(), 1, 1, None)
Exception in thread wait_until_finish_read:
Traceback (most recent call last):
File "/home/pdodeja/.pyenv/versions/3.10.14/lib/python3.10/threading.py",
line 1017, in _bootstrap_inner
Traceback (most recent call last):
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/debug_prism.py",
line 556, in <module>
with beam.Pipeline(options=opts) as p:
self.run()
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/pipeline.py",
line 663, in __exit__
File "/home/pdodeja/.pyenv/versions/3.10.14/lib/python3.10/threading.py",
line 954, in run
self.result.wait_until_finish()
self._target(*self._args, **self._kwargs)
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/portability/portable_runner.py",
line 568, in wait_until_finish
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/portability/portable_runner.py",
line 533, in read_messages
for message in self._message_stream:
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/grpc/_channel.py",
line 543, in __next__
raise self._runtime_exception
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/portability/portable_runner.py",
line 574, in _observe_state
return self._next()
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/grpc/_channel.py",
line 969, in _next
for state_response in self._state_stream:
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/grpc/_channel.py",
line 543, in __next__
return self._next()
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/grpc/_channel.py",
line 969, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC
that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Error received from peer
{created_time:"2025-09-27T09:32:15.774288033-04:00", grpc_status:4,
grpc_message:"Deadline Exceeded"}"
>
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC
that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Error received from peer
{created_time:"2025-09-27T09:32:15.773856543-04:00", grpc_status:4,
grpc_message:"Deadline Exceeded"}"
>
Exception in thread
run_worker_job-001[job]_ref_Environment_default_environment_1:
Traceback (most recent call last):
File "/home/pdodeja/.pyenv/versions/3.10.14/lib/python3.10/threading.py",
line 1017, in _bootstrap_inner
self.run()
File "/home/pdodeja/.pyenv/versions/3.10.14/lib/python3.10/threading.py",
line 954, in run
self._target(*self._args, **self._kwargs)
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 283, in run
getattr(self, SdkHarness.REQUEST_METHOD_PREFIX + request_type)(
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 343, in _request_process_bundle_progress
self._request_process_bundle_action(request)
File
"/home/pdodeja/programming/repositories/debugging/prismrunner/venv/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 352, in _request_process_bundle_action
self._report_progress_executor.submit(task)
File
"/home/pdodeja/.pyenv/versions/3.10.14/lib/python3.10/concurrent/futures/thread.py",
line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
```
The debugging output is quite long, let me know how I can get that to you in
case you need it. I'm running on Fedora 42 now with pretty recent versions of
everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
