nathancCG opened a new issue, #25014:
URL: https://github.com/apache/beam/issues/25014
Hi Team,
We recently went into this error: `TypeError: Cannot convert GlobalWindow to
apache_beam.utils.windowed_value._IntervalWindowBase`
Full error log:
`Error message from worker: generic::unknown: Traceback (most recent call
last):
File "apache_beam/runners/common.py", line 1458, in
apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in
apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in
apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1747, in
apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 237, in
apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 239, in
apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 198, in
apache_beam.runners.worker.operations.ConsumerSet.update_counters_start
File "apache_beam/runners/worker/opcounters.py", line 215, in
apache_beam.runners.worker.opcounters.OperationCounters.update_from
File "apache_beam/runners/worker/opcounters.py", line 267, in
apache_beam.runners.worker.opcounters.OperationCounters.do_sample
File "apache_beam/coders/coder_impl.py", line 1472, in
apache_beam.coders.coder_impl.WindowedValueCoderImpl.get_estimated_size_and_observables
File "apache_beam/coders/coder_impl.py", line 1490, in
apache_beam.coders.coder_impl.WindowedValueCoderImpl.get_estimated_size_and_observables
File "apache_beam/coders/coder_impl.py", line 1222, in
apache_beam.coders.coder_impl.SequenceCoderImpl.estimate_size
File "apache_beam/coders/coder_impl.py", line 1238, in
apache_beam.coders.coder_impl.SequenceCoderImpl.get_estimated_size_and_observables
File "apache_beam/coders/coder_impl.py", line 208, in
apache_beam.coders.coder_impl.CoderImpl.get_estimated_size_and_observables
File "apache_beam/coders/coder_impl.py", line 829, in
apache_beam.coders.coder_impl.IntervalWindowCoderImpl.estimate_size
TypeError: Cannot convert GlobalWindow to
apache_beam.utils.windowed_value._IntervalWindowBase`
our pipeline looks something like this:
` pipeline
| "Read PubSub Messages"
>> beam.io.ReadFromPubSub(
subscription=options.input_subscription,
with_attributes=False
)
| f"Group Messages into {options.window_size} seconds window"
>> beam.WindowInto(
window.FixedWindows(options.window_size),
allowed_lateness=Duration(seconds=1 * 60 * 60),
) # 1 hour
| "Extract data from requests" >>
beam.ParDo(ExtractDataFromRequest()
`
This pipeline has been running successfully for a couple months and these
errors only shown up recently and it seems nowhere to start debugging with. Any
ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
