stankiewicz commented on PR #37232: URL: https://github.com/apache/beam/pull/37232#issuecomment-3723110416
hi Suvrak, purpose of the issue was to track changes around https://github.com/apache/beam/blob/v2.70.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java#L135 where we have several context items in ElementMetadata that has to land in WindowedValue. ElementMetadata is used as transport structure between stages. offset and record_id id you've added are not part of it, they are not read from windmill so this change is wrong. In 2.71/master todo was removed from the code (https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java) and you can see drain being propagated from ElementMetadata. In the future we will be adding new context pieces like tracing context and cdc info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
