Hi Radek,
thanks for the design docs! The second one seems to require
authentication, can you open it, please?
Thanks,
Jan
On 5/6/25 20:59, Radek Stankiewicz via dev wrote:
hi all,
We’ve multiple projects in ideation, design or prototypes that share
the common problem - need to extend WindowedValue with additional
metadata.
Those projects are:
*
Drain mode - https://s.apache.org/beam-drain-mode
<https://s.apache.org/beam-drain-mode>
*
CDC metadata - https://s.apache.org/beam-cdc-metadata
<https://s.apache.org/beam-cdc-metadata>(super early prototype PR
<https://github.com/apache/beam/pull/34820>)
*
Open Telemetry integration - Open Telemetry PR discussion
<https://lists.apache.org/thread/hprbr1pcjfcg39sj9gz8tqmxj1zqt526>and
Open Telemetry PR <https://github.com/apache/beam/pull/34544>
Following those we’ve drafted a 1-pager proposal for extended element
metadata (https://s.apache.org/beam-element-extended-metadata
<https://s.apache.org/beam-element-extended-metadata>) and we seek
your opinion on it.
Extending a core item like this and adding features on top of it is
not a straightforward process. To make it easier we’ve drafted a
Capabilities negotiation framework
<https://docs.google.com/document/d/1Qwxrmi-EWrL5pbO2s3h9MKzJLtC5qEOJdrBvqft-vFc/edit?usp=sharing>to
document how existing runner_api protocols could be used to instruct
SDK when it is possible to use certain capabilities like extended
element metadata. Let me know what you think about it!
on behalf of the team,
Radek
