Re: [DESIGN] Beam Element Extended Metadata and future work

[Jan Lukavský]

Thanks.

The effort towards introducing "upserts" and "deletes" is really great! 
Actually I have recently sketched a PoC of a retract join [1]. It is 
currently entirely written on the SDK level, but it would greatly 
benefit from support in the model (e.g. "native" way of supporting the 
"RetractPCollection" [2]). The design doc is a great way to pull the 
community into designing this!

 Jan

[1] 
https://github.com/O2-Czech-Republic/proxima-platform/blob/master/beam/core/src/main/java/cz/o2/proxima/beam/core/transforms/retract/RetractJoin.java

[2] 
https://github.com/O2-Czech-Republic/proxima-platform/blob/master/beam/core/src/main/java/cz/o2/proxima/beam/core/transforms/retract/RetractPCollection.java

On 5/7/25 16:02, Kenneth Knowles wrote:
Ah, thanks for pointing it out. Fixed so anyone should be able to read 
and comment.

Kenn

On Wed, May 7, 2025 at 2:02 AM Jan Lukavský <je...@seznam.cz> wrote:

    Hi Radek,

    thanks for the design docs! The second one seems to require
    authentication, can you open it, please?

    Thanks,

     Jan

    On 5/6/25 20:59, Radek Stankiewicz via dev wrote:
    hi all,

    We’ve multiple projects in ideation, design or prototypes that
    share the common problem - need to extend WindowedValue with
    additional metadata.


    Those projects are:

     *

        Drain mode - https://s.apache.org/beam-drain-mode
        <https://s.apache.org/beam-drain-mode>

     *

        CDC metadata - https://s.apache.org/beam-cdc-metadata
        <https://s.apache.org/beam-cdc-metadata>(super early
        prototype PR <https://github.com/apache/beam/pull/34820>)

     *

        Open Telemetry integration - Open Telemetry PR discussion
        <https://lists.apache.org/thread/hprbr1pcjfcg39sj9gz8tqmxj1zqt526>and
        Open Telemetry PR <https://github.com/apache/beam/pull/34544>


    Following those we’ve drafted a 1-pager proposal for extended
    element metadata
    (https://s.apache.org/beam-element-extended-metadata
    <https://s.apache.org/beam-element-extended-metadata>) and we
    seek your opinion on it.


    Extending a core item like this and adding features on top of it
    is not a straightforward process. To make it easier we’ve drafted
    a Capabilities negotiation framework
    
<https://docs.google.com/document/d/1Qwxrmi-EWrL5pbO2s3h9MKzJLtC5qEOJdrBvqft-vFc/edit?usp=sharing>to
    document how existing runner_api protocols could be used to
    instruct SDK when it is possible to use certain capabilities like
    extended element metadata. Let me know what you think about it!

    on behalf of the team,
    Radek