Thank you for the follow up. If you think that presentation is useful, and this is not properly captured in docs, would you be kind enough to help us improve our docs? :)
It could be a link to that deck, and github issue, or new content in docs based on that presentation. On Sat, Aug 3, 2024 at 6:37 PM Jaehyeon Kim <dott...@gmail.com> wrote: > Hello, > > What I look into can actually be achieved by implementing one of the > caching strategies in a talk at Beam Summit 2022. > > - Strategies for caching data in Dataflow using Beam SDK > > <https://2022.beamsummit.org/sessions/strategies-for-caching-data-in-dataflow-using-beam-sdk/> > > Among the 4 options, I'd try a side input and the shared module > (with/without side input) first. > > Cheers, > Jaehyeon > > > On Thu, 1 Aug 2024 at 13:30, Jaehyeon Kim <dott...@gmail.com> wrote: > >> Thank you for letting me know. It is also available in the Python SDK - >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/util.py#L1656 >> >> However, it doesn't seem to meet the requirement that the side input >> values can change over time because the mentioned transform just seems to >> wait until the previous one gets completed. What I look into is, let's say, >> a customer attribute changes then order records should be enriched with the >> updated attribute. >> >> On Thu, 1 Aug 2024 at 13:14, LDesire <two_som...@icloud.com> wrote: >> >>> Hello. I found a similar example code. >>> You can use `Wait` PTransform. >>> Wait (Apache Beam 2.13.0) >>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>> beam.apache.org >>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>> [image: favicon.ico] >>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>> >>> Hope this helps. >>> >>> [image: stateful-beam-realtime.png] >>> >>> stateful-beam-realtime/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java >>> at 2cc16a9cf8460c5b0e4d749e81654273c14ffb00 · Jimmyst/stateful-beam-realtime >>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>> github.com >>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>> >>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>> >>> >>> 2024. 8. 1. 오전 11:52, Jaehyeon Kim <dott...@gmail.com> 작성: >>> >>> Hello, >>> >>> I'm looking into side input patterns especially slowly updating global >>> window side inputs - >>> https://beam.apache.org/documentation/patterns/side-inputs/ >>> >>> It'd be useful if we need to enrich eg) order records with customer >>> details where customer details would be taken as a side input. >>> >>> Let's say we have two Kafka topics, one for client records and the other >>> for order records. For the enrichment to work properly, consumption of >>> order records should wait until all customer records are read. >>> >>> Can you please inform me if it is achievable? >>> >>> Cheers, >>> Jaehyeon >>> >>> >>>
favicon.ico
Description: Binary data
