Hi Luke, We can say Map 2 as a kind of a template using which you want to enrich data in Map 1. As I mentioned in my previous post, this is a high level scenario.
All these logic are spread across several classes (with ~4K lines of code in total). As in any Java application, 1. The code has been modularized with multiple method calls 2. Passing around HashMaps<CustomObject> as argument to each method 3. Accessing the attributes of the custom object using getters and setters. This is a common pattern in a normal Java application but I have not seen such an example of code in Beam. On Mon, Jun 22, 2020 at 8:23 AM Luke Cwik <[email protected]> wrote: > Who reads map 1? > Can it be stale? > > It is unclear what you are trying to do in parallel and why you wouldn't > stick all this logic into a single DoFn / stateful DoFn. > > On Sat, Jun 20, 2020 at 7:14 PM Praveen K Viswanathan < > [email protected]> wrote: > >> Hello Everyone, >> >> I am in the process of implementing an existing pipeline (written using >> Java and Kafka) in Apache Beam. The data from the source stream is >> contrived and had to go through several steps of enrichment using REST API >> calls and parsing of JSON data. The key >> transformation in the existing pipeline is in shown below (a super high >> level flow) >> >> *Method A* >> ----Calls *Method B* >> ----Creates *Map 1, Map 2* >> ----Calls *Method C* >> ----Read *Map 2* >> ----Create *Map 3* >> ----*Method C* >> ----Read *Map 3* and >> ----update *Map 1* >> >> The Map we use are multi-level maps and I am thinking of having >> PCollections for each Maps and pass them as side inputs in a DoFn wherever >> I have transformations that need two or more Maps. But there are certain >> tasks which I want to make sure that I am following right approach, for >> instance updating one of the side input maps inside a DoFn. >> >> These are my initial thoughts/questions and I would like to get some >> expert advice on how we typically design such an interleaved transformation >> in Apache Beam. Appreciate your valuable insights on this. >> >> -- >> Thanks, >> Praveen K Viswanathan >> > -- Thanks, Praveen K Viswanathan
