pandasanjay commented on PR #35197:
URL: https://github.com/apache/beam/pull/35197#issuecomment-3247805448
> hey @pandasanjay, is this PR still relevant?
Hi @ahmedabu98, Yes..
thanks for taking a look.
I explored two approaches internally:
1. Enrichment — This implementation runs into Dataflow worker instability
because it manages threads manually and doesn’t play well with streaming
parallel processing. I could use help here: do you know of any recommended
patterns or examples to safely handle concurrent enrichment in streaming DoFns
(or alternatives that avoid manual thread management)?
2. Custom transformers — This approach works reliably. I’ll add that
implementation.
I’m currently on annual leave and will push the changes when I return. If
you have suggestions while I’m away, please share and I’ll incorporate them
once I’m back. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
