pkuzmickas commented on PR #39867: URL: https://github.com/apache/beam/pull/39867#issuecomment-5412513471
Thank you for the review! > If it's Flink 2.x only, revert the changes on common runners/flink/src/ Removed :+1: > What is the consideration of exclude portable runner? It would be beneficial to have classic/portable runner feature in sync, as the latter one is the only one available for Python and Go SDKs We have only thoroughly tested this with Java production workflows and would like to keep it tightly scoped. Portable side inputs also go through the Fn State API and SDK harness instead of SideInputReader, so this cache doesn't apply directly and seems like a bigger piece of work due to the different SDK harnesses. > Similarly what is the consideration to disable streaming? Due to the nature of side inputs in unbounded streaming pipelines, the same window can be updated later. Since the cache is shared across the task manager, I'm worried some subtasks could use stale data for a bit. In batch processing side inputs should only be written once before they're read, so this works well. But I am open to removing it if you think opt-in is enough here. Wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
