lostluck commented on PR #36188: URL: https://github.com/apache/beam/pull/36188#issuecomment-3309536718
> What do you prefer to unify that? Alternatively, we can put bundlePanes into stateData. > > https://github.com/apache/beam/blob/06ed09aabe160dd21777822f41c885c075a86fc1/sdks/go/pkg/beam/runners/prism/internal/engine/data.go#L38-L44 > > . > Given we already had that nested structure in stageState > > https://github.com/apache/beam/blob/06ed09aabe160dd21777822f41c885c075a86fc1/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go#L1176 Thinking this through... A trigger fires, and we compute a pane, and we start a bundle for it. Bundle contains that key so we only have that bundle running with that key. A trigger fires again. The bundle can't start since it uses the key. It needs the "subsequent" pane for that window+key. But since this is post aggregation (Aggregations which is inherently stateful, but also never done SDK side), there's no real "stateful" restriction for the key+window on the output here, unless the post-GBK transform is stateful itself. The pane needs to be preserved downstream of these sequences since the GBK is implemented as an outside of EM transform, so we can propagate it down to the post GBK transform... But at no point, are we "forking" the pane lineage, since as a state, the pane is pretty moot. Your implementation's idea is correct then I think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
