lostluck commented on issue #28959: URL: https://github.com/apache/beam/issues/28959#issuecomment-1769002010
I'm 50/50 on if that's the right behavior. SDFs get split up into several execution components, so if setup is expensive, it could be called once on a node that never actually needs those parts setup. This could be worked around by moving the necessary client setup code into a sync.Once guarded function, which is called before it's needed. But if we do feel it is a bug with Beam Go SDFs: Setup would be called like here for normal ParDos: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/pardo.go#L105 That would be fixed here in exec/sdf.go for Creating Restrictions. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L52C35-L52C35 But also here for Splitting and sizing restrictions: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L141 Here for truncations: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L268 It is already being called via the up call for ProcessSizedElementsAndRestrictions, but I'm mentioning it for completeness: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L460 And here for the fallback when SDFs are executing without sub element splits: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sdf.go#L994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
