sjvanrossum commented on issue #21089: URL: https://github.com/apache/beam/issues/21089#issuecomment-1374882616
@nivaldoh As promised on the dev thread I've just opened a PR at https://github.com/nivaldoh/beam/pull/20 with some worker code changes as well as a container and boot script based on the existing SDK containers. I assumed that Rust pipelines would typically be statically compiled like Go pipelines, so the boot script only looks for a single artifact file at the moment. The binaries must match between the launcher and worker if we were to use serde_traitobject to serialize the DoFns, I've got some additional changes coming up to provide some scaffolding for that. The user binary needs to be able to switch between pipeline construction and pipeline execution mode, so there's an init function much like the Go SDK requires to run soon after the binary is started. That init function needs to be in a different place, but that would require restructuring the crates a bit I think. Happy to sync on that at some point, I think most of the framework code could live in an apache-beam crate and optional features could live in separate crates e.g. apache-beam-io-gcp/aws/azure. The worker code I had started on uses a concurrent cache, such that we don't need to lock on the worker to interact with the caches and such that we can expire entries in the cache like the Java SDK does. Looking forward to continue working on this with you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
