lostluck opened a new issue, #24789: URL: https://github.com/apache/beam/issues/24789
### What would you like to happen? This is the tracking issue for changes related to adding a Portable local runner in Go to simplify SDK and User side testing. Goals: * Be able to test and configure FnAPI surface. The implementations of FnAPI surfaces are difficult to test without a portable runner, but how each production runner (like Dataflow, or Flink) uses the FnAPI vary from instance to instance. While the gold standard is running on every runner, that requires arbitrarily complex set up for each. * Validates Beam feature correctness across different configurations or variants in feature support. * Replace the Go Direct runner. Non-Goals: * Performance. The runner will by default be in a test operation mode and be biased towards executing single transforms at a time, within memory. This ensures coder correctness. That said, the configurability enables alternative operation modes that would improve the performance and reliability, such as fusion, bundle retries, multi-processing, or writing and reading large datasets to files between stages. * Distributed processing. It's intended to be a local runner, and so won't operate distinct workers on separate machines, though it may spawn multiple SDK containers as needed. The Beam Model doesn't require distributed processing. Again, there's no reason this couldn't be added later, but not at the expense of the primary correctness and testing aims. Initial prototype, which requires operation in loopback mode is being iterated on in a separate repo, but is being cleaned up for contribution to Beam. https://github.com/lostluck/experimental/tree/master/local ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [X] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
