That's good news about the direct runner, thanks! On Thu, Dec 29, 2022 at 2:02 PM Robert Bradshaw <rober...@google.com> wrote:
> On Thu, Jul 28, 2022 at 5:37 PM Chamikara Jayalath via dev > <dev@beam.apache.org> wrote: > > > > On Thu, Jul 28, 2022 at 4:51 PM Lina Mårtensson <lina@camus.energy> > wrote: > >> > >> Thanks for the detailed answers! > >> > >> I totally get the points about development & maintenance cost, and, > >> from a user perspective, about getting the performance right. > >> > >> I decided to try out the Spanner connector to get a sense of how well > >> the x-language approach works in our world, since that's an existing > >> x-language connector. > >> Overall, it works and with minimal intervention as you say - it is > >> very slow, though. > >> I'm a little confused about "portable runners" - if I understand this > >> correctly, this means we couldn't run with the DirectRunner anymore if > >> using an x-language connector? (At least it didn't work when I tried > >> it.) > > > > > > You'll have to use the portable DirectRunner - > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/portability > > > > Job service for this can be started using following command: > > python apache_beam/runners/portability/local_job_service_main.py -p > <port> > > Note that the Python direct runner is already a portable runner, so > you shouldn't have to do anything special (like start up a separate > job service and pass extra options) to run locally. Just use the > cross-language transforms as you would any normal Python transform. > > The goal is to make this as smooth and transparent as possible; please > keep coming back to us if you find rough edges. >
