Beam IO Connector

2023-08-12 Thread Jeremy Bloom
Hello- I am fairly new to Beam but have been working with Apache Spark for a number of years. The application I am developing uses a data pipeline to ingest JSON with a particular schema, uses it to prepare data for a service that I do not control (a mathematical optimization solver), runs the

Re: Beam IO Connector

2023-08-12 Thread Bruno Volpato via dev
Hi Jeremy, Apparently you are trying to use Beam's DirectRunner , which is mostly focused on small pipelines / testing purposes. Even if it runs in the JVM, there are protections in place to make sure your pipeline will be able to be