Hi all, At the moment, we have a BigQuery native source for Python SDK, which can be used only by Dataflow runner. Consequently, it doesn't work on portable runners, such as Flink.
Recently I have written a prototypical source which implements iobase.BoundedSource, so that other runners can read from BigQuery as well. It works the same way as in Java SDK [1], which means that it exports BigQuery table to JSON and returns TextSource objects in the split() call. However, it has the following problems: - it doesn't work on Direct runner, - its API is highly experimental. This is where my question begins. What should we do in order to provide support for reading from BigQuery on other runners than Dataflow? Do you think it's fine to continue working on the source I described? Or maybe it should be done in an entirely different way (not by exporting tables to JSON)? Thanks, Kamil [1] https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java