Actually, "UDF" might be the wrong terminology here - This is more of a "custom Python data source" than "Python user defined functions". (Although under the hood it can probably reuse lots of the UDF logic to execute the custom data source)
On Fri, Jun 3, 2022 at 2:49 PM Li Jin <ice.xell...@gmail.com> wrote: > What Yaron is going for is really something similar to custom data source > in Spark ( > https://levelup.gitconnected.com/easy-guide-to-create-a-custom-read-data-source-in-apache-spark-3-194afdc9627a) > that allows utilizing existing Python APIs that knows how to read data > source as a stream of record batches. > > >
