Hi Allison and devs, Although I was against this idea at first sight (probably because I'm a Scala dev), I think it could work as long as there are people who'd be interested in such an API. Were there any? I'm just curious. I've seen no emails requesting it.
I also doubt that Python devs would like to work on new data sources but support their wishes wholeheartedly :) Pozdrawiam, Jacek Laskowski ---- "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Fri, Jun 16, 2023 at 6:14 AM Allison Wang <allison.w...@databricks.com.invalid> wrote: > Hi everyone, > > I would like to start a discussion on “Python Data Source API”. > > This proposal aims to introduce a simple API in Python for Data Sources. > The idea is to enable Python developers to create data sources without > having to learn Scala or deal with the complexities of the current data > source APIs. The goal is to make a Python-based API that is simple and easy > to use, thus making Spark more accessible to the wider Python developer > community. This proposed approach is based on the recently introduced > Python user-defined table functions with extensions to support data sources. > > *SPIP Doc*: > https://docs.google.com/document/d/1oYrCKEKHzznljYfJO4kx5K_Npcgt1Slyfph3NEk7JRU/edit?usp=sharing > > *SPIP JIRA*: https://issues.apache.org/jira/browse/SPARK-44076 > > Looking forward to your feedback. > > Thanks, > Allison >
