Re: State of datasource api v2

2019-01-14 Thread Arnaud LARROQUE
Hi Vladimir, I've try to do the same here when I attempted to write a Spark connector for remote file. >From my point of view, There was a lot of change in the V2 API => Better semantic at least ! I understood that only continuous streaming use datasourceV2 (Not sure if im correct). But for file

State of datasource api v2

2019-01-14 Thread Vladimir Prus
Hi, I am trying to understand the state of datasource v2, and I'm a bit lost. On one hand, it is supposed to be more flexible approach, as described for example here: https://www.slideshare.net/databricks/apache-spark-data-source -v2-with-wenchen-fan-and-gengliang-wang On another hand, it ap