Hi, I'm a bit confused about the current state and the future plans of custom data sources in Structured Streaming.
So for DStreams we could write a Receiver as documented. Can this be used with Structured Streaming? Then we had the DataSource API with DefaultSource et. al. which was (in my opinion) never properly documented. With Spark 2.3 we got a new DataSourceV2 (which also was a marker interface), also not properly documented. Now with Spark 3 this seems to change again? ( https://issues.apache.org/jira/browse/SPARK-25390), at least the DataSourceV2 interface is gone, still no documentation but still called v2 somehow? Can anyone shed some light on the current state of data sources & sinks for batch & streaming in Spark 2.4 and 3.x? Thank you! Cheers, Lars
