Frederick Reiss commented on SPARK-10815:

I'm confused by the current description of this task.  As far as I can see, the 
interface for sources and sinks in Structured Streaming has no direct 
dependencies on RDDs. More precisely, the four traits (Source, Sink, 
StreamSourceProvider, and StreamSinkProvider) that directly comprise the 
interface do not depend on the {{RDD}} class. The DataStreamWriter and 
DataFrameWriter class that currently insulate users from Source, Sink, etc. 
also do not have any dependencies on {{RDD}}.

Is this issue intended perhaps to reference limitations of the implementations 
of Datasets that require unnecessary direct access to the Dataset's internal 

> API design: data sources and sinks
> ----------------------------------
>                 Key: SPARK-10815
>                 URL: https://issues.apache.org/jira/browse/SPARK-10815
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL, Streaming
>            Reporter: Reynold Xin
> The existing (in 2.0) source/sink interface for structured streaming depends 
> on RDDs. This dependency has two issues:
> 1. The RDD interface is wide and difficult to stabilize across versions. This 
> is similar to point 1 in https://issues.apache.org/jira/browse/SPARK-15689. 
> Ideally, a source/sink implementation created for Spark 2.x should work in 
> Spark 10.x, assuming the JVM is still around.
> 2. It is difficult to swap in/out a different execution engine.
> The purpose of this ticket is to create a stable interface that addresses the 
> above two.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to