[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20286


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/20286#discussion_r161976455
  
--- Diff: 
sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/Offset.java
 ---
@@ -17,12 +17,20 @@
 
 package org.apache.spark.sql.sources.v2.streaming.reader;
 
+import org.apache.spark.annotation.InterfaceStability;
+
 /**
- * An abstract representation of progress through a [[MicroBatchReader]] 
or [[ContinuousReader]].
- * During execution, Offsets provided by the data source implementation 
will be logged and used as
- * restart checkpoints. Sources should provide an Offset implementation 
which they can use to
- * reconstruct the stream position where the offset was taken.
+ * An abstract representation of progress through a {@link 
MicroBatchReader} or
+ * {@link ContinuousReader}.
+ * During execution, offsets provided by the data source implementation 
will be logged and used as
+ * restart checkpoints. Each source should provide an offset 
implementation which the source can use
+ * to reconstruct a position in the stream up to which data has been 
seen/processed.
+ *
+ * Note: This class currently extends {@link 
org.apache.spark.sql.execution.streaming.Offset} to
+ * maintain compatibility with DataSource V1 APIs. This will be extension 
will be removed once we
--- End diff --

ditto


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/20286#discussion_r161976313
  
--- Diff: 
sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/MicroBatchReader.java
 ---
@@ -25,7 +26,11 @@
 /**
  * A mix-in interface for {@link DataSourceV2Reader}. Data source readers 
can implement this
  * interface to indicate they allow micro-batch streaming reads.
+ *
+ * Note: This class currently extends {@link BaseStreamingSource} to 
maintain compatibility with
+ * DataSource V1 APIs. This will be extension will be removed once we get 
rid of V1 completely.
--- End diff --

ditto


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/20286#discussion_r161976371
  
--- Diff: 
sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/ContinuousReader.java
 ---
@@ -27,11 +28,15 @@
  * interface to allow reading in a continuous processing mode stream.
  *
  * Implementations must ensure each read task output is a {@link 
ContinuousDataReader}.
+ *
+ * Note: This class currently extends {@link BaseStreamingSource} to 
maintain compatibility with
+ * DataSource V1 APIs. This will be extension will be removed once we get 
rid of V1 completely.
--- End diff --

nit: This ~~will be~~ extension


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...

[tdas]

GitHub user tdas opened a pull request:

https://github.com/apache/spark/pull/20286

[SPARK-23119][SS] Minor fixes to V2 streaming APIs

## What changes were proposed in this pull request?

- Added `@InterfaceStability.Evolving` annotations
- Improved docs.

## How was this patch tested?
Existing tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdas/spark SPARK-23119

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20286


commit f6dfa58e2bc47f447c02d54a834e164e70083db9
Author: Tathagata Das 
Date:   2018-01-17T01:59:25Z

Minor fixes to V2 APIs




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org