Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208425737
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ *
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208425199
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ *
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208423440
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208392865
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ *
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208391449
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ *
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208390359
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208390264
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208389947
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208384141
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -80,17 +80,17 @@ object
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208383579
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExec.scala
---
@@ -93,21 +81,17 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208383098
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExec.scala
---
@@ -39,52 +36,43 @@ case class
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208380370
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceRDD.scala
---
@@ -51,18 +58,19 @@ class DataSourceRDD[T:
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208380139
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceRDD.scala
---
@@ -51,18 +58,19 @@ class DataSourceRDD[T:
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208373784
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208373424
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ *
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208373089
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208372469
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ *
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208372512
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/MicroBatchReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208371927
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208370532
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208370493
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ *
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208370391
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208368798
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReadSupport.java
---
@@ -0,0 +1,72 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208348226
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.java
---
@@ -20,18 +20,18 @@
import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208347697
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownRequiredColumns.java
---
@@ -21,22 +21,25 @@
import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208345467
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ScanConfig.java
---
@@ -18,22 +18,16 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208344984
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ScanConfig.java
---
@@ -18,22 +18,16 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208344510
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ScanConfig.java
---
@@ -18,22 +18,16 @@
package
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208343665
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadSupport.java
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208342404
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/PartitionReaderFactory.java
---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208341897
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/PartitionReaderFactory.java
---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208340913
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/BatchReadSupport.java
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208340329
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/BatchReadSupport.java
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208338263
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/StreamingWriteSupportProvider.java
---
@@ -29,24 +28,24 @@
* provide data writing
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208337966
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/BatchWriteSupportProvider.java
---
@@ -21,32 +21,32 @@
import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208337773
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/StreamingWriteSupportProvider.java
---
@@ -29,24 +28,24 @@
* provide data writing
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208337008
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/ContinuousReadSupportProvider.java
---
@@ -39,7 +52,7 @@
* @param options the
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208337291
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/MicroBatchReadSupportProvider.java
---
@@ -45,8 +51,8 @@
* @param options the
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208336624
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/MicroBatchReadSupportProvider.java
---
@@ -20,23 +20,29 @@
import java.util.Optional;
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208336499
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2.java ---
@@ -23,8 +23,9 @@
* The base interface for data source v2.
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208335865
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/ContinuousReadSupportProvider.java
---
@@ -20,17 +20,30 @@
import java.util.Optional;
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208335523
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/ContinuousReadSupportProvider.java
---
@@ -20,17 +20,30 @@
import java.util.Optional;
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208334273
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/BatchReadSupportProvider.java
---
@@ -19,18 +19,18 @@
import
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208333413
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/ContinuousReadSupportProvider.java
---
@@ -20,17 +20,30 @@
import java.util.Optional;
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208333129
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/BatchWriteSupportProvider.java
---
@@ -21,32 +21,32 @@
import
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208317741
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -270,11 +269,11 @@ private[kafka010]
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208180976
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadSupport.java
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208165977
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetsOnlyScanConfigBuilder.scala
---
@@ -0,0 +1,30 @@
+/*
+ *
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208133162
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadSupport.java
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208131160
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetsOnlyScanConfigBuilder.scala
---
@@ -0,0 +1,30 @@
+/*
+ *
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208114465
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownRequiredColumns.java
---
@@ -21,22 +21,25 @@
import
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208098178
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memoryV2.scala
---
@@ -132,35 +134,15 @@ class
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r208095752
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/sources/v2/JavaBatchDataSourceV2.java
---
@@ -1,114 +0,0 @@
-/*
- * Licensed to the
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r207978414
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -122,24 +119,22 @@ case class MemoryStream[A :
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r207976440
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -122,24 +119,22 @@ case class MemoryStream[A :
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/22009
[SPARK-24882][SQL] improve data source v2 API
## What changes were proposed in this pull request?
Improve the data source v2 API according to the [design
101 - 156 of 156 matches
Mail list logo
