HeartSaVioR commented on a change in pull request #27380: [SPARK-30669][SS]
Introduce AdmissionControl APIs for StructuredStreaming
URL: https://github.com/apache/spark/pull/27380#discussion_r373239763
##########
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/ReadLimit.java
##########
@@ -15,17 +15,28 @@
* limitations under the License.
*/
-package org.apache.spark.sql.execution.streaming.sources
+package org.apache.spark.sql.connector.read.streaming;
-import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, Offset}
+import org.apache.spark.annotation.Evolving;
-// A special `MicroBatchStream` that can get latestOffset with a start offset.
-trait RateControlMicroBatchStream extends MicroBatchStream {
+/**
+ * Interface representing limits on how much to read from a {@link
MicroBatchStream} when it
+ * implements {@link SupportsAdmissionControl}. There are several child
interfaces representing
+ * various kinds of limits.
+ *
+ * @see SupportsAdmissionControl#latestOffset(Offset, ReadLimit)
+ * @see ReadAllAvailable
+ * @see ReadMaxRows
+ */
+@Evolving
+public interface ReadLimit {
+ static ReadLimit maxRows(long rows) { return new ReadMaxRows(rows); }
- override def latestOffset(): Offset = {
- throw new IllegalAccessException(
- "latestOffset should not be called for RateControlMicroBatchReadSupport")
+ static ReadLimit maxFiles(int files) {
+ return new ReadMaxFiles(files);
}
- def latestOffset(start: Offset): Offset
+ static ReadLimit allAvailable() {
+ return ReadAllAvailable.SINGLETON;
Review comment:
nit: still have 4 spaces
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
