HeartSaVioR commented on a change in pull request #27380: [SPARK-30669][SS] Introduce AdmissionControl APIs for StructuredStreaming URL: https://github.com/apache/spark/pull/27380#discussion_r373239963
########## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/SupportsAdmissionControl.java ########## @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector.read.streaming; + +import org.apache.spark.annotation.Evolving; + +/** + * A mix-in interface for {@link SparkDataStream} streaming sources to signal that they can control + * the rate of data ingested into the system. These rate limits can come implicitly from the + * contract of triggers, e.g. Trigger.Once() requires that a micro-batch process all data + * available to the system at the start of the micro-batch. Alternatively, sources can decide to + * limit ingest through data source options. + * + * Through this interface, a MicroBatchStream should be able to return the next offset that it will + * process until given a {@link ReadLimit}. + * + * @since 3.0.0 + */ +@Evolving +public interface SupportsAdmissionControl extends SparkDataStream { + + /** + * Returns the read limits potentially passed to the data source through options when creating + * the data source. + */ + default ReadLimit getDefaultReadLimit() { + return ReadLimit.allAvailable(); Review comment: nit: still have 4 spaces ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
