This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new c165bce7e201 [SPARK-60732][SQL] Support admission control for 
streaming reads of native data sources
c165bce7e201 is described below

commit c165bce7e20196f7686e5bbab5dca67ebb64b307
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:46:20 2026 +0900

    [SPARK-60732][SQL] Support admission control for streaming reads of native 
data sources
    
    ### What changes were proposed in this pull request?
    
    Admission control for micro-batch streaming reads of native data sources, 
appended within version 1 of `spark_datasource.h`. The guide of SPARK-60101 
(#59424) lists it as a limitation. This PR is based on #59931 (SPARK-60723), 
whose commits it includes until that PR is merged.
    
    - **Capability and default limit.** A data source with the new capability 
`SPARK_DS_ADMISSION_CONTROL` returns the limit on the data of each micro-batch 
with `default_read_limit`, appended to `SparkDsLibrary`, in a size-prefixed 
`SparkDsReadLimit`: `SPARK_DS_READ_MAX_ROWS` with a maximum number of rows, or 
`SPARK_DS_READ_ALL_AVAILABLE`, typically from an option of the data source. 
Spark asks once, when the query starts. It fails the query for an unknown kind 
of limit, a maximum below  [...]
    - **Limited latest offsets.** `SparkDsLatestOffsetInfo` gets the limit as 
`const SparkDsReadLimit* limit`, the same struct that `default_read_limit` 
fills, so the limit has a single layout: the latest offset is then the end of 
the data of the next micro-batch, within the limit. Spark passes the default 
limit, or a limit of the kind `SPARK_DS_READ_ALL_AVAILABLE` for `Trigger.Once`, 
for `Trigger.AvailableNow` when Spark falls back to a single micro-batch, and 
for the other sources. When [...]
    - **`Trigger.AvailableNow`.** Spark first asks for the latest offset 
without a limit, as before, and then passes it as `run_end` with each limited 
request, so the library never returns an offset past it. The run then reads the 
data that was available when it started, in micro-batches within the limit. 
Spark only compares offsets for equality, as strings, to find new data, and 
does not order them, since they are defined by the library. A query that 
restarts starts a new run, with a new [...]
    - **Progress.** With a limit, the progress of the query reports the latest 
offset without the limit, as the Kafka source does. So with a limit, Spark 
calls `latest_offset` twice for some micro-batches, and the header says which 
calls: with `Trigger.AvailableNow`, the call without a limit before the first 
micro-batch is the end of the run, and with the other triggers, the call with 
`SPARK_DS_READ_ALL_AVAILABLE` before each limited one is only for the progress. 
Spark only asks with the  [...]
    
    On the Spark side, `ColumnarStreamReader` gets `defaultReadLimit()` and the 
limit and the end of the run (`runEnd`) in `latestOffset`, which 
`ColumnarMicroBatchStream` maps from `getDefaultReadLimit` and 
`latestOffset(start, limit)`. The header, the FFM bindings, the C++ test 
library and the guide are updated: admission control leaves the Limitations of 
the guide, and the capability table lists it.
    
    ### Why are the changes needed?
    
    Without a limit, every micro-batch reads all the data that is available, 
which can make the first micro-batch of a large source, or a backlog after a 
restart, too large. Spark's own sources offer such limits, for example 
`maxOffsetsPerTrigger` for Kafka and `maxFilesPerTrigger` for files.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for released versions: native data sources are new in the unreleased 
4.4.0. Data sources without the new capability, and libraries built against the 
header before these additions, keep reading all the available data.
    
    ### How was this patch tested?
    
    New tests in `NativeDataSourceSuite`:
    - "limit the rows of each micro-batch with admission control" checks 
micro-batches of 5, 5 and 2 rows, and the latest offset without the limit that 
the first one reports. It also checks `Trigger.AvailableNow` within the limit 
and up to the end of the run, `Trigger.Once` reading all the rows, and a single 
micro-batch without the capability, or with a default read limit of all the 
available rows. The test library records the start, the limit and the end of 
the run of each `latest_offset [...]
    - "start a new run of Trigger.AvailableNow with a limit when the query 
restarts" fails the second micro-batch of a run, and checks that the restarted 
query runs it again up to its logged end without asking for a latest offset, 
then reads up to a new end within the limit.
    - "reject the read limits that a micro-batch cannot have" covers an unknown 
kind of limit and a maximum of 0 rows, and checks that the stream reader is 
closed when the query fails. It also uses a library whose table ends before 
`default_read_limit`, as one built against the earlier header, that declares 
the capability.
    
    `NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass locally, with 
Java 25 for the tests. The C++ example of the guide builds against the new 
header.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59942 from HyukjinKwon/native-ds-admission-control.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 docs/sql-native-data-sources.md                    |  49 ++++++--
 .../datasources/v2/ffi/NativeLibrary.java          |  63 +++++++++-
 .../apache/spark/sql/datasource/spark_datasource.h | 114 ++++++++++++++---
 .../v2/columnar/ColumnarDataSource.scala           |  16 ++-
 .../datasources/v2/columnar/ColumnarTable.scala    |  51 +++++---
 .../datasources/v2/ffi/NativeDataSource.scala      |  61 +++++++--
 .../native-datasource/test_native_datasource.cc    |  70 +++++++++--
 .../v2/columnar/ColumnarDataSourceSuite.scala      |   2 +-
 .../datasources/v2/ffi/NativeDataSourceSuite.scala | 138 ++++++++++++++++++++-
 9 files changed, 489 insertions(+), 75 deletions(-)

diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index 049f49902e7c..9dbbfae59ea6 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -128,14 +128,37 @@ Spark calls the functions of the table as follows:
   the checkpoint of the source in a `SparkDsStreamReaderInfo`. When the 
checkpoint does not have the
   initial offset of the source yet, Spark asks for it with `initial_offset`, 
and keeps it in the
   checkpoint, so a query that restarts before its first micro-batch is 
committed starts from the
-  same offset. For each micro-batch, or only once, before the first 
micro-batch that it plans, with
-  `Trigger.AvailableNow`, it asks for the `latest_offset`, with the offset 
that the micro-batch
-  starts after, plans the partitions with `stream_partitions`, which also 
returns the state of the
-  stream reader, and calls `read` on the executors. Offsets are defined by the 
library, as compact
-  JSON on a single line: Spark stores each offset as a line of the offset log 
of the checkpoint,
-  and rejects an offset that is not valid JSON or contains a line break, such 
as pretty-printed
-  JSON. Spark keeps what it needs in the checkpoint of the source under the 
names that start with
-  `_spark`, which the library must not use.
+  same offset. For each micro-batch, it asks for the `latest_offset`, with the 
offset that the
+  micro-batch starts after, plans the partitions with `stream_partitions`, 
which also returns the
+  state of the stream reader, and calls `read` on the executors. With 
`Trigger.AvailableNow`, a run
+  reads the data that was available when it started: Spark asks for the 
`latest_offset` without a
+  limit once, before the first micro-batch that it plans, and reads up to it. 
Offsets are defined
+  by the library, as compact JSON on a single line: Spark stores each offset 
as a line of the
+  offset log of the checkpoint, and rejects an offset that is not valid JSON 
or contains a line
+  break, such as pretty-printed JSON. Spark finds new data by comparing the 
latest offset with the
+  start as strings, so the same position must always have the same JSON. Spark 
keeps what it needs
+  in the checkpoint of the source under the names that start with `_spark`, 
which the library must
+  not use. A data source with the capability
+  `SPARK_DS_ADMISSION_CONTROL` limits the data of each micro-batch, for 
example with an option for
+  a maximum number of rows: Spark asks for the limit once with 
`default_read_limit`, which sets a
+  `SparkDsReadLimit`, and passes it to the `latest_offset` that ends each 
micro-batch, in the
+  `limit` of its `SparkDsLatestOffsetInfo`, which then returns the end of the 
data of the next
+  micro-batch within the limit. When there is data after the start, the latest 
offset must be after
+  the start, even if the next unit of data, such as a file, is larger than the 
limit: Spark sees no
+  new data otherwise. With `Trigger.AvailableNow`, the latest offset without a 
limit that Spark asks
+  for first is the end of the run: Spark passes it as the `run_end` that the 
limited latest offsets
+  must not go beyond, so that the run reads the data that was available when 
it started in
+  micro-batches within the limit. A query that restarts starts a new run, with 
a new end. So with a
+  limit, Spark calls `latest_offset` twice for some micro-batches: with 
`Trigger.AvailableNow`, for
+  the first one, first without a limit for the end of the run, and with the 
other triggers, for each
+  one, first with `SPARK_DS_READ_ALL_AVAILABLE` only for the progress of the 
query. Spark does not
+  ask again with the limit when the latest offset without a limit is the same 
as the start, such as
+  while a query waits for new data: there is no data to limit then. So a 
library must not count the
+  calls as micro-batches. `Trigger.Once` ignores the limit, and so does 
`Trigger.AvailableNow` when
+  another source of the query does not support it and
+  `spark.sql.streaming.triggerAvailableNowWrapper.enabled` is false, as by 
default, where Spark
+  reads all the available data in a single micro-batch. Spark still asks for 
the limit then, and
+  fails the query if it is not valid.
 - **Writes**: on the driver, right before the tasks of the write (or of a 
micro-batch) start, Spark
   creates a writer with `create_writer`, which gets a `SparkDsWriteInfo` with 
the mode of the write,
   `SPARK_DS_WRITE_APPEND` or `SPARK_DS_WRITE_TRUNCATE`, and the ID of the 
query of a streaming
@@ -185,6 +208,7 @@ data source must not declare a capability whose functions 
are `NULL`:
 | All | `create`, `close` | `last_error`, `schema` |
 | `SPARK_DS_BATCH_READ` | `create_reader`, `partitions`, `read`, 
`close_reader` | `push_limit`, `prune_columns` |
 | `SPARK_DS_MICRO_BATCH_READ` | `create_stream_reader`, `initial_offset`, 
`latest_offset`, `stream_partitions`, `read`, `close_stream_reader` | 
`commit_offset` |
+| `SPARK_DS_ADMISSION_CONTROL`, with `SPARK_DS_MICRO_BATCH_READ` | 
`default_read_limit` | |
 | `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` | `create_writer`, 
`commit`, `abort`, `close_writer`, `create_data_writer`, `write`, 
`commit_data_writer`, `abort_data_writer`, `close_data_writer` | |
 
 `SPARK_DS_BATCH_TRUNCATE` declares that a data source supports 
`SPARK_DS_WRITE_TRUNCATE` in batch
@@ -216,8 +240,8 @@ task does not end.
 ### Compatibility
 
 The function table, `SparkDsLibrary`, and the structs that Spark passes to the 
library,
-`SparkDsHost`, `SparkDsTaskContext` and those whose names end with `Info`, 
such as
-`SparkDsCreateInfo`, start with their size, so that they can grow. The version 
of the interface,
+`SparkDsHost`, `SparkDsTaskContext`, `SparkDsReadLimit` and those whose names 
end with `Info`, such
+as `SparkDsCreateInfo`, start with their size, so that they can grow. The 
version of the interface,
 1, is `SPARK_DS_ABI_VERSION`, which the table and the manifest of a package 
declare. Within a
 version:
 
@@ -226,7 +250,9 @@ version:
 - Spark only reads the functions that fit in the size of the table of a 
library, so a library built
   against an earlier header keeps working with later releases of Spark, with 
the functions that it
   does not know as `NULL`. A library must only read the fields that fit in the 
size of a struct
-  that Spark passes, so that it also runs on earlier releases of Spark that 
support its version;
+  that Spark passes, and only set those of a struct that it fills, such as the 
`SparkDsReadLimit`
+  of `default_read_limit`, so that it also runs on earlier releases of Spark 
that support its
+  version;
 - Spark only uses a new capability, write mode or kind of read with a data 
source that declares
   it.
 
@@ -246,7 +272,6 @@ later as functions or fields appended to the structs of the 
interface:
   hint. Adaptive query execution can still broadcast it at run time, from the 
size of a shuffle of
   the scan;
 - custom metrics, shown in the Spark UI;
-- admission control for streaming reads, such as a maximum number of rows for 
each micro-batch;
 - continuous processing (`Trigger.Continuous`): a continuous query that reads 
from or writes to a
   native data source fails when it starts;
 - the pushdown of aggregates and of top-N queries (`ORDER BY` with `LIMIT`), 
and reporting the
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index 4095f7c7f15a..f6424880f413 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -77,6 +77,7 @@ final class NativeLibrary {
   static final long STREAMING_WRITE = 8;
   static final long BATCH_TRUNCATE = 16;
   static final long STREAMING_TRUNCATE = 32;
+  static final long ADMISSION_CONTROL = 64;
 
   // The modes of a write, SPARK_DS_WRITE_APPEND and SPARK_DS_WRITE_TRUNCATE.
   static final int WRITE_APPEND = 0;
@@ -86,6 +87,10 @@ final class NativeLibrary {
   static final int READ_BATCH = 0;
   static final int READ_MICRO_BATCH = 1;
 
+  // The kinds of read limits, SPARK_DS_READ_ALL_AVAILABLE and 
SPARK_DS_READ_MAX_ROWS.
+  static final int READ_ALL_AVAILABLE = 0;
+  static final int READ_MAX_ROWS = 1;
+
   // The C types of the parameters and the results of the functions.
   private enum C { VOID, BOOL, I32, I64, PTR }
 
@@ -115,7 +120,8 @@ final class NativeLibrary {
     WRITE("write", C.I32, C.PTR, C.PTR),
     COMMIT_DATA_WRITER("commit_data_writer", C.I32, C.PTR, C.PTR, C.PTR, 
C.PTR),
     ABORT_DATA_WRITER("abort_data_writer", C.I32, C.PTR, C.PTR),
-    CLOSE_DATA_WRITER("close_data_writer", C.VOID, C.PTR);
+    CLOSE_DATA_WRITER("close_data_writer", C.VOID, C.PTR),
+    DEFAULT_READ_LIMIT("default_read_limit", C.I32, C.PTR, C.PTR);
 
     /** The name of the field of the function in the table. */
     final String fieldName;
@@ -158,8 +164,13 @@ final class NativeLibrary {
   private static final long COMMIT_OFFSET_INFO_END = 8;
   private static final long STREAM_READER_INFO_SIZE = 16;
   private static final long STREAM_READER_INFO_CHECKPOINT_LOCATION = 8;
-  private static final long LATEST_OFFSET_INFO_SIZE = 16;
+  private static final long LATEST_OFFSET_INFO_SIZE = 32;
   private static final long LATEST_OFFSET_INFO_START = 8;
+  private static final long LATEST_OFFSET_INFO_LIMIT = 16;
+  private static final long LATEST_OFFSET_INFO_RUN_END = 24;
+  private static final long READ_LIMIT_SIZE = 24;
+  private static final long READ_LIMIT_KIND = 8;
+  private static final long READ_LIMIT_MAX_ROWS = 16;
   private static final long TASK_SIZE = 40;
   private static final long TASK_STAGE_ID = 8;
   private static final long TASK_STAGE_ATTEMPT_NUMBER = 12;
@@ -254,6 +265,12 @@ final class NativeLibrary {
     layout.put("SparkDsLatestOffsetInfo", LATEST_OFFSET_INFO_SIZE);
     layout.put("SparkDsLatestOffsetInfo.size", 0L);
     layout.put("SparkDsLatestOffsetInfo.start", LATEST_OFFSET_INFO_START);
+    layout.put("SparkDsLatestOffsetInfo.limit", LATEST_OFFSET_INFO_LIMIT);
+    layout.put("SparkDsLatestOffsetInfo.run_end", LATEST_OFFSET_INFO_RUN_END);
+    layout.put("SparkDsReadLimit", READ_LIMIT_SIZE);
+    layout.put("SparkDsReadLimit.size", 0L);
+    layout.put("SparkDsReadLimit.kind", READ_LIMIT_KIND);
+    layout.put("SparkDsReadLimit.max_rows", READ_LIMIT_MAX_ROWS);
     layout.put("SparkDsTaskContext", TASK_SIZE);
     layout.put("SparkDsTaskContext.size", 0L);
     layout.put("SparkDsTaskContext.stage_id", TASK_STAGE_ID);
@@ -866,22 +883,62 @@ final class NativeLibrary {
     }
   }
 
+  /** A limit on the data of a micro-batch, as in SparkDsReadLimit. */
+  static final class ReadLimitInfo {
+    /** {@link #READ_ALL_AVAILABLE} or {@link #READ_MAX_ROWS}, or what the 
library set. */
+    final int kind;
+    /** The maximum number of rows with {@link #READ_MAX_ROWS}. */
+    final long maxRows;
+
+    ReadLimitInfo(int kind, long maxRows) {
+      this.kind = kind;
+      this.maxRows = maxRows;
+    }
+  }
+
   /**
    * Returns the latest offset.
    *
    * @param start the offset that the next micro-batch starts after
+   * @param limit the limit on the data after start
+   * @param runEnd the offset that a run of Trigger.AvailableNow reads up to, 
or null
    */
-  String latestOffset(long streamReader, String start) throws Throwable {
+  String latestOffset(long streamReader, String start, ReadLimitInfo limit, 
String runEnd)
+      throws Throwable {
     MethodHandle f = function(Function.LATEST_OFFSET);
     try (Memory memory = new Memory(); Sink sink = new Sink()) {
+      long readLimit = memory.zeroed(READ_LIMIT_SIZE);
+      Platform.putLong(null, readLimit, READ_LIMIT_SIZE);
+      Platform.putInt(null, readLimit + READ_LIMIT_KIND, limit.kind);
+      Platform.putLong(null, readLimit + READ_LIMIT_MAX_ROWS, limit.maxRows);
       long info = memory.zeroed(LATEST_OFFSET_INFO_SIZE);
       Platform.putLong(null, info, LATEST_OFFSET_INFO_SIZE);
       Platform.putLong(null, info + LATEST_OFFSET_INFO_START, 
memory.string(start));
+      Platform.putLong(null, info + LATEST_OFFSET_INFO_LIMIT, readLimit);
+      Platform.putLong(null, info + LATEST_OFFSET_INFO_RUN_END,
+        runEnd == null ? 0 : memory.string(runEnd));
       check((int) f.invokeExact(streamReader, info, Ffm.SINK, sink.id), 
Function.LATEST_OFFSET);
       return offset(Function.LATEST_OFFSET, sink);
     }
   }
 
+  /**
+   * Returns the default read limit of a stream reader, whose kind is {@link 
#READ_ALL_AVAILABLE}
+   * if the library sets nothing.
+   */
+  ReadLimitInfo defaultReadLimit(long streamReader) throws Throwable {
+    MethodHandle f = function(Function.DEFAULT_READ_LIMIT);
+    try (Memory memory = new Memory()) {
+      long limit = memory.zeroed(READ_LIMIT_SIZE);
+      Platform.putLong(null, limit, READ_LIMIT_SIZE);
+      Platform.putInt(null, limit + READ_LIMIT_KIND, READ_ALL_AVAILABLE);
+      check((int) f.invokeExact(streamReader, limit), 
Function.DEFAULT_READ_LIMIT);
+      return new ReadLimitInfo(
+        Platform.getInt(null, limit + READ_LIMIT_KIND),
+        Platform.getLong(null, limit + READ_LIMIT_MAX_ROWS));
+    }
+  }
+
   /** Returns the offset that a function passed to the sink, which it must 
pass. */
   private static String offset(Function function, Sink sink) throws Throwable {
     byte[] offset = single(sink, function);
diff --git 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index a8d7785ee1b1..b716b6469c59 100644
--- 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++ 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -64,15 +64,28 @@
  * - Streaming reads. On the driver, Spark creates a stream reader with 
create_stream_reader, with
  *   the columns to read and the location of the checkpoint of the source, and 
keeps it until the
  *   streaming query stops. When the checkpoint does not have the initial 
offset of the source yet,
- *   Spark asks for it with initial_offset, and keeps it in the checkpoint. 
For each micro-batch, or
- *   only before the first one with Trigger.AvailableNow, it asks for the 
latest_offset, with the
- *   offset that the micro-batch starts after, plans the partitions with 
stream_partitions, which
- *   also returns the state of the stream reader, then calls read on the 
executors. Offsets are
- *   defined by the library, as compact JSON on a single line: Spark stores 
each offset as a line of
- *   the offset log of the checkpoint, so it rejects an offset that is not 
valid JSON or contains a
- *   line break. When a streaming query restarts, from its checkpoint, Spark 
creates a new stream
- *   reader and may call stream_partitions again for the last range that it 
planned before, so the
- *   partitions of a range must read the same data each time.
+ *   Spark asks for it with initial_offset, and keeps it in the checkpoint. 
For each micro-batch, it
+ *   asks for the latest_offset, with the offset that the micro-batch starts 
after, plans the
+ *   partitions with stream_partitions, which also returns the state of the 
stream reader, then
+ *   calls read on the executors. With Trigger.AvailableNow, a run reads the 
data that was
+ *   available when it started: Spark asks for the latest_offset without a 
limit once, before the
+ *   first micro-batch that it plans, as the end of the run, and reads up to 
it. A data source with
+ *   the capability SPARK_DS_ADMISSION_CONTROL limits the data of each 
micro-batch: Spark asks for
+ *   the limit once with default_read_limit, and passes it to the 
latest_offset that ends each
+ *   micro-batch, also with Trigger.AvailableNow, whose run then reads up to 
its end in several
+ *   micro-batches. So with a limit, Spark calls latest_offset twice for some 
micro-batches: with
+ *   Trigger.AvailableNow, for the first one, first without a limit for the 
end of the run, and
+ *   with the other triggers, for each one, first with 
SPARK_DS_READ_ALL_AVAILABLE only for the
+ *   progress of the query. Spark does not ask again with the limit when the 
latest offset without
+ *   a limit is the same as the start, such as while a query waits for new 
data: there is no data
+ *   to limit then. So a library must not count the calls of latest_offset as
+ *   micro-batches, or reserve data in them. Offsets are defined by the 
library, as compact JSON on
+ *   a single line: Spark stores each offset as a line of the offset log of 
the checkpoint, so it
+ *   rejects an offset that is not valid JSON or contains a line break. Spark 
finds new data by
+ *   comparing the latest offset with the start as strings, so the same 
position must always have
+ *   the same JSON. When a streaming query restarts, from its checkpoint, 
Spark creates a new
+ *   stream reader and may call stream_partitions again for the last range 
that it planned before,
+ *   so the partitions of a range must read the same data each time.
  * - Writes. On the driver, right before it starts the tasks of the write (or 
of a micro-batch),
  *   Spark creates a writer with create_writer, which also returns the state 
of the writer. If Spark
  *   rejects what create_writer returned, such as a state set more than once, 
it aborts the writer,
@@ -179,11 +192,12 @@
  * - Options. Option keys are passed in lower case, because Spark matches them
  *   case-insensitively.
  * - Optional functions. A library only sets the functions of the operations 
it supports in its
- *   table, and leaves the others NULL. A data source must not declare a 
capability whose
- *   functions are NULL: Spark then fails the operation. The pushdown 
functions are optional:
- *   without them, Spark pushes nothing down. So are schema, without which the 
user must specify
- *   the schema, commit_offset and last_error. A reader or a writer without a 
state does not call
- *   the set_state sink of partitions, stream_partitions or create_writer.
+ *   table, and leaves the others NULL. A data source must not declare a 
capability whose functions
+ *   are NULL: Spark then fails the operation, as for default_read_limit with 
the capability
+ *   SPARK_DS_ADMISSION_CONTROL. The pushdown functions are optional: without 
them, Spark pushes
+ *   nothing down. So are schema, without which the user must specify the 
schema, commit_offset and
+ *   last_error. A reader or a writer without a state does not call the 
set_state sink of
+ *   partitions, stream_partitions or create_writer.
  * - Threads. Spark calls the library from multiple threads, and calls on 
different handles may
  *   run concurrently. Calls on the same handle never run concurrently. A 
handle may be released
  *   from a thread other than the threads that used it, such as a JVM Cleaner 
thread when Spark
@@ -201,7 +215,9 @@
  *   never change, nor do the semantics of the existing functions;
  * - Spark only reads the functions that fit in the size of the table of a 
library, so a library
  *   built against an earlier header keeps working, with the functions that it 
does not know as
- *   NULL. A library must only read the fields that fit in the size of a 
struct that Spark passes;
+ *   NULL. A library must only read the fields that fit in the size of a 
struct that Spark passes,
+ *   and only set those of a struct that it fills, such as the 
SparkDsReadLimit of
+ *   default_read_limit;
  * - Spark only uses a new capability, write mode or kind of read with a data 
source that declares
  *   it.
  * An incompatible change uses a new SPARK_DS_ABI_VERSION: spark_ds_init gets 
the newest version
@@ -377,6 +393,25 @@ typedef struct SparkDsInitialOffsetInfo {
   int64_t size;
 } SparkDsInitialOffsetInfo;
 
+/* The kinds of the limits on the data of a micro-batch, see SparkDsReadLimit. 
*/
+#define SPARK_DS_READ_ALL_AVAILABLE 0
+#define SPARK_DS_READ_MAX_ROWS 1
+
+/*
+ * A limit on the data of each micro-batch of a streaming read, which 
default_read_limit returns
+ * for a data source with the capability SPARK_DS_ADMISSION_CONTROL, and 
latest_offset gets in its
+ * SparkDsLatestOffsetInfo. In default_read_limit, the library sets the fields 
that fit in the size
+ * that Spark set.
+ */
+typedef struct SparkDsReadLimit {
+  /* The size of this struct in the header that Spark was built with, as in 
SparkDsHost. */
+  int64_t size;
+  /* SPARK_DS_READ_ALL_AVAILABLE, as Spark initializes it, or 
SPARK_DS_READ_MAX_ROWS. */
+  int32_t kind;
+  /* With SPARK_DS_READ_MAX_ROWS, the maximum number of rows of a micro-batch, 
at least 1. */
+  int64_t max_rows;
+} SparkDsReadLimit;
+
 /* A request for the latest offset of a stream reader, as latest_offset gets 
it. */
 typedef struct SparkDsLatestOffsetInfo {
   /* The size of this struct in the header that Spark was built with, as in 
SparkDsHost. */
@@ -384,10 +419,31 @@ typedef struct SparkDsLatestOffsetInfo {
   /*
    * The offset, as JSON, that the next micro-batch starts after: the end of 
the last micro-batch,
    * or the initial offset before the first one, for every trigger. With 
Trigger.AvailableNow,
-   * Spark asks for the latest offset only once, before the first micro-batch 
that it plans, and
-   * reads up to it.
+   * Spark asks for the latest offset without a limit once, before the first 
micro-batch that it
+   * plans, and reads up to it: in a single micro-batch, or with a limit, in 
micro-batches of
+   * which it asks for each latest offset with that offset as the end.
    */
   const char* start;
+  /*
+   * The limit on the data after start, never NULL: the default read limit of 
a data source with
+   * the capability SPARK_DS_ADMISSION_CONTROL, or a limit of the kind 
SPARK_DS_READ_ALL_AVAILABLE,
+   * as always for the other data sources, for Trigger.Once, and for 
Trigger.AvailableNow when Spark
+   * runs a single micro-batch because another source of the query does not 
support it. With a
+   * limit, the latest offset is the end of the data of the next micro-batch, 
within the limit.
+   * When there is data after start, the latest offset must be after start, 
even if the next unit
+   * of data that an offset can end, such as a file, is larger than the limit: 
Spark sees no new
+   * data otherwise, and never reads further.
+   */
+  const SparkDsReadLimit* limit;
+  /*
+   * With Trigger.AvailableNow and a limit of the kind SPARK_DS_READ_MAX_ROWS, 
the end of the run:
+   * the offset that the run reads up to, which the latest offset must not go 
beyond, so that the
+   * run reads the data that was available when it started; NULL otherwise, as 
in the call without
+   * a limit that asks for the end of the run. It is not the end of the next 
micro-batch. A query
+   * that restarts starts a new run, with a new end: Spark first runs again 
the micro-batch that it
+   * planned but did not commit, if any, up to the end in its offset log, 
without asking.
+   */
+  const char* run_end;
 } SparkDsLatestOffsetInfo;
 
 /* A request for the partitions of a micro-batch, as stream_partitions gets 
it. */
@@ -553,6 +609,7 @@ typedef struct SparkDsFinishDataWriterInfo {
 #define SPARK_DS_STREAMING_WRITE 8     /* streaming writes, with create_writer 
*/
 #define SPARK_DS_BATCH_TRUNCATE 16     /* batch writes with the mode 
SPARK_DS_WRITE_TRUNCATE */
 #define SPARK_DS_STREAMING_TRUNCATE 32 /* streaming writes with the mode 
SPARK_DS_WRITE_TRUNCATE */
+#define SPARK_DS_ADMISSION_CONTROL 64  /* micro-batch reads with read limits 
(default_read_limit) */
 
 /*
  * The functions of a library, which spark_ds_init returns. A library sets the 
functions of the
@@ -686,9 +743,10 @@ typedef struct SparkDsLibrary {
                         void* sink);
 
   /*
-   * Returns the most recent offset available.
+   * Returns the most recent offset available after start, within the limit 
and up to the end of
+   * the run in its info. Spark may call it more than once for a micro-batch, 
see Streaming reads.
    *
-   * info: the offset that the next micro-batch starts after
+   * info: the offset that the next micro-batch starts after, the limit, and 
the end of the run
    * set_offset: called once with the offset, as compact JSON on a single line
    */
   int (*latest_offset)(void* reader, const SparkDsLatestOffsetInfo* info, 
SparkDsSink set_offset,
@@ -811,6 +869,24 @@ typedef struct SparkDsLibrary {
 
   /* Releases a data writer handle, after its commit or abort. */
   void (*close_data_writer)(void* data_writer);
+
+  /* 
-------------------------------------------------------------------------------------------
+   * Admission control (driver)
+   * 
-------------------------------------------------------------------------------------------
 */
+
+  /*
+   * Returns the limit on the data of each micro-batch of a stream reader, 
typically from the
+   * options of the data source, such as a maximum number of rows. Spark asks 
once, when the
+   * streaming query starts, for a data source with the capability 
SPARK_DS_ADMISSION_CONTROL, and
+   * passes the limit to the latest_offset that ends each micro-batch, except 
with Trigger.Once,
+   * and with Trigger.AvailableNow when another source of the query does not 
support it and
+   * spark.sql.streaming.triggerAvailableNowWrapper.enabled is false, as by 
default, where Spark
+   * reads all the available data in a single micro-batch. Spark asks for the 
limit then too, and
+   * fails the query if it is not valid, though it does not pass it.
+   *
+   * limit: the limit to set, initialized by Spark with its size and 
SPARK_DS_READ_ALL_AVAILABLE
+   */
+  int (*default_read_limit)(void* reader, SparkDsReadLimit* limit);
 } SparkDsLibrary;
 
 /*
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index a98cb66a485a..13f7ba56c879 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -21,6 +21,7 @@ import java.util
 import org.apache.spark.sql.connector.catalog.TableCapability
 import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, 
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
 import org.apache.spark.sql.connector.read.{InputPartition, 
PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming.ReadLimit
 import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -123,12 +124,23 @@ trait ColumnarStreamReader {
   def initialOffset(): String
 
   /**
-   * Returns the latest offset available.
+   * Returns the limit on the data of each micro-batch, which Spark asks for 
once, when the query
+   * starts: `ReadLimit.allAvailable()` by default, or a `ReadLimit.maxRows`.
+   */
+  def defaultReadLimit(): ReadLimit = ReadLimit.allAvailable()
+
+  /**
+   * Returns the latest offset available, within a limit.
    *
    * @param start the offset that the next micro-batch starts after: the end 
of the last
    *              micro-batch, or the initial offset before the first one
+   * @param limit `ReadLimit.allAvailable()`, or the default read limit: the 
latest offset is then
+   *              the end of the data of the next micro-batch. Spark only 
passes the default read
+   *              limit when the latest offset without a limit is not `start`
+   * @param runEnd with `Trigger.AvailableNow` and a limit, the offset that 
the run reads up to,
+   *               which the latest offset does not go beyond
    */
-  def latestOffset(start: String): String
+  def latestOffset(start: String, limit: ReadLimit, runEnd: Option[String]): 
String
 
   /**
    * Plans the partitions that read the data after `start` up to and including 
`end`. Spark plans
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
index 6b4d8c0db3dc..81cb1e0e9ac5 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
@@ -224,12 +224,12 @@ class ColumnarBatchScan(planner: ColumnarBatchPlanner) 
extends Batch {
 case class ColumnarStreamOffset(json: String) extends Offset
 
 /**
- * The micro-batch stream of a [[ColumnarDataSource]]. It supports admission 
control only so that
- * the reader gets the offset that each micro-batch starts after, the end of 
the last one, for
- * every trigger: its default read limit reads all the available data. With
- * `Trigger.AvailableNow`, Spark runs micro-batches up to the first latest 
offset that Spark asks
- * for, rather than wrapping the stream in one that asks for the latest offset 
from the initial
- * offset.
+ * The micro-batch stream of a [[ColumnarDataSource]]. The reader gets the 
offset that each
+ * micro-batch starts after, the end of the last one, for every trigger, and 
the limit on the data
+ * of a micro-batch, its default read limit, all the available data unless it 
limits it. With
+ * `Trigger.AvailableNow`, Spark runs micro-batches up to the first latest 
offset without a limit
+ * that Spark asks for, rather than wrapping the stream in one that asks for 
the latest offset from
+ * the initial offset: in one micro-batch, or within the limit in several.
  *
  * @param checkpointLocation the location of the checkpoint of the source, 
where the stream keeps
  *                           the initial offset, under 
`ColumnarMicroBatchStream.InitialOffsetLog`
@@ -247,11 +247,22 @@ class ColumnarMicroBatchStream(reader: 
ColumnarStreamReader, checkpointLocation:
   @GuardedBy("this")
   private var plannedPartitions: Try[Array[InputPartition]] = _
 
-  // The offset that the reader returned last, which the progress of the query 
reports.
+  // The latest offset without a limit that the reader returned last, which 
the progress of the
+  // query reports.
   @volatile private var lastLatestOffset: Offset = _
+  // The limit on the data of each micro-batch, asked once, when the query 
starts. Spark asks for
+  // it before it records the streams to stop when the query stops, so a 
stream whose limit fails
+  // stops its reader itself.
+  private lazy val readLimit = try {
+    reader.defaultReadLimit()
+  } catch {
+    case e: Throwable =>
+      try reader.stop() catch { case stopError: Throwable => 
e.addSuppressed(stopError) }
+      throw e
+  }
   // Whether the trigger is Trigger.AvailableNow.
   @volatile private var availableNow = false
-  // The end of the run with Trigger.AvailableNow: the first latest offset 
asked.
+  // The end of the run with Trigger.AvailableNow: the first latest offset 
without a limit asked.
   @volatile private var availableNowEnd: Offset = _
   // Spark asks for the initial offset for the start of the first latest 
offset, and again to plan
   // the first micro-batch, which both get the same offset. The reader is only 
asked when the
@@ -275,17 +286,27 @@ class ColumnarMicroBatchStream(reader: 
ColumnarStreamReader, checkpointLocation:
       "latestOffset(Offset, ReadLimit) should be called instead of 
latestOffset().")
   }
 
+  override def getDefaultReadLimit: ReadLimit = readLimit
+
   override def latestOffset(start: Offset, limit: ReadLimit): Offset = {
-    val latest = if (availableNow && availableNowEnd != null) {
-      availableNowEnd
+    def ask(limit: ReadLimit, runEnd: Option[Offset]): Offset =
+      ColumnarStreamOffset(reader.latestOffset(start.json(), limit, 
runEnd.map(_.json())))
+    // The latest offset without a limit, which the progress reports: with 
Trigger.AvailableNow,
+    // the end of the run, asked once, and otherwise asked for each 
micro-batch.
+    val latest = if (!availableNow) {
+      ask(ReadLimit.allAvailable(), None)
     } else {
-      ColumnarStreamOffset(reader.latestOffset(start.json()))
-    }
-    if (availableNow) {
-      availableNowEnd = latest
+      if (availableNowEnd == null) availableNowEnd = 
ask(ReadLimit.allAvailable(), None)
+      availableNowEnd
     }
     lastLatestOffset = latest
-    latest
+    // When the latest offset without a limit is the start, as Spark compares 
offsets, there is no
+    // data after the start to limit, so the reader is not asked again with 
the limit.
+    if (limit == ReadLimit.allAvailable() || latest.json() == start.json()) {
+      latest
+    } else {
+      ask(limit, Option(availableNowEnd))
+    }
   }
 
   override def prepareForTriggerAvailableNow(): Unit = {
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 40e1f2d44d49..e0530d283a9f 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -44,6 +44,7 @@ import org.apache.spark.sql.classic.SparkSession
 import org.apache.spark.sql.connector.catalog.TableCapability
 import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, 
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
 import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, 
PartitionReaderFactory, SupportsArrowStreamRead}
+import org.apache.spark.sql.connector.read.streaming.{ReadAllAvailable, 
ReadLimit, ReadMaxRows}
 import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
 import org.apache.spark.sql.errors.{QueryCompilationErrors, 
QueryExecutionErrors}
 import 
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchPlanner, 
ColumnarDataSource, ColumnarOnlyPartitionReaderFactory, ColumnarPlannedBatch, 
ColumnarStreamReader, ColumnarStreamWriter, ColumnarWriter}
@@ -128,15 +129,20 @@ class NativeDataSource(
     NativeArrow.checkSupported(schema, name, "plan a scan of")
     val executorLocation = NativeDataSourceRegistry.distribute(location)
     val library = NativeLibraries.get(location)
-    val handle = create(library, "plan a scan of", 
NativeLibrary.MICRO_BATCH_READ,
-        "DATA_SOURCE_MICRO_BATCH_SCAN_NOT_SUPPORTED", 
NativeDataSource.MicroBatchReadFunctions) {
-      dataSource =>
-        NativeArrow.exportSchema(schema, timeZoneId) { address =>
-          library.createStreamReader(dataSource.handle, address, 
checkpointLocation)
-        }
+    val (handle, admissionControl) = create(library, "plan a scan of",
+        NativeLibrary.MICRO_BATCH_READ, 
"DATA_SOURCE_MICRO_BATCH_SCAN_NOT_SUPPORTED",
+        NativeDataSource.MicroBatchReadFunctions) { dataSource =>
+      val admissionControl = (dataSource.capabilities & 
NativeLibrary.ADMISSION_CONTROL) != 0
+      if (admissionControl) {
+        requireFunctions(library, "plan a scan of", 
Seq(NativeLibrary.Function.DEFAULT_READ_LIMIT))
+      }
+      val handle = NativeArrow.exportSchema(schema, timeZoneId) { address =>
+        library.createStreamReader(dataSource.handle, address, 
checkpointLocation)
+      }
+      (handle, admissionControl)
     }
     new NativeDataSourceStreamReader(executorLocation, name, schema, 
timeZoneId,
-      new NativeHandle(handle, library.closeStreamReader))
+      new NativeHandle(handle, library.closeStreamReader), admissionControl)
   }
 
   override def writer(schema: StructType, overwrite: Boolean): ColumnarWriter 
= {
@@ -333,7 +339,8 @@ class NativeDataSourceStreamReader(
     name: String,
     schema: StructType,
     timeZoneId: String,
-    handle: NativeHandle)
+    handle: NativeHandle,
+    admissionControl: Boolean)
   extends ColumnarStreamReader {
   import NativeCalls._
 
@@ -357,9 +364,41 @@ class NativeDataSourceStreamReader(
     checkOffset("initial_offset", call(name, "plan a scan 
of")(library.initialOffset(handle.get)))
   }
 
-  override def latestOffset(start: String): String = synchronized {
-    checkOffset("latest_offset",
-      call(name, "plan a scan of")(library.latestOffset(handle.get, start)))
+  /** The limit that the library returns, with the capability 
SPARK_DS_ADMISSION_CONTROL. */
+  override def defaultReadLimit(): ReadLimit = synchronized {
+    if (!admissionControl) {
+      ReadLimit.allAvailable()
+    } else {
+      val limit = call(name, "plan a scan 
of")(library.defaultReadLimit(handle.get))
+      val (kind, maxRows) = (limit.kind, limit.maxRows)
+      kind match {
+        case NativeLibrary.READ_ALL_AVAILABLE => ReadLimit.allAvailable()
+        case NativeLibrary.READ_MAX_ROWS if maxRows >= 1 => 
ReadLimit.maxRows(maxRows)
+        case NativeLibrary.READ_MAX_ROWS =>
+          throw error(name, "plan a scan of", s"default_read_limit returned a 
maximum of " +
+            s"$maxRows rows, but a micro-batch reads at least 1 row.")
+        case _ =>
+          throw error(name, "plan a scan of", s"default_read_limit returned 
the unknown kind " +
+            s"of read limit $kind.")
+      }
+    }
+  }
+
+  override def latestOffset(
+      start: String,
+      limit: ReadLimit,
+      runEnd: Option[String]): String = synchronized {
+    val info = limit match {
+      case _: ReadAllAvailable =>
+        new NativeLibrary.ReadLimitInfo(NativeLibrary.READ_ALL_AVAILABLE, 0L)
+      case maxRows: ReadMaxRows =>
+        new NativeLibrary.ReadLimitInfo(NativeLibrary.READ_MAX_ROWS, 
maxRows.maxRows())
+      case other => throw SparkException.internalError(
+        s"A native data source does not support the read limit $other.")
+    }
+    checkOffset("latest_offset", call(name, "plan a scan of") {
+      library.latestOffset(handle.get, start, info, runEnd.orNull)
+    })
   }
 
   /**
diff --git 
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc 
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index 87551a695b31..de2f5b76a57d 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -61,15 +61,20 @@
 //                   "closed_path", closing the stream reader creates that 
file. With the option
 //                   "checkpoint_path", creating the stream reader writes its 
checkpoint location to
 //                   that file, and with "starts_path", each call to 
latest_offset appends its start
-//                   offset to that file, as a line. With 
"initial_offsets_path", each call to
-//                   initial_offset appends a line to that file, and returns 
the number of calls
-//                   before it as the offset, as an initial offset that 
depends on when it is asked
-//                   does. With "columns_path", creating the stream reader 
writes the columns to
-//                   read to that file, and with "ranges_path", each call to 
stream_partitions
-//                   appends the start and the end offsets that it got to that 
file, as a line.
-//                   The partitions and the state that stream_partitions 
returns have the number of
-//                   the call, which read checks. With the option 
"state_padding", the state has
-//                   that many more bytes.
+//                   offset to that file, as a line. With the option 
"max_rows_per_trigger", it
+//                   declares the capability SPARK_DS_ADMISSION_CONTROL, and 
its default read limit
+//                   is that many rows, or with "bad_read_limit", an unknown 
"kind" of limit or a
+//                   maximum of 0 "rows". With "limits_path", each call to 
latest_offset appends its
+//                   start, its limit, "all" or the maximum number of rows, 
and its end of the run
+//                   or "-" to that file, as a line, such as "0 5 12". With 
"initial_offsets_path",
+//                   each call to initial_offset appends a line to that file, 
and returns the number
+//                   of calls before it as the offset, as an initial offset 
that depends on when it
+//                   is asked does. With "columns_path", creating the stream 
reader writes the
+//                   columns to read to that file, and with "ranges_path", 
each call to
+//                   stream_partitions appends the start and the end offsets 
that it got to that
+//                   file, as a line. The partitions and the state that 
stream_partitions returns
+//                   have the number of the call, which read checks. With the 
option
+//                   "state_padding", the state has that many more bytes.
 //
 // native_range supports batch reads, native_counter micro-batch streaming 
reads and native_sink
 // batch and streaming writes, which may replace the existing data. With the 
option
@@ -1130,7 +1135,11 @@ uint64_t capabilitiesOf(const DataSource& source) {
   if (!declared.empty()) return std::stoull(declared);
   if (source.name == name("native_range")) return SPARK_DS_BATCH_READ;
 #ifndef READ_ONLY
-  if (source.name == name("native_counter")) return SPARK_DS_MICRO_BATCH_READ;
+  if (source.name == name("native_counter")) {
+    bool limited = !get(source.options, "max_rows_per_trigger", "").empty() ||
+                   !get(source.options, "bad_read_limit", "").empty();
+    return SPARK_DS_MICRO_BATCH_READ | (limited ? SPARK_DS_ADMISSION_CONTROL : 
0);
+  }
   if (source.name == name("native_sink")) {
     return SPARK_DS_BATCH_WRITE | SPARK_DS_STREAMING_WRITE | 
SPARK_DS_BATCH_TRUNCATE |
            SPARK_DS_STREAMING_TRUNCATE;
@@ -1253,7 +1262,8 @@ int createStreamReader(void* handle, struct ArrowSchema* 
schema,
     reader->state["batch_size"] = get(source->options, "batch_size", "1000");
     for (const char* option :
          {"commit_path", "bad_offset", "bad_initial_offset", "closed_path", 
"starts_path",
-          "initial_offsets_path", "ranges_path", "states", "stateless", 
"task_path", "fail"}) {
+          "initial_offsets_path", "ranges_path", "states", "stateless", 
"task_path",
+          "max_rows_per_trigger", "bad_read_limit", "limits_path", "fail"}) {
       reader->state[option] = get(source->options, option, "");
     }
     std::string padding = get(source->options, "state_padding", "");
@@ -1379,6 +1389,26 @@ int latestOffset(void* handle, const 
SparkDsLatestOffsetInfo* info, SparkDsSink
     auto start = 
static_cast<int64_t>(JsonParser(info->start).parse()["offset"].number);
     int64_t latest =
         std::min(getLong(state, "max_offset", 10), start + getLong(state, 
"step", 10));
+    // Within the limit, and up to the end of a run of Trigger.AvailableNow.
+    if (info->limit == nullptr) throw std::runtime_error("no read limit");
+    checkSize("SparkDsReadLimit", info->limit->size, sizeof(SparkDsReadLimit));
+    std::string limit = "all";
+    if (info->limit->kind == SPARK_DS_READ_MAX_ROWS) {
+      latest = std::min(latest, start + info->limit->max_rows);
+      limit = std::to_string(info->limit->max_rows);
+    } else if (info->limit->kind != SPARK_DS_READ_ALL_AVAILABLE) {
+      throw std::runtime_error("unknown kind of read limit " + 
std::to_string(info->limit->kind));
+    }
+    std::string run_end = "-";
+    if (info->run_end != nullptr) {
+      auto end = 
static_cast<int64_t>(JsonParser(info->run_end).parse()["offset"].number);
+      latest = std::min(latest, end);
+      run_end = std::to_string(end);
+    }
+    std::string limits_path = get(state, "limits_path", "");
+    if (!limits_path.empty()) {
+      appendLine(limits_path, std::to_string(start) + " " + limit + " " + 
run_end);
+    }
     sendOffset(state, "bad_offset", set_offset, sink, latest);
   });
 }
@@ -1773,6 +1803,23 @@ int abortDataWriter(void* handle, const 
SparkDsFinishDataWriterInfo* info) {
   });
 }
 
+int defaultReadLimit(void* handle, SparkDsReadLimit* limit) {
+  return guarded([&] {
+    checkSize("SparkDsReadLimit", limit->size, sizeof(SparkDsReadLimit));
+    const Options& state = fromHandle<Reader>(handle)->state;
+    std::string bad = get(state, "bad_read_limit", "");
+    if (bad == "kind") {
+      limit->kind = 7;
+    } else if (bad == "rows") {
+      limit->kind = SPARK_DS_READ_MAX_ROWS;
+      limit->max_rows = 0;
+    } else if (!get(state, "max_rows_per_trigger", "").empty()) {
+      limit->kind = SPARK_DS_READ_MAX_ROWS;
+      limit->max_rows = getLong(state, "max_rows_per_trigger", 0);
+    }
+  });
+}
+
 void closeDataWriter(void* handle) {
   if (usedAfterClose(handle)) return;
   std::unique_ptr<DataWriter> writer(fromHandle<DataWriter>(handle));
@@ -1825,6 +1872,7 @@ void closeDataWriter(void* handle) {
   table.commit_data_writer = commitDataWriter;
   table.abort_data_writer = abortDataWriter;
   table.close_data_writer = closeDataWriter;
+  table.default_read_limit = defaultReadLimit;
 #endif
   return table;
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 92ef0d87de92..bfcb536aeb0d 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -1101,7 +1101,7 @@ class TestRangeStreamReader(end: Long, schema: 
StructType, table: String)
 
   override def initialOffset(): String = "{\"offset\":0}"
 
-  override def latestOffset(start: String): String = {
+  override def latestOffset(start: String, limit: ReadLimit, runEnd: 
Option[String]): String = {
     TestRangeStreamReader.starts.compute(table, (_, old) => 
Option(old).getOrElse(Nil) :+ start)
     s"""{"offset":$end}"""
   }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 1928bbac3722..72d1f09e1b93 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -40,7 +40,7 @@ import org.scalatest.time.SpanSugar._
 import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TaskContext, TaskKilledException, TestUtils}
 import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
 import org.apache.spark.broadcast.Broadcast
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, 
QueryTest, Row}
+import org.apache.spark.sql.{AnalysisException, Dataset, 
IntegratedUDFTestUtils, QueryTest, Row}
 import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
 import org.apache.spark.sql.classic.{DataFrame, SparkSession}
 import org.apache.spark.sql.connector.catalog.SupportsRead
@@ -91,6 +91,9 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     compile("test_native_datasource_ne", "-DNO_LAST_ERROR", 
"-DNAME_PREFIX=\"ne_\"")
   private lazy val smallTableLibrary = compile("test_native_datasource_small",
     "-DTABLE_SIZE=offsetof(SparkDsLibrary,read)", "-DNAME_PREFIX=\"small_\"")
+  // A library built against the header before default_read_limit, whose table 
ends before it.
+  private lazy val noReadLimitLibrary = compile("test_native_datasource_nrl",
+    "-DTABLE_SIZE=offsetof(SparkDsLibrary,default_read_limit)", 
"-DNAME_PREFIX=\"nrl_\"")
   private lazy val failingInitLibrary =
     compile("test_native_datasource_if", "-DINIT_FAILURE=1", 
"-DNAME_PREFIX=\"if_\"")
   private lazy val noTableLibrary =
@@ -1042,6 +1045,139 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     assert(written.map(_.toLong).sorted.toSeq == (0L until 10L))
   }
 
+  nativeTest("limit the rows of each micro-batch with admission control") {
+    val session = newSession(defaultPackage)
+    // Runs the query with a trigger until it read the rows [0, 12), and 
returns the number of rows
+    // of each micro-batch, the latest offset that the first one reported, and 
the start, the limit
+    // and the end of the run of each call of latest_offset, as "limits_path" 
records them.
+    def run(
+        trigger: Option[Trigger],
+        options: (String, String)*): (Seq[Long], String, Seq[String]) = {
+      val limitsPath = new File(Utils.createTempDir(), "limits")
+      val writer = 
session.readStream.format("native_counter").option("max_offset", "12")
+        .option("limits_path", limitsPath.getPath)
+        .options(options.toMap).load()
+        .writeStream
+        .format("noop")
+        .option("checkpointLocation", Utils.createTempDir().getPath)
+      val query = trigger.map(writer.trigger).getOrElse(writer).start()
+      try {
+        if (trigger.isDefined) query.awaitTermination() else 
query.processAllAvailable()
+      } finally {
+        query.stop()
+      }
+      val batches = query.recentProgress.filter(_.numInputRows > 0).toSeq
+      (batches.map(_.numInputRows), batches.head.sources.head.latestOffset,
+        Files.readAllLines(limitsPath.toPath).asScala.toSeq)
+    }
+    // Each micro-batch reads at most 5 rows, all available from the start, 
while the progress
+    // reports the latest offset without the limit, which Spark asks for 
first. Spark only asks
+    // with the limit while there is data after the start.
+    val (rows, latest, calls) = run(None, "max_rows_per_trigger" -> "5")
+    assert((rows, latest) == ((Seq(5L, 5L, 2L), """{"offset":12}""")))
+    assert(calls.take(6) == Seq("0 all -", "0 5 -", "5 all -", "5 5 -", "10 
all -", "10 5 -"))
+    assert(calls.drop(6).nonEmpty && calls.drop(6).forall(_ == "12 all -"), 
calls)
+    // A run of Trigger.AvailableNow reads the rows available when it starts, 
within the limit:
+    // Spark asks for the end of the run once, without a limit, and passes it 
with the limit.
+    assert(run(Some(Trigger.AvailableNow()), "max_rows_per_trigger" -> "5") ==
+      ((Seq(5L, 5L, 2L), """{"offset":12}""", Seq("0 all -", "0 5 12", "5 5 
12", "10 5 12"))))
+    // Only the rows [0, 6) are available when the run starts, and 
native_counter makes 6 more
+    // rows available after each start: the run stops at the end that it got 
when it started,
+    // without asking again once it reached it.
+    assert(run(Some(Trigger.AvailableNow()), "max_rows_per_trigger" -> "5", 
"step" -> "6") ==
+      ((Seq(5L, 1L), """{"offset":6}""", Seq("0 all -", "0 5 6", "5 5 6"))))
+    // Trigger.Once ignores the limit, and reads all the available rows in one 
micro-batch.
+    assert(run(Some(Trigger.Once()), "max_rows_per_trigger" -> "5") ==
+      ((Seq(12L), """{"offset":12}""", Seq("0 all -"))))
+    // Without the capability, Spark asks for all the available rows, once for 
each micro-batch.
+    val (unlimitedRows, _, unlimitedCalls) = run(None, "max_rows_per_trigger" 
-> "5",
+      "capabilities" -> NativeLibrary.MICRO_BATCH_READ.toString)
+    assert(unlimitedRows == Seq(12L))
+    assert(unlimitedCalls.head == "0 all -" && unlimitedCalls.tail.forall(_ == 
"12 all -"),
+      unlimitedCalls)
+    // So it does with the capability, when the default read limit is all the 
available rows.
+    val limited = NativeLibrary.MICRO_BATCH_READ | 
NativeLibrary.ADMISSION_CONTROL
+    assert(run(None, "capabilities" -> limited.toString)._1 == Seq(12L))
+  }
+
+  nativeTest("start a new run of Trigger.AvailableNow with a limit when the 
query restarts") {
+    val session = newSession(defaultPackage)
+    val checkpoint = Utils.createTempDir()
+    val dir = Utils.createTempDir()
+    // The rows of each micro-batch that succeeded, with its ID.
+    val batches = ArrayBuffer.empty[(Long, Seq[Long])]
+    // Runs the query, which records the start, the limit and the end of the 
run of each call of
+    // latest_offset in the file `limits`.
+    def run(failingBatch: Long, limits: String): Unit = {
+      val writeBatch: (Dataset[Row], Long) => Unit = { (df, id) =>
+        if (id == failingBatch) throw new IllegalStateException(s"micro-batch 
$id fails")
+        batches += id -> df.collect().map(_.getLong(0)).toSeq.sorted
+      }
+      // A run from 0 ends at 6, since native_counter makes 6 more rows 
available after each start.
+      val query = session.readStream.format("native_counter")
+        .option("max_offset", "12").option("step", 
"6").option("max_rows_per_trigger", "5")
+        .option("limits_path", new File(dir, limits).getPath)
+        .load()
+        .writeStream
+        .trigger(Trigger.AvailableNow())
+        .option("checkpointLocation", checkpoint.getPath)
+        .foreachBatch(writeBatch)
+        .start()
+      try query.awaitTermination() finally query.stop()
+    }
+    def calls(limits: String): Seq[String] =
+      Files.readAllLines(new File(dir, limits).toPath).asScala.toSeq
+    // The second micro-batch, [5, 6), fails, so it is planned but not 
committed.
+    val e = intercept[Exception](run(failingBatch = 1, "first"))
+    assert(Iterator.iterate[Throwable](e)(_.getCause).takeWhile(_ != null)
+      .exists(_.getMessage == "micro-batch 1 fails"))
+    assert(batches.toSeq == Seq(0L -> (0L until 5L)))
+    assert(calls("first") == Seq("0 all -", "0 5 6", "5 5 6"))
+    // The new run first runs that micro-batch again, up to its end in the 
offset log, without
+    // asking for a latest offset, then asks for a new end, 12, and reads up 
to it within the limit.
+    run(failingBatch = -1, "second")
+    assert(batches.toSeq ==
+      Seq(0L -> (0L until 5L), 1L -> Seq(5L), 2L -> (6L until 11L), 3L -> 
Seq(11L)))
+    assert(calls("second") == Seq("6 all -", "6 5 12", "11 5 12"))
+  }
+
+  nativeTest("reject the read limits that a micro-batch cannot have") {
+    // Checks the error of a query, and returns whether its stream reader was 
closed then.
+    def check(format: String, options: Map[String, String], msg: String): 
Boolean = {
+      val packages = if (format.startsWith("nrl_")) {
+        createPackage(noReadLimitLibrary, dataSources("nrl_"))
+      } else {
+        defaultPackage
+      }
+      val closedPath = new File(Utils.createTempDir(), "closed")
+      val e = intercept[Exception] {
+        val query = 
newSession(packages).readStream.format(format).options(options)
+          .option("closed_path", closedPath.getPath)
+          .load()
+          .writeStream
+          .format("noop")
+          .option("checkpointLocation", Utils.createTempDir().getPath)
+          .start()
+        try query.processAllAvailable() finally query.stop()
+      }
+      checkError(
+        exception = findError(e, "NATIVE_DATA_SOURCE_ERROR"),
+        condition = "NATIVE_DATA_SOURCE_ERROR",
+        parameters = Map("action" -> "plan a scan of", "name" -> format, "msg" 
-> msg))
+      closedPath.isFile
+    }
+    // Spark asks for the limit before it records the streams to stop when the 
query stops, so the
+    // stream closes its reader itself when the limit fails.
+    assert(check("native_counter", Map("bad_read_limit" -> "kind"),
+      "default_read_limit returned the unknown kind of read limit 7."))
+    assert(check("native_counter", Map("bad_read_limit" -> "rows"),
+      "default_read_limit returned a maximum of 0 rows, but a micro-batch 
reads at least 1 row."))
+    // A data source must not declare a capability whose functions are NULL. 
Spark checks them
+    // before it creates the stream reader.
+    assert(!check("nrl_native_counter", Map("max_rows_per_trigger" -> "5"),
+      "The library does not implement default_read_limit."))
+  }
+
   nativeTest("ask for the initial offset once, and keep it in the checkpoint") 
{
     val session = newSession(defaultPackage)
     val checkpoint = Utils.createTempDir()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to