This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c165bce7e201 [SPARK-60732][SQL] Support admission control for
streaming reads of native data sources
c165bce7e201 is described below
commit c165bce7e20196f7686e5bbab5dca67ebb64b307
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:46:20 2026 +0900
[SPARK-60732][SQL] Support admission control for streaming reads of native
data sources
### What changes were proposed in this pull request?
Admission control for micro-batch streaming reads of native data sources,
appended within version 1 of `spark_datasource.h`. The guide of SPARK-60101
(#59424) lists it as a limitation. This PR is based on #59931 (SPARK-60723),
whose commits it includes until that PR is merged.
- **Capability and default limit.** A data source with the new capability
`SPARK_DS_ADMISSION_CONTROL` returns the limit on the data of each micro-batch
with `default_read_limit`, appended to `SparkDsLibrary`, in a size-prefixed
`SparkDsReadLimit`: `SPARK_DS_READ_MAX_ROWS` with a maximum number of rows, or
`SPARK_DS_READ_ALL_AVAILABLE`, typically from an option of the data source.
Spark asks once, when the query starts. It fails the query for an unknown kind
of limit, a maximum below [...]
- **Limited latest offsets.** `SparkDsLatestOffsetInfo` gets the limit as
`const SparkDsReadLimit* limit`, the same struct that `default_read_limit`
fills, so the limit has a single layout: the latest offset is then the end of
the data of the next micro-batch, within the limit. Spark passes the default
limit, or a limit of the kind `SPARK_DS_READ_ALL_AVAILABLE` for `Trigger.Once`,
for `Trigger.AvailableNow` when Spark falls back to a single micro-batch, and
for the other sources. When [...]
- **`Trigger.AvailableNow`.** Spark first asks for the latest offset
without a limit, as before, and then passes it as `run_end` with each limited
request, so the library never returns an offset past it. The run then reads the
data that was available when it started, in micro-batches within the limit.
Spark only compares offsets for equality, as strings, to find new data, and
does not order them, since they are defined by the library. A query that
restarts starts a new run, with a new [...]
- **Progress.** With a limit, the progress of the query reports the latest
offset without the limit, as the Kafka source does. So with a limit, Spark
calls `latest_offset` twice for some micro-batches, and the header says which
calls: with `Trigger.AvailableNow`, the call without a limit before the first
micro-batch is the end of the run, and with the other triggers, the call with
`SPARK_DS_READ_ALL_AVAILABLE` before each limited one is only for the progress.
Spark only asks with the [...]
On the Spark side, `ColumnarStreamReader` gets `defaultReadLimit()` and the
limit and the end of the run (`runEnd`) in `latestOffset`, which
`ColumnarMicroBatchStream` maps from `getDefaultReadLimit` and
`latestOffset(start, limit)`. The header, the FFM bindings, the C++ test
library and the guide are updated: admission control leaves the Limitations of
the guide, and the capability table lists it.
### Why are the changes needed?
Without a limit, every micro-batch reads all the data that is available,
which can make the first micro-batch of a large source, or a backlog after a
restart, too large. Spark's own sources offer such limits, for example
`maxOffsetsPerTrigger` for Kafka and `maxFilesPerTrigger` for files.
### Does this PR introduce _any_ user-facing change?
No for released versions: native data sources are new in the unreleased
4.4.0. Data sources without the new capability, and libraries built against the
header before these additions, keep reading all the available data.
### How was this patch tested?
New tests in `NativeDataSourceSuite`:
- "limit the rows of each micro-batch with admission control" checks
micro-batches of 5, 5 and 2 rows, and the latest offset without the limit that
the first one reports. It also checks `Trigger.AvailableNow` within the limit
and up to the end of the run, `Trigger.Once` reading all the rows, and a single
micro-batch without the capability, or with a default read limit of all the
available rows. The test library records the start, the limit and the end of
the run of each `latest_offset [...]
- "start a new run of Trigger.AvailableNow with a limit when the query
restarts" fails the second micro-batch of a run, and checks that the restarted
query runs it again up to its logged end without asking for a latest offset,
then reads up to a new end within the limit.
- "reject the read limits that a micro-batch cannot have" covers an unknown
kind of limit and a maximum of 0 rows, and checks that the stream reader is
closed when the query fails. It also uses a library whose table ends before
`default_read_limit`, as one built against the earlier header, that declares
the capability.
`NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass locally, with
Java 25 for the tests. The C++ example of the guide builds against the new
header.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59942 from HyukjinKwon/native-ds-admission-control.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/sql-native-data-sources.md | 49 ++++++--
.../datasources/v2/ffi/NativeLibrary.java | 63 +++++++++-
.../apache/spark/sql/datasource/spark_datasource.h | 114 ++++++++++++++---
.../v2/columnar/ColumnarDataSource.scala | 16 ++-
.../datasources/v2/columnar/ColumnarTable.scala | 51 +++++---
.../datasources/v2/ffi/NativeDataSource.scala | 61 +++++++--
.../native-datasource/test_native_datasource.cc | 70 +++++++++--
.../v2/columnar/ColumnarDataSourceSuite.scala | 2 +-
.../datasources/v2/ffi/NativeDataSourceSuite.scala | 138 ++++++++++++++++++++-
9 files changed, 489 insertions(+), 75 deletions(-)
diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index 049f49902e7c..9dbbfae59ea6 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -128,14 +128,37 @@ Spark calls the functions of the table as follows:
the checkpoint of the source in a `SparkDsStreamReaderInfo`. When the
checkpoint does not have the
initial offset of the source yet, Spark asks for it with `initial_offset`,
and keeps it in the
checkpoint, so a query that restarts before its first micro-batch is
committed starts from the
- same offset. For each micro-batch, or only once, before the first
micro-batch that it plans, with
- `Trigger.AvailableNow`, it asks for the `latest_offset`, with the offset
that the micro-batch
- starts after, plans the partitions with `stream_partitions`, which also
returns the state of the
- stream reader, and calls `read` on the executors. Offsets are defined by the
library, as compact
- JSON on a single line: Spark stores each offset as a line of the offset log
of the checkpoint,
- and rejects an offset that is not valid JSON or contains a line break, such
as pretty-printed
- JSON. Spark keeps what it needs in the checkpoint of the source under the
names that start with
- `_spark`, which the library must not use.
+ same offset. For each micro-batch, it asks for the `latest_offset`, with the
offset that the
+ micro-batch starts after, plans the partitions with `stream_partitions`,
which also returns the
+ state of the stream reader, and calls `read` on the executors. With
`Trigger.AvailableNow`, a run
+ reads the data that was available when it started: Spark asks for the
`latest_offset` without a
+ limit once, before the first micro-batch that it plans, and reads up to it.
Offsets are defined
+ by the library, as compact JSON on a single line: Spark stores each offset
as a line of the
+ offset log of the checkpoint, and rejects an offset that is not valid JSON
or contains a line
+ break, such as pretty-printed JSON. Spark finds new data by comparing the
latest offset with the
+ start as strings, so the same position must always have the same JSON. Spark
keeps what it needs
+ in the checkpoint of the source under the names that start with `_spark`,
which the library must
+ not use. A data source with the capability
+ `SPARK_DS_ADMISSION_CONTROL` limits the data of each micro-batch, for
example with an option for
+ a maximum number of rows: Spark asks for the limit once with
`default_read_limit`, which sets a
+ `SparkDsReadLimit`, and passes it to the `latest_offset` that ends each
micro-batch, in the
+ `limit` of its `SparkDsLatestOffsetInfo`, which then returns the end of the
data of the next
+ micro-batch within the limit. When there is data after the start, the latest
offset must be after
+ the start, even if the next unit of data, such as a file, is larger than the
limit: Spark sees no
+ new data otherwise. With `Trigger.AvailableNow`, the latest offset without a
limit that Spark asks
+ for first is the end of the run: Spark passes it as the `run_end` that the
limited latest offsets
+ must not go beyond, so that the run reads the data that was available when
it started in
+ micro-batches within the limit. A query that restarts starts a new run, with
a new end. So with a
+ limit, Spark calls `latest_offset` twice for some micro-batches: with
`Trigger.AvailableNow`, for
+ the first one, first without a limit for the end of the run, and with the
other triggers, for each
+ one, first with `SPARK_DS_READ_ALL_AVAILABLE` only for the progress of the
query. Spark does not
+ ask again with the limit when the latest offset without a limit is the same
as the start, such as
+ while a query waits for new data: there is no data to limit then. So a
library must not count the
+ calls as micro-batches. `Trigger.Once` ignores the limit, and so does
`Trigger.AvailableNow` when
+ another source of the query does not support it and
+ `spark.sql.streaming.triggerAvailableNowWrapper.enabled` is false, as by
default, where Spark
+ reads all the available data in a single micro-batch. Spark still asks for
the limit then, and
+ fails the query if it is not valid.
- **Writes**: on the driver, right before the tasks of the write (or of a
micro-batch) start, Spark
creates a writer with `create_writer`, which gets a `SparkDsWriteInfo` with
the mode of the write,
`SPARK_DS_WRITE_APPEND` or `SPARK_DS_WRITE_TRUNCATE`, and the ID of the
query of a streaming
@@ -185,6 +208,7 @@ data source must not declare a capability whose functions
are `NULL`:
| All | `create`, `close` | `last_error`, `schema` |
| `SPARK_DS_BATCH_READ` | `create_reader`, `partitions`, `read`,
`close_reader` | `push_limit`, `prune_columns` |
| `SPARK_DS_MICRO_BATCH_READ` | `create_stream_reader`, `initial_offset`,
`latest_offset`, `stream_partitions`, `read`, `close_stream_reader` |
`commit_offset` |
+| `SPARK_DS_ADMISSION_CONTROL`, with `SPARK_DS_MICRO_BATCH_READ` |
`default_read_limit` | |
| `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` | `create_writer`,
`commit`, `abort`, `close_writer`, `create_data_writer`, `write`,
`commit_data_writer`, `abort_data_writer`, `close_data_writer` | |
`SPARK_DS_BATCH_TRUNCATE` declares that a data source supports
`SPARK_DS_WRITE_TRUNCATE` in batch
@@ -216,8 +240,8 @@ task does not end.
### Compatibility
The function table, `SparkDsLibrary`, and the structs that Spark passes to the
library,
-`SparkDsHost`, `SparkDsTaskContext` and those whose names end with `Info`,
such as
-`SparkDsCreateInfo`, start with their size, so that they can grow. The version
of the interface,
+`SparkDsHost`, `SparkDsTaskContext`, `SparkDsReadLimit` and those whose names
end with `Info`, such
+as `SparkDsCreateInfo`, start with their size, so that they can grow. The
version of the interface,
1, is `SPARK_DS_ABI_VERSION`, which the table and the manifest of a package
declare. Within a
version:
@@ -226,7 +250,9 @@ version:
- Spark only reads the functions that fit in the size of the table of a
library, so a library built
against an earlier header keeps working with later releases of Spark, with
the functions that it
does not know as `NULL`. A library must only read the fields that fit in the
size of a struct
- that Spark passes, so that it also runs on earlier releases of Spark that
support its version;
+ that Spark passes, and only set those of a struct that it fills, such as the
`SparkDsReadLimit`
+ of `default_read_limit`, so that it also runs on earlier releases of Spark
that support its
+ version;
- Spark only uses a new capability, write mode or kind of read with a data
source that declares
it.
@@ -246,7 +272,6 @@ later as functions or fields appended to the structs of the
interface:
hint. Adaptive query execution can still broadcast it at run time, from the
size of a shuffle of
the scan;
- custom metrics, shown in the Spark UI;
-- admission control for streaming reads, such as a maximum number of rows for
each micro-batch;
- continuous processing (`Trigger.Continuous`): a continuous query that reads
from or writes to a
native data source fails when it starts;
- the pushdown of aggregates and of top-N queries (`ORDER BY` with `LIMIT`),
and reporting the
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index 4095f7c7f15a..f6424880f413 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -77,6 +77,7 @@ final class NativeLibrary {
static final long STREAMING_WRITE = 8;
static final long BATCH_TRUNCATE = 16;
static final long STREAMING_TRUNCATE = 32;
+ static final long ADMISSION_CONTROL = 64;
// The modes of a write, SPARK_DS_WRITE_APPEND and SPARK_DS_WRITE_TRUNCATE.
static final int WRITE_APPEND = 0;
@@ -86,6 +87,10 @@ final class NativeLibrary {
static final int READ_BATCH = 0;
static final int READ_MICRO_BATCH = 1;
+ // The kinds of read limits, SPARK_DS_READ_ALL_AVAILABLE and
SPARK_DS_READ_MAX_ROWS.
+ static final int READ_ALL_AVAILABLE = 0;
+ static final int READ_MAX_ROWS = 1;
+
// The C types of the parameters and the results of the functions.
private enum C { VOID, BOOL, I32, I64, PTR }
@@ -115,7 +120,8 @@ final class NativeLibrary {
WRITE("write", C.I32, C.PTR, C.PTR),
COMMIT_DATA_WRITER("commit_data_writer", C.I32, C.PTR, C.PTR, C.PTR,
C.PTR),
ABORT_DATA_WRITER("abort_data_writer", C.I32, C.PTR, C.PTR),
- CLOSE_DATA_WRITER("close_data_writer", C.VOID, C.PTR);
+ CLOSE_DATA_WRITER("close_data_writer", C.VOID, C.PTR),
+ DEFAULT_READ_LIMIT("default_read_limit", C.I32, C.PTR, C.PTR);
/** The name of the field of the function in the table. */
final String fieldName;
@@ -158,8 +164,13 @@ final class NativeLibrary {
private static final long COMMIT_OFFSET_INFO_END = 8;
private static final long STREAM_READER_INFO_SIZE = 16;
private static final long STREAM_READER_INFO_CHECKPOINT_LOCATION = 8;
- private static final long LATEST_OFFSET_INFO_SIZE = 16;
+ private static final long LATEST_OFFSET_INFO_SIZE = 32;
private static final long LATEST_OFFSET_INFO_START = 8;
+ private static final long LATEST_OFFSET_INFO_LIMIT = 16;
+ private static final long LATEST_OFFSET_INFO_RUN_END = 24;
+ private static final long READ_LIMIT_SIZE = 24;
+ private static final long READ_LIMIT_KIND = 8;
+ private static final long READ_LIMIT_MAX_ROWS = 16;
private static final long TASK_SIZE = 40;
private static final long TASK_STAGE_ID = 8;
private static final long TASK_STAGE_ATTEMPT_NUMBER = 12;
@@ -254,6 +265,12 @@ final class NativeLibrary {
layout.put("SparkDsLatestOffsetInfo", LATEST_OFFSET_INFO_SIZE);
layout.put("SparkDsLatestOffsetInfo.size", 0L);
layout.put("SparkDsLatestOffsetInfo.start", LATEST_OFFSET_INFO_START);
+ layout.put("SparkDsLatestOffsetInfo.limit", LATEST_OFFSET_INFO_LIMIT);
+ layout.put("SparkDsLatestOffsetInfo.run_end", LATEST_OFFSET_INFO_RUN_END);
+ layout.put("SparkDsReadLimit", READ_LIMIT_SIZE);
+ layout.put("SparkDsReadLimit.size", 0L);
+ layout.put("SparkDsReadLimit.kind", READ_LIMIT_KIND);
+ layout.put("SparkDsReadLimit.max_rows", READ_LIMIT_MAX_ROWS);
layout.put("SparkDsTaskContext", TASK_SIZE);
layout.put("SparkDsTaskContext.size", 0L);
layout.put("SparkDsTaskContext.stage_id", TASK_STAGE_ID);
@@ -866,22 +883,62 @@ final class NativeLibrary {
}
}
+ /** A limit on the data of a micro-batch, as in SparkDsReadLimit. */
+ static final class ReadLimitInfo {
+ /** {@link #READ_ALL_AVAILABLE} or {@link #READ_MAX_ROWS}, or what the
library set. */
+ final int kind;
+ /** The maximum number of rows with {@link #READ_MAX_ROWS}. */
+ final long maxRows;
+
+ ReadLimitInfo(int kind, long maxRows) {
+ this.kind = kind;
+ this.maxRows = maxRows;
+ }
+ }
+
/**
* Returns the latest offset.
*
* @param start the offset that the next micro-batch starts after
+ * @param limit the limit on the data after start
+ * @param runEnd the offset that a run of Trigger.AvailableNow reads up to,
or null
*/
- String latestOffset(long streamReader, String start) throws Throwable {
+ String latestOffset(long streamReader, String start, ReadLimitInfo limit,
String runEnd)
+ throws Throwable {
MethodHandle f = function(Function.LATEST_OFFSET);
try (Memory memory = new Memory(); Sink sink = new Sink()) {
+ long readLimit = memory.zeroed(READ_LIMIT_SIZE);
+ Platform.putLong(null, readLimit, READ_LIMIT_SIZE);
+ Platform.putInt(null, readLimit + READ_LIMIT_KIND, limit.kind);
+ Platform.putLong(null, readLimit + READ_LIMIT_MAX_ROWS, limit.maxRows);
long info = memory.zeroed(LATEST_OFFSET_INFO_SIZE);
Platform.putLong(null, info, LATEST_OFFSET_INFO_SIZE);
Platform.putLong(null, info + LATEST_OFFSET_INFO_START,
memory.string(start));
+ Platform.putLong(null, info + LATEST_OFFSET_INFO_LIMIT, readLimit);
+ Platform.putLong(null, info + LATEST_OFFSET_INFO_RUN_END,
+ runEnd == null ? 0 : memory.string(runEnd));
check((int) f.invokeExact(streamReader, info, Ffm.SINK, sink.id),
Function.LATEST_OFFSET);
return offset(Function.LATEST_OFFSET, sink);
}
}
+ /**
+ * Returns the default read limit of a stream reader, whose kind is {@link
#READ_ALL_AVAILABLE}
+ * if the library sets nothing.
+ */
+ ReadLimitInfo defaultReadLimit(long streamReader) throws Throwable {
+ MethodHandle f = function(Function.DEFAULT_READ_LIMIT);
+ try (Memory memory = new Memory()) {
+ long limit = memory.zeroed(READ_LIMIT_SIZE);
+ Platform.putLong(null, limit, READ_LIMIT_SIZE);
+ Platform.putInt(null, limit + READ_LIMIT_KIND, READ_ALL_AVAILABLE);
+ check((int) f.invokeExact(streamReader, limit),
Function.DEFAULT_READ_LIMIT);
+ return new ReadLimitInfo(
+ Platform.getInt(null, limit + READ_LIMIT_KIND),
+ Platform.getLong(null, limit + READ_LIMIT_MAX_ROWS));
+ }
+ }
+
/** Returns the offset that a function passed to the sink, which it must
pass. */
private static String offset(Function function, Sink sink) throws Throwable {
byte[] offset = single(sink, function);
diff --git
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index a8d7785ee1b1..b716b6469c59 100644
---
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -64,15 +64,28 @@
* - Streaming reads. On the driver, Spark creates a stream reader with
create_stream_reader, with
* the columns to read and the location of the checkpoint of the source, and
keeps it until the
* streaming query stops. When the checkpoint does not have the initial
offset of the source yet,
- * Spark asks for it with initial_offset, and keeps it in the checkpoint.
For each micro-batch, or
- * only before the first one with Trigger.AvailableNow, it asks for the
latest_offset, with the
- * offset that the micro-batch starts after, plans the partitions with
stream_partitions, which
- * also returns the state of the stream reader, then calls read on the
executors. Offsets are
- * defined by the library, as compact JSON on a single line: Spark stores
each offset as a line of
- * the offset log of the checkpoint, so it rejects an offset that is not
valid JSON or contains a
- * line break. When a streaming query restarts, from its checkpoint, Spark
creates a new stream
- * reader and may call stream_partitions again for the last range that it
planned before, so the
- * partitions of a range must read the same data each time.
+ * Spark asks for it with initial_offset, and keeps it in the checkpoint.
For each micro-batch, it
+ * asks for the latest_offset, with the offset that the micro-batch starts
after, plans the
+ * partitions with stream_partitions, which also returns the state of the
stream reader, then
+ * calls read on the executors. With Trigger.AvailableNow, a run reads the
data that was
+ * available when it started: Spark asks for the latest_offset without a
limit once, before the
+ * first micro-batch that it plans, as the end of the run, and reads up to
it. A data source with
+ * the capability SPARK_DS_ADMISSION_CONTROL limits the data of each
micro-batch: Spark asks for
+ * the limit once with default_read_limit, and passes it to the
latest_offset that ends each
+ * micro-batch, also with Trigger.AvailableNow, whose run then reads up to
its end in several
+ * micro-batches. So with a limit, Spark calls latest_offset twice for some
micro-batches: with
+ * Trigger.AvailableNow, for the first one, first without a limit for the
end of the run, and
+ * with the other triggers, for each one, first with
SPARK_DS_READ_ALL_AVAILABLE only for the
+ * progress of the query. Spark does not ask again with the limit when the
latest offset without
+ * a limit is the same as the start, such as while a query waits for new
data: there is no data
+ * to limit then. So a library must not count the calls of latest_offset as
+ * micro-batches, or reserve data in them. Offsets are defined by the
library, as compact JSON on
+ * a single line: Spark stores each offset as a line of the offset log of
the checkpoint, so it
+ * rejects an offset that is not valid JSON or contains a line break. Spark
finds new data by
+ * comparing the latest offset with the start as strings, so the same
position must always have
+ * the same JSON. When a streaming query restarts, from its checkpoint,
Spark creates a new
+ * stream reader and may call stream_partitions again for the last range
that it planned before,
+ * so the partitions of a range must read the same data each time.
* - Writes. On the driver, right before it starts the tasks of the write (or
of a micro-batch),
* Spark creates a writer with create_writer, which also returns the state
of the writer. If Spark
* rejects what create_writer returned, such as a state set more than once,
it aborts the writer,
@@ -179,11 +192,12 @@
* - Options. Option keys are passed in lower case, because Spark matches them
* case-insensitively.
* - Optional functions. A library only sets the functions of the operations
it supports in its
- * table, and leaves the others NULL. A data source must not declare a
capability whose
- * functions are NULL: Spark then fails the operation. The pushdown
functions are optional:
- * without them, Spark pushes nothing down. So are schema, without which the
user must specify
- * the schema, commit_offset and last_error. A reader or a writer without a
state does not call
- * the set_state sink of partitions, stream_partitions or create_writer.
+ * table, and leaves the others NULL. A data source must not declare a
capability whose functions
+ * are NULL: Spark then fails the operation, as for default_read_limit with
the capability
+ * SPARK_DS_ADMISSION_CONTROL. The pushdown functions are optional: without
them, Spark pushes
+ * nothing down. So are schema, without which the user must specify the
schema, commit_offset and
+ * last_error. A reader or a writer without a state does not call the
set_state sink of
+ * partitions, stream_partitions or create_writer.
* - Threads. Spark calls the library from multiple threads, and calls on
different handles may
* run concurrently. Calls on the same handle never run concurrently. A
handle may be released
* from a thread other than the threads that used it, such as a JVM Cleaner
thread when Spark
@@ -201,7 +215,9 @@
* never change, nor do the semantics of the existing functions;
* - Spark only reads the functions that fit in the size of the table of a
library, so a library
* built against an earlier header keeps working, with the functions that it
does not know as
- * NULL. A library must only read the fields that fit in the size of a
struct that Spark passes;
+ * NULL. A library must only read the fields that fit in the size of a
struct that Spark passes,
+ * and only set those of a struct that it fills, such as the
SparkDsReadLimit of
+ * default_read_limit;
* - Spark only uses a new capability, write mode or kind of read with a data
source that declares
* it.
* An incompatible change uses a new SPARK_DS_ABI_VERSION: spark_ds_init gets
the newest version
@@ -377,6 +393,25 @@ typedef struct SparkDsInitialOffsetInfo {
int64_t size;
} SparkDsInitialOffsetInfo;
+/* The kinds of the limits on the data of a micro-batch, see SparkDsReadLimit.
*/
+#define SPARK_DS_READ_ALL_AVAILABLE 0
+#define SPARK_DS_READ_MAX_ROWS 1
+
+/*
+ * A limit on the data of each micro-batch of a streaming read, which
default_read_limit returns
+ * for a data source with the capability SPARK_DS_ADMISSION_CONTROL, and
latest_offset gets in its
+ * SparkDsLatestOffsetInfo. In default_read_limit, the library sets the fields
that fit in the size
+ * that Spark set.
+ */
+typedef struct SparkDsReadLimit {
+ /* The size of this struct in the header that Spark was built with, as in
SparkDsHost. */
+ int64_t size;
+ /* SPARK_DS_READ_ALL_AVAILABLE, as Spark initializes it, or
SPARK_DS_READ_MAX_ROWS. */
+ int32_t kind;
+ /* With SPARK_DS_READ_MAX_ROWS, the maximum number of rows of a micro-batch,
at least 1. */
+ int64_t max_rows;
+} SparkDsReadLimit;
+
/* A request for the latest offset of a stream reader, as latest_offset gets
it. */
typedef struct SparkDsLatestOffsetInfo {
/* The size of this struct in the header that Spark was built with, as in
SparkDsHost. */
@@ -384,10 +419,31 @@ typedef struct SparkDsLatestOffsetInfo {
/*
* The offset, as JSON, that the next micro-batch starts after: the end of
the last micro-batch,
* or the initial offset before the first one, for every trigger. With
Trigger.AvailableNow,
- * Spark asks for the latest offset only once, before the first micro-batch
that it plans, and
- * reads up to it.
+ * Spark asks for the latest offset without a limit once, before the first
micro-batch that it
+ * plans, and reads up to it: in a single micro-batch, or with a limit, in
micro-batches of
+ * which it asks for each latest offset with that offset as the end.
*/
const char* start;
+ /*
+ * The limit on the data after start, never NULL: the default read limit of
a data source with
+ * the capability SPARK_DS_ADMISSION_CONTROL, or a limit of the kind
SPARK_DS_READ_ALL_AVAILABLE,
+ * as always for the other data sources, for Trigger.Once, and for
Trigger.AvailableNow when Spark
+ * runs a single micro-batch because another source of the query does not
support it. With a
+ * limit, the latest offset is the end of the data of the next micro-batch,
within the limit.
+ * When there is data after start, the latest offset must be after start,
even if the next unit
+ * of data that an offset can end, such as a file, is larger than the limit:
Spark sees no new
+ * data otherwise, and never reads further.
+ */
+ const SparkDsReadLimit* limit;
+ /*
+ * With Trigger.AvailableNow and a limit of the kind SPARK_DS_READ_MAX_ROWS,
the end of the run:
+ * the offset that the run reads up to, which the latest offset must not go
beyond, so that the
+ * run reads the data that was available when it started; NULL otherwise, as
in the call without
+ * a limit that asks for the end of the run. It is not the end of the next
micro-batch. A query
+ * that restarts starts a new run, with a new end: Spark first runs again
the micro-batch that it
+ * planned but did not commit, if any, up to the end in its offset log,
without asking.
+ */
+ const char* run_end;
} SparkDsLatestOffsetInfo;
/* A request for the partitions of a micro-batch, as stream_partitions gets
it. */
@@ -553,6 +609,7 @@ typedef struct SparkDsFinishDataWriterInfo {
#define SPARK_DS_STREAMING_WRITE 8 /* streaming writes, with create_writer
*/
#define SPARK_DS_BATCH_TRUNCATE 16 /* batch writes with the mode
SPARK_DS_WRITE_TRUNCATE */
#define SPARK_DS_STREAMING_TRUNCATE 32 /* streaming writes with the mode
SPARK_DS_WRITE_TRUNCATE */
+#define SPARK_DS_ADMISSION_CONTROL 64 /* micro-batch reads with read limits
(default_read_limit) */
/*
* The functions of a library, which spark_ds_init returns. A library sets the
functions of the
@@ -686,9 +743,10 @@ typedef struct SparkDsLibrary {
void* sink);
/*
- * Returns the most recent offset available.
+ * Returns the most recent offset available after start, within the limit
and up to the end of
+ * the run in its info. Spark may call it more than once for a micro-batch,
see Streaming reads.
*
- * info: the offset that the next micro-batch starts after
+ * info: the offset that the next micro-batch starts after, the limit, and
the end of the run
* set_offset: called once with the offset, as compact JSON on a single line
*/
int (*latest_offset)(void* reader, const SparkDsLatestOffsetInfo* info,
SparkDsSink set_offset,
@@ -811,6 +869,24 @@ typedef struct SparkDsLibrary {
/* Releases a data writer handle, after its commit or abort. */
void (*close_data_writer)(void* data_writer);
+
+ /*
-------------------------------------------------------------------------------------------
+ * Admission control (driver)
+ *
-------------------------------------------------------------------------------------------
*/
+
+ /*
+ * Returns the limit on the data of each micro-batch of a stream reader,
typically from the
+ * options of the data source, such as a maximum number of rows. Spark asks
once, when the
+ * streaming query starts, for a data source with the capability
SPARK_DS_ADMISSION_CONTROL, and
+ * passes the limit to the latest_offset that ends each micro-batch, except
with Trigger.Once,
+ * and with Trigger.AvailableNow when another source of the query does not
support it and
+ * spark.sql.streaming.triggerAvailableNowWrapper.enabled is false, as by
default, where Spark
+ * reads all the available data in a single micro-batch. Spark asks for the
limit then too, and
+ * fails the query if it is not valid, though it does not pass it.
+ *
+ * limit: the limit to set, initialized by Spark with its size and
SPARK_DS_READ_ALL_AVAILABLE
+ */
+ int (*default_read_limit)(void* reader, SparkDsReadLimit* limit);
} SparkDsLibrary;
/*
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index a98cb66a485a..13f7ba56c879 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -21,6 +21,7 @@ import java.util
import org.apache.spark.sql.connector.catalog.TableCapability
import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ,
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
import org.apache.spark.sql.connector.read.{InputPartition,
PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming.ReadLimit
import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -123,12 +124,23 @@ trait ColumnarStreamReader {
def initialOffset(): String
/**
- * Returns the latest offset available.
+ * Returns the limit on the data of each micro-batch, which Spark asks for
once, when the query
+ * starts: `ReadLimit.allAvailable()` by default, or a `ReadLimit.maxRows`.
+ */
+ def defaultReadLimit(): ReadLimit = ReadLimit.allAvailable()
+
+ /**
+ * Returns the latest offset available, within a limit.
*
* @param start the offset that the next micro-batch starts after: the end
of the last
* micro-batch, or the initial offset before the first one
+ * @param limit `ReadLimit.allAvailable()`, or the default read limit: the
latest offset is then
+ * the end of the data of the next micro-batch. Spark only
passes the default read
+ * limit when the latest offset without a limit is not `start`
+ * @param runEnd with `Trigger.AvailableNow` and a limit, the offset that
the run reads up to,
+ * which the latest offset does not go beyond
*/
- def latestOffset(start: String): String
+ def latestOffset(start: String, limit: ReadLimit, runEnd: Option[String]):
String
/**
* Plans the partitions that read the data after `start` up to and including
`end`. Spark plans
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
index 6b4d8c0db3dc..81cb1e0e9ac5 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
@@ -224,12 +224,12 @@ class ColumnarBatchScan(planner: ColumnarBatchPlanner)
extends Batch {
case class ColumnarStreamOffset(json: String) extends Offset
/**
- * The micro-batch stream of a [[ColumnarDataSource]]. It supports admission
control only so that
- * the reader gets the offset that each micro-batch starts after, the end of
the last one, for
- * every trigger: its default read limit reads all the available data. With
- * `Trigger.AvailableNow`, Spark runs micro-batches up to the first latest
offset that Spark asks
- * for, rather than wrapping the stream in one that asks for the latest offset
from the initial
- * offset.
+ * The micro-batch stream of a [[ColumnarDataSource]]. The reader gets the
offset that each
+ * micro-batch starts after, the end of the last one, for every trigger, and
the limit on the data
+ * of a micro-batch, its default read limit, all the available data unless it
limits it. With
+ * `Trigger.AvailableNow`, Spark runs micro-batches up to the first latest
offset without a limit
+ * that Spark asks for, rather than wrapping the stream in one that asks for
the latest offset from
+ * the initial offset: in one micro-batch, or within the limit in several.
*
* @param checkpointLocation the location of the checkpoint of the source,
where the stream keeps
* the initial offset, under
`ColumnarMicroBatchStream.InitialOffsetLog`
@@ -247,11 +247,22 @@ class ColumnarMicroBatchStream(reader:
ColumnarStreamReader, checkpointLocation:
@GuardedBy("this")
private var plannedPartitions: Try[Array[InputPartition]] = _
- // The offset that the reader returned last, which the progress of the query
reports.
+ // The latest offset without a limit that the reader returned last, which
the progress of the
+ // query reports.
@volatile private var lastLatestOffset: Offset = _
+ // The limit on the data of each micro-batch, asked once, when the query
starts. Spark asks for
+ // it before it records the streams to stop when the query stops, so a
stream whose limit fails
+ // stops its reader itself.
+ private lazy val readLimit = try {
+ reader.defaultReadLimit()
+ } catch {
+ case e: Throwable =>
+ try reader.stop() catch { case stopError: Throwable =>
e.addSuppressed(stopError) }
+ throw e
+ }
// Whether the trigger is Trigger.AvailableNow.
@volatile private var availableNow = false
- // The end of the run with Trigger.AvailableNow: the first latest offset
asked.
+ // The end of the run with Trigger.AvailableNow: the first latest offset
without a limit asked.
@volatile private var availableNowEnd: Offset = _
// Spark asks for the initial offset for the start of the first latest
offset, and again to plan
// the first micro-batch, which both get the same offset. The reader is only
asked when the
@@ -275,17 +286,27 @@ class ColumnarMicroBatchStream(reader:
ColumnarStreamReader, checkpointLocation:
"latestOffset(Offset, ReadLimit) should be called instead of
latestOffset().")
}
+ override def getDefaultReadLimit: ReadLimit = readLimit
+
override def latestOffset(start: Offset, limit: ReadLimit): Offset = {
- val latest = if (availableNow && availableNowEnd != null) {
- availableNowEnd
+ def ask(limit: ReadLimit, runEnd: Option[Offset]): Offset =
+ ColumnarStreamOffset(reader.latestOffset(start.json(), limit,
runEnd.map(_.json())))
+ // The latest offset without a limit, which the progress reports: with
Trigger.AvailableNow,
+ // the end of the run, asked once, and otherwise asked for each
micro-batch.
+ val latest = if (!availableNow) {
+ ask(ReadLimit.allAvailable(), None)
} else {
- ColumnarStreamOffset(reader.latestOffset(start.json()))
- }
- if (availableNow) {
- availableNowEnd = latest
+ if (availableNowEnd == null) availableNowEnd =
ask(ReadLimit.allAvailable(), None)
+ availableNowEnd
}
lastLatestOffset = latest
- latest
+ // When the latest offset without a limit is the start, as Spark compares
offsets, there is no
+ // data after the start to limit, so the reader is not asked again with
the limit.
+ if (limit == ReadLimit.allAvailable() || latest.json() == start.json()) {
+ latest
+ } else {
+ ask(limit, Option(availableNowEnd))
+ }
}
override def prepareForTriggerAvailableNow(): Unit = {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 40e1f2d44d49..e0530d283a9f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -44,6 +44,7 @@ import org.apache.spark.sql.classic.SparkSession
import org.apache.spark.sql.connector.catalog.TableCapability
import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ,
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader,
PartitionReaderFactory, SupportsArrowStreamRead}
+import org.apache.spark.sql.connector.read.streaming.{ReadAllAvailable,
ReadLimit, ReadMaxRows}
import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
import
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchPlanner,
ColumnarDataSource, ColumnarOnlyPartitionReaderFactory, ColumnarPlannedBatch,
ColumnarStreamReader, ColumnarStreamWriter, ColumnarWriter}
@@ -128,15 +129,20 @@ class NativeDataSource(
NativeArrow.checkSupported(schema, name, "plan a scan of")
val executorLocation = NativeDataSourceRegistry.distribute(location)
val library = NativeLibraries.get(location)
- val handle = create(library, "plan a scan of",
NativeLibrary.MICRO_BATCH_READ,
- "DATA_SOURCE_MICRO_BATCH_SCAN_NOT_SUPPORTED",
NativeDataSource.MicroBatchReadFunctions) {
- dataSource =>
- NativeArrow.exportSchema(schema, timeZoneId) { address =>
- library.createStreamReader(dataSource.handle, address,
checkpointLocation)
- }
+ val (handle, admissionControl) = create(library, "plan a scan of",
+ NativeLibrary.MICRO_BATCH_READ,
"DATA_SOURCE_MICRO_BATCH_SCAN_NOT_SUPPORTED",
+ NativeDataSource.MicroBatchReadFunctions) { dataSource =>
+ val admissionControl = (dataSource.capabilities &
NativeLibrary.ADMISSION_CONTROL) != 0
+ if (admissionControl) {
+ requireFunctions(library, "plan a scan of",
Seq(NativeLibrary.Function.DEFAULT_READ_LIMIT))
+ }
+ val handle = NativeArrow.exportSchema(schema, timeZoneId) { address =>
+ library.createStreamReader(dataSource.handle, address,
checkpointLocation)
+ }
+ (handle, admissionControl)
}
new NativeDataSourceStreamReader(executorLocation, name, schema,
timeZoneId,
- new NativeHandle(handle, library.closeStreamReader))
+ new NativeHandle(handle, library.closeStreamReader), admissionControl)
}
override def writer(schema: StructType, overwrite: Boolean): ColumnarWriter
= {
@@ -333,7 +339,8 @@ class NativeDataSourceStreamReader(
name: String,
schema: StructType,
timeZoneId: String,
- handle: NativeHandle)
+ handle: NativeHandle,
+ admissionControl: Boolean)
extends ColumnarStreamReader {
import NativeCalls._
@@ -357,9 +364,41 @@ class NativeDataSourceStreamReader(
checkOffset("initial_offset", call(name, "plan a scan
of")(library.initialOffset(handle.get)))
}
- override def latestOffset(start: String): String = synchronized {
- checkOffset("latest_offset",
- call(name, "plan a scan of")(library.latestOffset(handle.get, start)))
+ /** The limit that the library returns, with the capability
SPARK_DS_ADMISSION_CONTROL. */
+ override def defaultReadLimit(): ReadLimit = synchronized {
+ if (!admissionControl) {
+ ReadLimit.allAvailable()
+ } else {
+ val limit = call(name, "plan a scan
of")(library.defaultReadLimit(handle.get))
+ val (kind, maxRows) = (limit.kind, limit.maxRows)
+ kind match {
+ case NativeLibrary.READ_ALL_AVAILABLE => ReadLimit.allAvailable()
+ case NativeLibrary.READ_MAX_ROWS if maxRows >= 1 =>
ReadLimit.maxRows(maxRows)
+ case NativeLibrary.READ_MAX_ROWS =>
+ throw error(name, "plan a scan of", s"default_read_limit returned a
maximum of " +
+ s"$maxRows rows, but a micro-batch reads at least 1 row.")
+ case _ =>
+ throw error(name, "plan a scan of", s"default_read_limit returned
the unknown kind " +
+ s"of read limit $kind.")
+ }
+ }
+ }
+
+ override def latestOffset(
+ start: String,
+ limit: ReadLimit,
+ runEnd: Option[String]): String = synchronized {
+ val info = limit match {
+ case _: ReadAllAvailable =>
+ new NativeLibrary.ReadLimitInfo(NativeLibrary.READ_ALL_AVAILABLE, 0L)
+ case maxRows: ReadMaxRows =>
+ new NativeLibrary.ReadLimitInfo(NativeLibrary.READ_MAX_ROWS,
maxRows.maxRows())
+ case other => throw SparkException.internalError(
+ s"A native data source does not support the read limit $other.")
+ }
+ checkOffset("latest_offset", call(name, "plan a scan of") {
+ library.latestOffset(handle.get, start, info, runEnd.orNull)
+ })
}
/**
diff --git
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index 87551a695b31..de2f5b76a57d 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -61,15 +61,20 @@
// "closed_path", closing the stream reader creates that
file. With the option
// "checkpoint_path", creating the stream reader writes its
checkpoint location to
// that file, and with "starts_path", each call to
latest_offset appends its start
-// offset to that file, as a line. With
"initial_offsets_path", each call to
-// initial_offset appends a line to that file, and returns
the number of calls
-// before it as the offset, as an initial offset that
depends on when it is asked
-// does. With "columns_path", creating the stream reader
writes the columns to
-// read to that file, and with "ranges_path", each call to
stream_partitions
-// appends the start and the end offsets that it got to that
file, as a line.
-// The partitions and the state that stream_partitions
returns have the number of
-// the call, which read checks. With the option
"state_padding", the state has
-// that many more bytes.
+// offset to that file, as a line. With the option
"max_rows_per_trigger", it
+// declares the capability SPARK_DS_ADMISSION_CONTROL, and
its default read limit
+// is that many rows, or with "bad_read_limit", an unknown
"kind" of limit or a
+// maximum of 0 "rows". With "limits_path", each call to
latest_offset appends its
+// start, its limit, "all" or the maximum number of rows,
and its end of the run
+// or "-" to that file, as a line, such as "0 5 12". With
"initial_offsets_path",
+// each call to initial_offset appends a line to that file,
and returns the number
+// of calls before it as the offset, as an initial offset
that depends on when it
+// is asked does. With "columns_path", creating the stream
reader writes the
+// columns to read to that file, and with "ranges_path",
each call to
+// stream_partitions appends the start and the end offsets
that it got to that
+// file, as a line. The partitions and the state that
stream_partitions returns
+// have the number of the call, which read checks. With the
option
+// "state_padding", the state has that many more bytes.
//
// native_range supports batch reads, native_counter micro-batch streaming
reads and native_sink
// batch and streaming writes, which may replace the existing data. With the
option
@@ -1130,7 +1135,11 @@ uint64_t capabilitiesOf(const DataSource& source) {
if (!declared.empty()) return std::stoull(declared);
if (source.name == name("native_range")) return SPARK_DS_BATCH_READ;
#ifndef READ_ONLY
- if (source.name == name("native_counter")) return SPARK_DS_MICRO_BATCH_READ;
+ if (source.name == name("native_counter")) {
+ bool limited = !get(source.options, "max_rows_per_trigger", "").empty() ||
+ !get(source.options, "bad_read_limit", "").empty();
+ return SPARK_DS_MICRO_BATCH_READ | (limited ? SPARK_DS_ADMISSION_CONTROL :
0);
+ }
if (source.name == name("native_sink")) {
return SPARK_DS_BATCH_WRITE | SPARK_DS_STREAMING_WRITE |
SPARK_DS_BATCH_TRUNCATE |
SPARK_DS_STREAMING_TRUNCATE;
@@ -1253,7 +1262,8 @@ int createStreamReader(void* handle, struct ArrowSchema*
schema,
reader->state["batch_size"] = get(source->options, "batch_size", "1000");
for (const char* option :
{"commit_path", "bad_offset", "bad_initial_offset", "closed_path",
"starts_path",
- "initial_offsets_path", "ranges_path", "states", "stateless",
"task_path", "fail"}) {
+ "initial_offsets_path", "ranges_path", "states", "stateless",
"task_path",
+ "max_rows_per_trigger", "bad_read_limit", "limits_path", "fail"}) {
reader->state[option] = get(source->options, option, "");
}
std::string padding = get(source->options, "state_padding", "");
@@ -1379,6 +1389,26 @@ int latestOffset(void* handle, const
SparkDsLatestOffsetInfo* info, SparkDsSink
auto start =
static_cast<int64_t>(JsonParser(info->start).parse()["offset"].number);
int64_t latest =
std::min(getLong(state, "max_offset", 10), start + getLong(state,
"step", 10));
+ // Within the limit, and up to the end of a run of Trigger.AvailableNow.
+ if (info->limit == nullptr) throw std::runtime_error("no read limit");
+ checkSize("SparkDsReadLimit", info->limit->size, sizeof(SparkDsReadLimit));
+ std::string limit = "all";
+ if (info->limit->kind == SPARK_DS_READ_MAX_ROWS) {
+ latest = std::min(latest, start + info->limit->max_rows);
+ limit = std::to_string(info->limit->max_rows);
+ } else if (info->limit->kind != SPARK_DS_READ_ALL_AVAILABLE) {
+ throw std::runtime_error("unknown kind of read limit " +
std::to_string(info->limit->kind));
+ }
+ std::string run_end = "-";
+ if (info->run_end != nullptr) {
+ auto end =
static_cast<int64_t>(JsonParser(info->run_end).parse()["offset"].number);
+ latest = std::min(latest, end);
+ run_end = std::to_string(end);
+ }
+ std::string limits_path = get(state, "limits_path", "");
+ if (!limits_path.empty()) {
+ appendLine(limits_path, std::to_string(start) + " " + limit + " " +
run_end);
+ }
sendOffset(state, "bad_offset", set_offset, sink, latest);
});
}
@@ -1773,6 +1803,23 @@ int abortDataWriter(void* handle, const
SparkDsFinishDataWriterInfo* info) {
});
}
+int defaultReadLimit(void* handle, SparkDsReadLimit* limit) {
+ return guarded([&] {
+ checkSize("SparkDsReadLimit", limit->size, sizeof(SparkDsReadLimit));
+ const Options& state = fromHandle<Reader>(handle)->state;
+ std::string bad = get(state, "bad_read_limit", "");
+ if (bad == "kind") {
+ limit->kind = 7;
+ } else if (bad == "rows") {
+ limit->kind = SPARK_DS_READ_MAX_ROWS;
+ limit->max_rows = 0;
+ } else if (!get(state, "max_rows_per_trigger", "").empty()) {
+ limit->kind = SPARK_DS_READ_MAX_ROWS;
+ limit->max_rows = getLong(state, "max_rows_per_trigger", 0);
+ }
+ });
+}
+
void closeDataWriter(void* handle) {
if (usedAfterClose(handle)) return;
std::unique_ptr<DataWriter> writer(fromHandle<DataWriter>(handle));
@@ -1825,6 +1872,7 @@ void closeDataWriter(void* handle) {
table.commit_data_writer = commitDataWriter;
table.abort_data_writer = abortDataWriter;
table.close_data_writer = closeDataWriter;
+ table.default_read_limit = defaultReadLimit;
#endif
return table;
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 92ef0d87de92..bfcb536aeb0d 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -1101,7 +1101,7 @@ class TestRangeStreamReader(end: Long, schema:
StructType, table: String)
override def initialOffset(): String = "{\"offset\":0}"
- override def latestOffset(start: String): String = {
+ override def latestOffset(start: String, limit: ReadLimit, runEnd:
Option[String]): String = {
TestRangeStreamReader.starts.compute(table, (_, old) =>
Option(old).getOrElse(Nil) :+ start)
s"""{"offset":$end}"""
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 1928bbac3722..72d1f09e1b93 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -40,7 +40,7 @@ import org.scalatest.time.SpanSugar._
import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException,
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable,
SparkUnsupportedOperationException, TaskContext, TaskKilledException, TestUtils}
import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
import org.apache.spark.broadcast.Broadcast
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils,
QueryTest, Row}
+import org.apache.spark.sql.{AnalysisException, Dataset,
IntegratedUDFTestUtils, QueryTest, Row}
import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
import org.apache.spark.sql.classic.{DataFrame, SparkSession}
import org.apache.spark.sql.connector.catalog.SupportsRead
@@ -91,6 +91,9 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
compile("test_native_datasource_ne", "-DNO_LAST_ERROR",
"-DNAME_PREFIX=\"ne_\"")
private lazy val smallTableLibrary = compile("test_native_datasource_small",
"-DTABLE_SIZE=offsetof(SparkDsLibrary,read)", "-DNAME_PREFIX=\"small_\"")
+ // A library built against the header before default_read_limit, whose table
ends before it.
+ private lazy val noReadLimitLibrary = compile("test_native_datasource_nrl",
+ "-DTABLE_SIZE=offsetof(SparkDsLibrary,default_read_limit)",
"-DNAME_PREFIX=\"nrl_\"")
private lazy val failingInitLibrary =
compile("test_native_datasource_if", "-DINIT_FAILURE=1",
"-DNAME_PREFIX=\"if_\"")
private lazy val noTableLibrary =
@@ -1042,6 +1045,139 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
assert(written.map(_.toLong).sorted.toSeq == (0L until 10L))
}
+ nativeTest("limit the rows of each micro-batch with admission control") {
+ val session = newSession(defaultPackage)
+ // Runs the query with a trigger until it read the rows [0, 12), and
returns the number of rows
+ // of each micro-batch, the latest offset that the first one reported, and
the start, the limit
+ // and the end of the run of each call of latest_offset, as "limits_path"
records them.
+ def run(
+ trigger: Option[Trigger],
+ options: (String, String)*): (Seq[Long], String, Seq[String]) = {
+ val limitsPath = new File(Utils.createTempDir(), "limits")
+ val writer =
session.readStream.format("native_counter").option("max_offset", "12")
+ .option("limits_path", limitsPath.getPath)
+ .options(options.toMap).load()
+ .writeStream
+ .format("noop")
+ .option("checkpointLocation", Utils.createTempDir().getPath)
+ val query = trigger.map(writer.trigger).getOrElse(writer).start()
+ try {
+ if (trigger.isDefined) query.awaitTermination() else
query.processAllAvailable()
+ } finally {
+ query.stop()
+ }
+ val batches = query.recentProgress.filter(_.numInputRows > 0).toSeq
+ (batches.map(_.numInputRows), batches.head.sources.head.latestOffset,
+ Files.readAllLines(limitsPath.toPath).asScala.toSeq)
+ }
+ // Each micro-batch reads at most 5 rows, all available from the start,
while the progress
+ // reports the latest offset without the limit, which Spark asks for
first. Spark only asks
+ // with the limit while there is data after the start.
+ val (rows, latest, calls) = run(None, "max_rows_per_trigger" -> "5")
+ assert((rows, latest) == ((Seq(5L, 5L, 2L), """{"offset":12}""")))
+ assert(calls.take(6) == Seq("0 all -", "0 5 -", "5 all -", "5 5 -", "10
all -", "10 5 -"))
+ assert(calls.drop(6).nonEmpty && calls.drop(6).forall(_ == "12 all -"),
calls)
+ // A run of Trigger.AvailableNow reads the rows available when it starts,
within the limit:
+ // Spark asks for the end of the run once, without a limit, and passes it
with the limit.
+ assert(run(Some(Trigger.AvailableNow()), "max_rows_per_trigger" -> "5") ==
+ ((Seq(5L, 5L, 2L), """{"offset":12}""", Seq("0 all -", "0 5 12", "5 5
12", "10 5 12"))))
+ // Only the rows [0, 6) are available when the run starts, and
native_counter makes 6 more
+ // rows available after each start: the run stops at the end that it got
when it started,
+ // without asking again once it reached it.
+ assert(run(Some(Trigger.AvailableNow()), "max_rows_per_trigger" -> "5",
"step" -> "6") ==
+ ((Seq(5L, 1L), """{"offset":6}""", Seq("0 all -", "0 5 6", "5 5 6"))))
+ // Trigger.Once ignores the limit, and reads all the available rows in one
micro-batch.
+ assert(run(Some(Trigger.Once()), "max_rows_per_trigger" -> "5") ==
+ ((Seq(12L), """{"offset":12}""", Seq("0 all -"))))
+ // Without the capability, Spark asks for all the available rows, once for
each micro-batch.
+ val (unlimitedRows, _, unlimitedCalls) = run(None, "max_rows_per_trigger"
-> "5",
+ "capabilities" -> NativeLibrary.MICRO_BATCH_READ.toString)
+ assert(unlimitedRows == Seq(12L))
+ assert(unlimitedCalls.head == "0 all -" && unlimitedCalls.tail.forall(_ ==
"12 all -"),
+ unlimitedCalls)
+ // So it does with the capability, when the default read limit is all the
available rows.
+ val limited = NativeLibrary.MICRO_BATCH_READ |
NativeLibrary.ADMISSION_CONTROL
+ assert(run(None, "capabilities" -> limited.toString)._1 == Seq(12L))
+ }
+
+ nativeTest("start a new run of Trigger.AvailableNow with a limit when the
query restarts") {
+ val session = newSession(defaultPackage)
+ val checkpoint = Utils.createTempDir()
+ val dir = Utils.createTempDir()
+ // The rows of each micro-batch that succeeded, with its ID.
+ val batches = ArrayBuffer.empty[(Long, Seq[Long])]
+ // Runs the query, which records the start, the limit and the end of the
run of each call of
+ // latest_offset in the file `limits`.
+ def run(failingBatch: Long, limits: String): Unit = {
+ val writeBatch: (Dataset[Row], Long) => Unit = { (df, id) =>
+ if (id == failingBatch) throw new IllegalStateException(s"micro-batch
$id fails")
+ batches += id -> df.collect().map(_.getLong(0)).toSeq.sorted
+ }
+ // A run from 0 ends at 6, since native_counter makes 6 more rows
available after each start.
+ val query = session.readStream.format("native_counter")
+ .option("max_offset", "12").option("step",
"6").option("max_rows_per_trigger", "5")
+ .option("limits_path", new File(dir, limits).getPath)
+ .load()
+ .writeStream
+ .trigger(Trigger.AvailableNow())
+ .option("checkpointLocation", checkpoint.getPath)
+ .foreachBatch(writeBatch)
+ .start()
+ try query.awaitTermination() finally query.stop()
+ }
+ def calls(limits: String): Seq[String] =
+ Files.readAllLines(new File(dir, limits).toPath).asScala.toSeq
+ // The second micro-batch, [5, 6), fails, so it is planned but not
committed.
+ val e = intercept[Exception](run(failingBatch = 1, "first"))
+ assert(Iterator.iterate[Throwable](e)(_.getCause).takeWhile(_ != null)
+ .exists(_.getMessage == "micro-batch 1 fails"))
+ assert(batches.toSeq == Seq(0L -> (0L until 5L)))
+ assert(calls("first") == Seq("0 all -", "0 5 6", "5 5 6"))
+ // The new run first runs that micro-batch again, up to its end in the
offset log, without
+ // asking for a latest offset, then asks for a new end, 12, and reads up
to it within the limit.
+ run(failingBatch = -1, "second")
+ assert(batches.toSeq ==
+ Seq(0L -> (0L until 5L), 1L -> Seq(5L), 2L -> (6L until 11L), 3L ->
Seq(11L)))
+ assert(calls("second") == Seq("6 all -", "6 5 12", "11 5 12"))
+ }
+
+ nativeTest("reject the read limits that a micro-batch cannot have") {
+ // Checks the error of a query, and returns whether its stream reader was
closed then.
+ def check(format: String, options: Map[String, String], msg: String):
Boolean = {
+ val packages = if (format.startsWith("nrl_")) {
+ createPackage(noReadLimitLibrary, dataSources("nrl_"))
+ } else {
+ defaultPackage
+ }
+ val closedPath = new File(Utils.createTempDir(), "closed")
+ val e = intercept[Exception] {
+ val query =
newSession(packages).readStream.format(format).options(options)
+ .option("closed_path", closedPath.getPath)
+ .load()
+ .writeStream
+ .format("noop")
+ .option("checkpointLocation", Utils.createTempDir().getPath)
+ .start()
+ try query.processAllAvailable() finally query.stop()
+ }
+ checkError(
+ exception = findError(e, "NATIVE_DATA_SOURCE_ERROR"),
+ condition = "NATIVE_DATA_SOURCE_ERROR",
+ parameters = Map("action" -> "plan a scan of", "name" -> format, "msg"
-> msg))
+ closedPath.isFile
+ }
+ // Spark asks for the limit before it records the streams to stop when the
query stops, so the
+ // stream closes its reader itself when the limit fails.
+ assert(check("native_counter", Map("bad_read_limit" -> "kind"),
+ "default_read_limit returned the unknown kind of read limit 7."))
+ assert(check("native_counter", Map("bad_read_limit" -> "rows"),
+ "default_read_limit returned a maximum of 0 rows, but a micro-batch
reads at least 1 row."))
+ // A data source must not declare a capability whose functions are NULL.
Spark checks them
+ // before it creates the stream reader.
+ assert(!check("nrl_native_counter", Map("max_rows_per_trigger" -> "5"),
+ "The library does not implement default_read_limit."))
+ }
+
nativeTest("ask for the initial offset once, and keep it in the checkpoint")
{
val session = newSession(defaultPackage)
val checkpoint = Utils.createTempDir()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]