This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8f1ce6974c8b [SPARK-60726][SQL] Add an API to read a partition of a
native data source as an Arrow C stream
8f1ce6974c8b is described below
commit 8f1ce6974c8b7410073e14f7fb82d88b63c24dea
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:43:02 2026 +0900
[SPARK-60726][SQL] Add an API to read a partition of a native data source
as an Arrow C stream
### What changes were proposed in this pull request?
A follow-up to SPARK-60101 (#59424), which zhouyuan asked for in
https://github.com/apache/spark/pull/59424#issuecomment-6196064644. The design
questions were listed in the reply
https://github.com/apache/spark/pull/59424#issuecomment-6204557249. The change
is an `Evolving` API, like the other mix-ins of
`org.apache.spark.sql.connector.read`, to read a partition of a native scan as
an Arrow C stream:
```java
Evolving
public interface SupportsArrowStreamRead extends PartitionReaderFactory {
void openArrowStream(InputPartition partition, long streamAddress);
}
```
The reader factories of the batch and micro-batch scans of native data
sources implement it, with the stream that `read` of the library initializes.
Its arrays have exactly the columns of the read schema of the scan, since
`read` returns exactly the columns of its schema to read (#59931), also for a
library that does not prune the others. This PR is based on #59931 (version 1
of the interface), whose commit it includes until that PR is merged. The
questions of the reply are settled as [...]
- **Which object.** The engine gets the reader factory of the scan from its
plan, `BatchScanExec.readerFactory`, and the partitions with it. The factory of
a batch scan holds the location of the library, the name of the data source,
the read schema, its time zone and the state of the reader.
- **Who owns the task context.** `openArrowStream` is called on the thread
of the running task that reads the partition, and fails otherwise. The engine
owns the stream, which it may move, and releases it, also after the task ended
and on another thread. Spark frees the `SparkDsTaskContext` once the library
released its stream: it wraps the four callbacks of the stream with its own,
which call those of the library, keyed by a `private_data` of Spark. Once the
task failed or completed, [...]
- **What the engine checks.** Spark does not check the arrays of such a
stream, while it checks those that it imports itself, because what to check
depends on the importer. Arrow Java ignores the offsets of the arrays, for
example, while Arrow C++ and arrow-rs do not. Spark still checks that the
library initialized the stream, with non-NULL callbacks. The callbacks can be
called from any thread, but never concurrently, as the header already says, and
they call into the JVM, so the eng [...]
`NativePartitionReader` opens its streams with the same code. The release
of an engine's stream closes its `NativeLibrary.Task` on whichever thread the
engine releases it, and only the release that removes the stream from the
wrapped streams does, so the task is closed once. The docs describe the API
under "Native Execution Engines".
The reader factories of micro-batch scans implement it too:
`ColumnarStreamReader.readerFactory()` lets a stream reader return its own
factory, `NativeMicroBatchPartitionReaderFactory` for native data sources. The
partitions of each micro-batch hold the state that the stream reader returned
with them, so that a factory that an engine keeps reads them with that state,
also once Spark planned the next micro-batch. A state larger than 64 KiB is
broadcast, so that each executor fetches it [...]
### Why are the changes needed?
Without it, a query on a native data source under a native engine such as
Gluten imports the data into Arrow Java. A `ColumnarToRowExec` then converts it
to rows, which the engine converts back to its own columnar format:
```
read -> ArrowArrayStream -> Arrow Java -> ArrowColumnVector ->
ColumnarToRow -> RowToVeloxColumnar
```
With the API, the engine imports the stream of the library directly.
Library loading, the checksums of the packages and `spark_ds_init` stay in
Spark, so the same library runs on Spark alone and under the engine.
### Does this PR introduce _any_ user-facing change?
It adds an API for engines, `SupportsArrowStreamRead`, in 4.4.0. Nothing
changes for queries. For a library, the `partition_id` of a read with Spark's
own reader is now the index of the partition in the scan, rather than the
partition of the task, which differs after a union or a coalesce; native data
sources are new in the unreleased 4.4.0.
### How was this patch tested?
"open the partitions of batch and micro-batch scans as Arrow C streams" in
`NativeDataSourceSuite` acts as an engine: in a single task, it opens the
streams of both partitions, reads them on another thread, and checks the data
and the partitions that the library got. It also reads a stream to its end in
its task and releases it after the task ended, from the driver, and checks that
the context stays alive until then. It checks the preconditions of
`openArrowStream`, including a comple [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59934 from HyukjinKwon/native-ds-arrow-stream-read.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit e06d5d3c6990898f2b935d5e90dcd2b19b982863)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/sql-native-data-sources.md | 47 +-
.../connector/read/SupportsArrowStreamRead.java | 62 +++
.../datasources/v2/ffi/NativeLibrary.java | 349 +++++++++++-
.../apache/spark/sql/datasource/spark_datasource.h | 45 +-
.../v2/columnar/ColumnarDataSource.scala | 16 +-
.../datasources/v2/columnar/ColumnarTable.scala | 12 +-
.../datasources/v2/ffi/NativeDataSource.scala | 349 ++++++++++--
.../native-datasource/test_native_datasource.cc | 44 +-
.../v2/columnar/ColumnarDataSourceSuite.scala | 8 +-
.../datasources/v2/ffi/NativeDataSourceSuite.scala | 604 ++++++++++++++++++++-
10 files changed, 1395 insertions(+), 141 deletions(-)
diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index 03b5eae4ab0c..049f49902e7c 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -116,11 +116,12 @@ Spark calls the functions of the table as follows:
not specify a schema, Spark asks for it with `schema`.
- **Batch reads**: on the driver, Spark creates a reader with `create_reader`,
pushes operations
down with `push_limit` and `prune_columns`, and plans the partitions with
`partitions`, which
- also returns the state of the reader. Then it calls `read` on the executors,
in one task for each
- partition, with a `SparkDsReadInfo` that has the partition, the schema to
read and the state of
- the reader, and `read` exports an `ArrowArrayStream`. Its callbacks can be
called from any
- thread, but never concurrently. The schema to read has only the columns that
the query needs,
- which `prune_columns` got, and the arrays of the stream have exactly these
columns, also when the
+ also returns the state of the reader. Then it calls `read` on the executors
for each partition,
+ in tasks that may each read several partitions, for example after a coalesce
or a union, with a
+ `SparkDsReadInfo` that has the partition, the schema to read and the state
of the reader, and
+ `read` exports an `ArrowArrayStream`. Its callbacks can be called from any
thread, but never
+ concurrently. The schema to read has only the columns that the query needs,
which
+ `prune_columns` got, and the arrays of the stream have exactly these
columns, also when the
library does not implement `prune_columns`, which only lets it plan to read
them.
- **Streaming reads**: on the driver, Spark creates a stream reader with
`create_stream_reader`,
which gets the columns to read, which may be only those that the query
needs, and the location of
@@ -201,13 +202,16 @@ to, but it cannot check that the buffers of the arrays
hold those values, becaus
interface does not carry the sizes of buffers: a library that exports a buffer
that is too short
makes Spark read past its end.
-`read` and `create_data_writer` get a `SparkDsTaskContext`, with the IDs of
the task, and its
-partition and attempt, and an `is_interrupted` function that returns whether
the task was killed,
-for example because its query was canceled. Spark cannot interrupt native
code: when a task is
-killed, Spark stops reading only once the `get_next` function of the stream
returns. So a read or
-a write that runs for long, or blocks, for example on network I/O, should
check `is_interrupted`
-regularly and use timeouts. The task reaper, `spark.task.reaper.enabled` and
-`spark.task.reaper.killTimeout`, kills an executor whose killed task does not
end.
+`read` and `create_data_writer` get a `SparkDsTaskContext`, with the IDs of
the task, its
+attempt, and the partition: for `read`, the index of the partition in the list
that the planning
+returned, which is unique within a scan but not across the scans that a task
reads, and for
+`create_data_writer`, the partition of the task. It also has an
`is_interrupted` function that
+returns whether the task was killed, for example because its query was
canceled. Spark cannot
+interrupt native code: when a task is killed, Spark stops reading only once
the `get_next` function
+of the stream returns. So a read or a write that runs for long, or blocks, for
example on network
+I/O, should check `is_interrupted` regularly and use timeouts. The task reaper,
+`spark.task.reaper.enabled` and `spark.task.reaper.killTimeout`, kills an
executor whose killed
+task does not end.
### Compatibility
@@ -269,6 +273,25 @@ Increase it for libraries that use much memory, and bound
the memory that a libr
example with the size of its batches: on YARN or Kubernetes, an executor that
uses more memory than
it requested is killed.
+### Native Execution Engines
+
+An engine that runs queries in native code within Spark, such as Apache Gluten
or Apache DataFusion
+Comet, can read a native data source without Spark importing its data on the
JVM. The reader factory
+of a batch or micro-batch scan of a native data source implements
+`org.apache.spark.sql.connector.read.SupportsArrowStreamRead`: in the task
that reads a partition,
+`openArrowStream` initializes an Arrow C stream that the engine allocated with
the stream that
+`read` returns. The engine reads the stream in the task that opened it, and
owns and releases the
+stream, which it may do after the task ended: Spark frees the
`SparkDsTaskContext` of the read once
+the library released its stream. Once the task ended, `is_interrupted` returns
true, so the library
+may stop reading then. The callbacks of the stream can be called from any
thread, but never
+concurrently. Each call goes through the JVM, which then calls the callback of
the stream of the
+library: this adds tens of nanoseconds to the call, while the import of a
batch on the JVM takes
+microseconds. The first call on a native thread also attaches the thread to
the JVM, which takes
+tens of microseconds, and the thread stays attached until it exits. Since the
callbacks call into
+the JVM, the engine releases the stream before the JVM exits. Spark does not
check the arrays of
+such a stream, unlike the streams that it reads itself, so the engine checks
what its import of the
+arrays requires.
+
## Finding Native Data Sources
When no Java or Python data source has the name that a query uses, Spark looks
for a native data
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
new file mode 100644
index 000000000000..7b772a6d9951
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * A mix-in interface for a {@link PartitionReaderFactory} that also reads a
partition as an
+ * <a href="https://arrow.apache.org/docs/format/CStreamInterface.html">Arrow
C stream</a>, for an
+ * engine that processes Arrow data in native code: the engine imports the
arrays of the stream
+ * itself, instead of reading the {@link
org.apache.spark.sql.vectorized.ColumnarBatch}es of
+ * {@link #createColumnarReader(InputPartition)}. The reader factories of the
batch and
+ * micro-batch scans of native data sources implement it, with the stream that
the library
+ * returns.
+ * <p>
+ * Each array of the stream is a struct array whose fields are the columns of
the read schema of
+ * the scan. Spark only checks that the source initialized the callbacks of
the stream, not its
+ * arrays, while it checks the arrays of the streams that it reads itself, so
the engine checks
+ * what its import of the arrays requires, such as the schema of the stream
against the read
+ * schema, and the offsets and the lengths of the arrays. The callbacks of the
stream can be called
+ * from any thread, but never concurrently. Each call goes through the JVM,
which then calls the
+ * callback of the stream of the source, so the engine releases the stream
before the JVM exits.
+ * The first call on a native thread attaches the thread to the JVM, until the
thread exits.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface SupportsArrowStreamRead extends PartitionReaderFactory {
+
+ /**
+ * Initializes an Arrow C stream with the data of a partition. It is called
in the running task
+ * that reads the partition, on a thread that has its {@code TaskContext},
and fails otherwise.
+ * The caller reads the stream in that task: once the task ended, the source
may stop returning
+ * data and fail. The caller owns
+ * the stream, which it may move, and releases it, which may happen after
the task ended: Spark
+ * frees what the stream uses, such as the context of the task, once the
stream is released. A
+ * stream that is never released leaks.
+ *
+ * @param partition a partition of the scan, as the scan planned it
+ * @param streamAddress the address of a {@code struct ArrowArrayStream}
that the caller
+ * allocated, and whose release callback is NULL, as it
is in a
+ * zero-initialized struct. If this method fails after
it checked its
+ * arguments, the stream is released. A stream whose
release callback
+ * is not NULL is left as it is.
+ */
+ void openArrowStream(InputPartition partition, long streamAddress);
+}
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index 2185e8d73c12..ba0e2e683916 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -189,6 +189,13 @@ final class NativeLibrary {
private static final long FINISH_WRITE_INFO_NUM_MESSAGES = 32;
private static final long FINISH_WRITE_INFO_REJECTED = 40;
private static final long FINISH_DATA_WRITER_INFO_SIZE = 8;
+ // struct ArrowArrayStream of the Arrow C stream interface, which Spark
wraps for engines.
+ private static final long STREAM_GET_SCHEMA = 0;
+ private static final long STREAM_GET_NEXT = 8;
+ private static final long STREAM_GET_LAST_ERROR = 16;
+ private static final long STREAM_RELEASE = 24;
+ private static final long STREAM_PRIVATE_DATA = 32;
+ private static final long STREAM_SIZE = 40;
// The structs that only hold their size, which the library gets as const
pointers. Until fields
// are added to them, a single struct of each kind is allocated once, and
never freed.
@@ -283,6 +290,13 @@ final class NativeLibrary {
layout.put("SparkDsFinishWriteInfo.rejected", FINISH_WRITE_INFO_REJECTED);
layout.put("SparkDsFinishDataWriterInfo", FINISH_DATA_WRITER_INFO_SIZE);
layout.put("SparkDsFinishDataWriterInfo.size", 0L);
+ // The streams of engines, which Spark wraps, see EngineStream.
+ layout.put("ArrowArrayStream", STREAM_SIZE);
+ layout.put("ArrowArrayStream.get_schema", STREAM_GET_SCHEMA);
+ layout.put("ArrowArrayStream.get_next", STREAM_GET_NEXT);
+ layout.put("ArrowArrayStream.get_last_error", STREAM_GET_LAST_ERROR);
+ layout.put("ArrowArrayStream.release", STREAM_RELEASE);
+ layout.put("ArrowArrayStream.private_data", STREAM_PRIVATE_DATA);
return layout;
}
@@ -492,15 +506,7 @@ final class NativeLibrary {
MethodHandle handle = (MethodHandle) DOWNCALL.invoke(
LINKER, address, descriptor(result, parameters), NO_OPTIONS);
// Passes and returns the pointers, which are MemorySegments in the FFM
API, as addresses.
- for (int i = 0; i < parameters.length; i++) {
- if (parameters[i] == C.PTR) {
- handle = MethodHandles.filterArguments(handle, i, OF_ADDRESS);
- }
- }
- if (result == C.PTR) {
- handle = MethodHandles.filterReturnValue(handle, ADDRESS_OF);
- }
- return handle;
+ return convertPointers(handle, 0, OF_ADDRESS, ADDRESS_OF, result,
parameters);
}
/**
@@ -509,14 +515,33 @@ final class NativeLibrary {
* function lives as long as the JVM.
*/
static long upcall(MethodHandle target, C result, C... parameters) throws
Throwable {
+ target = convertPointers(target, 0, ADDRESS_OF, OF_ADDRESS, result,
parameters);
+ Object stub = UPCALL.invoke(
+ LINKER, target, descriptor(result, parameters), GLOBAL_ARENA,
NO_OPTIONS);
+ return (long) ADDRESS_OF.invoke(stub);
+ }
+
+ /**
+ * Applies `parameterFilter` to the parameters of `handle` that are
pointers, the C parameters
+ * starting at the parameter `first`, and `resultFilter` to a result that
is a pointer: a
+ * downcall converts the addresses to MemorySegments and back, and an
upcall the other way.
+ */
+ static MethodHandle convertPointers(
+ MethodHandle handle,
+ int first,
+ MethodHandle parameterFilter,
+ MethodHandle resultFilter,
+ C result,
+ C... parameters) {
for (int i = 0; i < parameters.length; i++) {
if (parameters[i] == C.PTR) {
- target = MethodHandles.filterArguments(target, i, ADDRESS_OF);
+ handle = MethodHandles.filterArguments(handle, first + i,
parameterFilter);
}
}
- Object stub = UPCALL.invoke(
- LINKER, target, descriptor(result, parameters), GLOBAL_ARENA,
NO_OPTIONS);
- return (long) ADDRESS_OF.invoke(stub);
+ if (result == C.PTR) {
+ handle = MethodHandles.filterReturnValue(handle, resultFilter);
+ }
+ return handle;
}
/**
@@ -525,11 +550,7 @@ final class NativeLibrary {
*/
static long host(String path, String sparkVersion) throws Throwable {
long log = upcall(MethodHandles.insertArguments(LOG, 0, path), C.VOID,
C.I32, C.PTR);
- byte[] version = sparkVersion.getBytes(StandardCharsets.UTF_8);
- long versionAddress = Platform.allocateMemory(version.length + 1L);
- Platform.copyMemory(
- version, Platform.BYTE_ARRAY_OFFSET, null, versionAddress,
version.length);
- Platform.putByte(null, versionAddress + version.length, (byte) 0);
+ long versionAddress = toCString(sparkVersion);
long host = Platform.allocateMemory(HOST_SIZE);
Platform.setMemory(host, (byte) 0, HOST_SIZE);
Platform.putLong(null, host, HOST_SIZE);
@@ -1063,9 +1084,8 @@ final class NativeLibrary {
/** Copies the string as a NUL-terminated UTF-8 string. */
long string(String value) {
- byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
- long address = zeroed(bytes.length + 1L);
- Platform.copyMemory(bytes, Platform.BYTE_ARRAY_OFFSET, null, address,
bytes.length);
+ long address = toCString(value);
+ allocations.add(address);
return address;
}
@@ -1084,6 +1104,15 @@ final class NativeLibrary {
}
}
+ /** Copies a string to native memory, as a NUL-terminated UTF-8 string,
which the caller frees. */
+ private static long toCString(String value) {
+ byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
+ long address = Platform.allocateMemory(bytes.length + 1L);
+ Platform.copyMemory(bytes, Platform.BYTE_ARRAY_OFFSET, null, address,
bytes.length);
+ Platform.putByte(null, address + bytes.length, (byte) 0);
+ return address;
+ }
+
private static byte[] readCString(long address) {
int length = 0;
while (Platform.getByte(null, address + length) != 0) {
@@ -1098,12 +1127,16 @@ final class NativeLibrary {
// Task contexts
//
---------------------------------------------------------------------------------------------
- // Whether the task of each SparkDsTaskContext was killed, by the address of
the struct.
+ // Whether the library should stop reading or writing with each
SparkDsTaskContext, which
+ // is_interrupted answers, by the address of the struct.
private static final ConcurrentHashMap<Long, BooleanSupplier> tasks = new
ConcurrentHashMap<>();
/**
* The {@code SparkDsTaskContext} of a task that reads a partition or writes
data. The reader or
- * the data writer closes it once the library no longer uses it, which frees
it.
+ * the data writer closes it once the library no longer uses it, which frees
it. For the stream of
+ * a read that an engine got, the release of the stream closes it, on
whichever thread the engine
+ * releases the stream: only the release that removes the stream from {@code
engineStreams} does,
+ * so it is closed once (see {@link EngineStream}).
*/
static final class Task implements AutoCloseable {
final long address;
@@ -1153,6 +1186,276 @@ final class NativeLibrary {
}
}
+ /** The number of task contexts that are not closed yet, for tests. */
+ static int openTaskContexts() {
+ return tasks.size();
+ }
+
+ //
---------------------------------------------------------------------------------------------
+ // Streams of engines
+ //
---------------------------------------------------------------------------------------------
+
+ // The errno values that the callbacks of a stream return.
+ private static final int EINVAL = 22;
+ private static final int EIO = 5;
+
+ /**
+ * The callbacks of the streams that Spark wraps for engines, see {@link
EngineStream}, the
+ * method handles that call the callbacks of the wrapped streams, and what
the callbacks return
+ * for a stream that Spark does not know. They are created when the first
stream is wrapped, so
+ * that a JVM that wraps no stream does not create them, and a failure to
create them only fails
+ * the streams of engines.
+ */
+ private static final class EngineStreamCallbacks {
+ // The addresses of the C functions that call Java, which live as long as
the JVM.
+ static final long GET_SCHEMA;
+ static final long GET_NEXT;
+ static final long GET_LAST_ERROR;
+ static final long RELEASE;
+ // The callbacks of struct ArrowArrayStream, called with the address of
the function first:
+ // get_schema and get_next, get_last_error, and release.
+ static final MethodHandle INT_CALLBACK;
+ static final MethodHandle POINTER_CALLBACK;
+ static final MethodHandle VOID_CALLBACK;
+ // The message of get_last_error for a stream that Spark does not know,
which lives as long as
+ // the JVM.
+ static final long UNKNOWN_STREAM_ERROR;
+
+ static {
+ try {
+ MethodHandles.Lookup lookup = MethodHandles.lookup();
+ // Linker.downcallHandle(FunctionDescriptor, Linker.Option...), whose
method handles take
+ // the address of the function to call as their first argument.
+ MethodHandle unboundDowncall = lookup.findVirtual(
+ Class.forName("java.lang.foreign.Linker"), "downcallHandle",
+ MethodType.methodType(MethodHandle.class,
+ Class.forName("java.lang.foreign.FunctionDescriptor"),
Ffm.NO_OPTIONS.getClass()))
+ .asFixedArity();
+ GET_SCHEMA = Ffm.upcall(
+ lookup.findStatic(NativeLibrary.class, "engineStreamGetSchema",
+ MethodType.methodType(int.class, long.class, long.class)),
+ C.I32, C.PTR, C.PTR);
+ GET_NEXT = Ffm.upcall(
+ lookup.findStatic(NativeLibrary.class, "engineStreamGetNext",
+ MethodType.methodType(int.class, long.class, long.class)),
+ C.I32, C.PTR, C.PTR);
+ GET_LAST_ERROR = Ffm.upcall(
+ lookup.findStatic(NativeLibrary.class, "engineStreamGetLastError",
+ MethodType.methodType(long.class, long.class)),
+ C.PTR, C.PTR);
+ RELEASE = Ffm.upcall(
+ lookup.findStatic(NativeLibrary.class, "engineStreamRelease",
+ MethodType.methodType(void.class, long.class)),
+ C.VOID, C.PTR);
+ INT_CALLBACK = callback(unboundDowncall, C.I32, C.PTR, C.PTR);
+ POINTER_CALLBACK = callback(unboundDowncall, C.PTR, C.PTR);
+ VOID_CALLBACK = callback(unboundDowncall, C.VOID, C.PTR);
+ UNKNOWN_STREAM_ERROR = toCString("The stream was released, or is not a
stream of Spark.");
+ } catch (Throwable e) {
+ // The message names the cause, because the JVM only keeps this
message for the later
+ // uses of the class.
+ throw new IllegalStateException(
+ "Cannot create the callbacks of the streams of engines: " +
rootCause(e), e);
+ }
+ }
+
+ /**
+ * Returns a method handle that calls the C function at the address of its
first argument, a
+ * {@code long}, with the given types, as {@link Ffm#downcall} does for a
fixed address.
+ */
+ private static MethodHandle callback(MethodHandle unboundDowncall, C
result, C... parameters)
+ throws Throwable {
+ MethodHandle handle = (MethodHandle) unboundDowncall.invoke(
+ Ffm.LINKER, Ffm.descriptor(result, parameters), Ffm.NO_OPTIONS);
+ handle = MethodHandles.filterArguments(handle, 0, Ffm.OF_ADDRESS);
+ return Ffm.convertPointers(handle, 1, Ffm.OF_ADDRESS, Ffm.ADDRESS_OF,
result, parameters);
+ }
+ }
+
+ private static final AtomicLong nextEngineStreamId = new AtomicLong(1);
+ // The streams wrapped for engines that are not released yet, by their
private_data.
+ private static final ConcurrentHashMap<Long, EngineStream> engineStreams =
+ new ConcurrentHashMap<>();
+
+ /**
+ * The stream of a read that an engine got with {@code
SupportsArrowStreamRead}, which ties the
+ * task context of the read to the stream: the context is closed once the
library released the
+ * stream, whenever and on whichever thread the engine releases it, also
after the task ended.
+ * The stream of the library is moved to memory of Spark, and the stream of
the engine gets
+ * callbacks of Spark, which call the current callbacks of the library's
stream, and a
+ * private_data that identifies the wrapped stream, so that the engine can
move the stream too.
+ */
+ private static final class EngineStream {
+ final long inner;
+ final Task task;
+ // The name of the data source, for the errors.
+ final String name;
+ // The message of the last error of Spark, rather than of the library, or
0 if the last call
+ // returned an error of the library, or none. The callbacks never run
concurrently, but may
+ // run on different threads one after the other.
+ volatile long sparkError;
+
+ EngineStream(long inner, Task task, String name) {
+ this.inner = inner;
+ this.task = task;
+ this.name = name;
+ }
+
+ long callback(long offset) {
+ return Platform.getLong(null, inner + offset);
+ }
+
+ /**
+ * Records an error of Spark in the given callback, which get_last_error
then returns, and logs
+ * it. The message is built here, since building it can throw too, as a
StackOverflowError on
+ * the small stack of a thread of an engine can.
+ */
+ int fail(String callback, Throwable e) {
+ try {
+ clearError();
+ sparkError = toCString("The " + callback + " of the stream of the
native data source " +
+ name + " failed: " + rootCause(e));
+ LOGGER.error("The {} of the stream of the native data source {}
failed", e,
+ MDC.of(LogKeys.FUNCTION_NAME, callback), MDC.of(LogKeys.DATA_SOURCE,
name));
+ } catch (Throwable ignored) {
+ // An upcall must not throw.
+ }
+ return EIO;
+ }
+
+ void clearError() {
+ if (sparkError != 0) {
+ Platform.freeMemory(sparkError);
+ sparkError = 0;
+ }
+ }
+ }
+
+ /**
+ * Wraps the stream at the given address, which read initialized with
non-NULL callbacks, so that
+ * releasing it closes the given task context (see {@link EngineStream}). If
it fails, the stream
+ * is left as it is.
+ *
+ * @param name the name of the data source, for the errors of the stream
+ */
+ static void wrapEngineStream(long stream, Task task, String name) {
+ // Creates the callbacks first, so that the stream is left as it is if
that fails.
+ long getSchema;
+ long getNext;
+ long getLastError;
+ long release;
+ try {
+ getSchema = EngineStreamCallbacks.GET_SCHEMA;
+ getNext = EngineStreamCallbacks.GET_NEXT;
+ getLastError = EngineStreamCallbacks.GET_LAST_ERROR;
+ release = EngineStreamCallbacks.RELEASE;
+ } catch (ExceptionInInitializerError | NoClassDefFoundError e) {
+ // The first use of a class that failed to initialize gets the error,
and the later ones a
+ // NoClassDefFoundError, whose cause has its message: both name the root
cause here, as the
+ // errors of load do.
+ throw new IllegalStateException(
+ "Cannot create the callbacks of the streams of engines: " +
rootCause(e), e);
+ }
+ long inner = Platform.allocateMemory(STREAM_SIZE);
+ long id = nextEngineStreamId.getAndIncrement();
+ try {
+ Platform.copyMemory(null, stream, null, inner, STREAM_SIZE);
+ engineStreams.put(id, new EngineStream(inner, task, name));
+ } catch (Throwable e) {
+ engineStreams.remove(id);
+ Platform.freeMemory(inner);
+ throw e;
+ }
+ Platform.putLong(null, stream + STREAM_GET_SCHEMA, getSchema);
+ Platform.putLong(null, stream + STREAM_GET_NEXT, getNext);
+ Platform.putLong(null, stream + STREAM_GET_LAST_ERROR, getLastError);
+ Platform.putLong(null, stream + STREAM_RELEASE, release);
+ Platform.putLong(null, stream + STREAM_PRIVATE_DATA, id);
+ }
+
+ /** The number of streams wrapped for engines that are not released yet, for
tests. */
+ static int openEngineStreams() {
+ return engineStreams.size();
+ }
+
+ private static EngineStream engineStream(long stream) {
+ return engineStreams.get(Platform.getLong(null, stream +
STREAM_PRIVATE_DATA));
+ }
+
+ // The callbacks of the streams wrapped for engines. They must not throw: an
exception thrown by
+ // an upcall crashes the JVM.
+
+ private static int engineStreamGetSchema(long stream, long out) {
+ return engineStreamCall(stream, out, STREAM_GET_SCHEMA, "get_schema");
+ }
+
+ private static int engineStreamGetNext(long stream, long out) {
+ return engineStreamCall(stream, out, STREAM_GET_NEXT, "get_next");
+ }
+
+ private static int engineStreamCall(long stream, long out, long offset,
String callback) {
+ EngineStream wrapped = null;
+ try {
+ wrapped = engineStream(stream);
+ if (wrapped == null) {
+ return EINVAL;
+ }
+ wrapped.clearError();
+ return (int) EngineStreamCallbacks.INT_CALLBACK.invokeExact(
+ wrapped.callback(offset), wrapped.inner, out);
+ } catch (Throwable e) {
+ return wrapped == null ? EIO : wrapped.fail(callback, e);
+ }
+ }
+
+ private static long engineStreamGetLastError(long stream) {
+ try {
+ EngineStream wrapped = engineStream(stream);
+ if (wrapped == null) {
+ return EngineStreamCallbacks.UNKNOWN_STREAM_ERROR;
+ }
+ if (wrapped.sparkError != 0) {
+ return wrapped.sparkError;
+ }
+ return (long) EngineStreamCallbacks.POINTER_CALLBACK.invokeExact(
+ wrapped.callback(STREAM_GET_LAST_ERROR), wrapped.inner);
+ } catch (Throwable e) {
+ return 0;
+ }
+ }
+
+ private static void engineStreamRelease(long stream) {
+ EngineStream wrapped = null;
+ try {
+ wrapped = engineStreams.remove(Platform.getLong(null, stream +
STREAM_PRIVATE_DATA));
+ if (wrapped != null) {
+ try {
+ EngineStreamCallbacks.VOID_CALLBACK.invokeExact(
+ wrapped.callback(STREAM_RELEASE), wrapped.inner);
+ } finally {
+ // The library no longer uses the task once it released its stream.
+ wrapped.clearError();
+ Platform.freeMemory(wrapped.inner);
+ wrapped.task.close();
+ }
+ }
+ } catch (Throwable e) {
+ try {
+ if (wrapped == null) {
+ LOGGER.error("The release of the stream of a native data source
failed", e);
+ } else {
+ LOGGER.error("The {} of the stream of the native data source {}
failed", e,
+ MDC.of(LogKeys.FUNCTION_NAME, "release"),
MDC.of(LogKeys.DATA_SOURCE, wrapped.name));
+ }
+ } catch (Throwable ignored) {
+ // An upcall must not throw.
+ }
+ } finally {
+ // Marks the stream released, as the C stream interface requires of
release.
+ Platform.putLong(null, stream + STREAM_RELEASE, 0);
+ }
+ }
+
//
---------------------------------------------------------------------------------------------
// Sink
//
---------------------------------------------------------------------------------------------
diff --git
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index 1aa4eb346885..a8d7785ee1b1 100644
---
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -57,8 +57,9 @@
* - Batch reads. On the driver, Spark creates a reader with create_reader,
pushes operations down
* by calling, each at most once and in this order, push_limit and
prune_columns, plans the
* partitions with partitions, which also returns the state of the reader,
and closes it. Then it
- * calls read on the executors, in one task for each partition, with the
schema to read and the
- * state of the reader. If Spark never plans the partitions, for example to
only explain the plan
+ * calls read on the executors for each partition, with the schema to read
and the state of the
+ * reader, in tasks that may each read several partitions, for example after
a coalesce or a
+ * union. If Spark never plans the partitions, for example to only explain
the plan
* of the query, it closes the reader once the reader is no longer
referenced.
* - Streaming reads. On the driver, Spark creates a stream reader with
create_stream_reader, with
* the columns to read and the location of the checkpoint of the source, and
keeps it until the
@@ -408,7 +409,14 @@ typedef struct SparkDsCommitOffsetInfo {
/*
* The task that calls read or create_data_writer, on an executor. It stays
valid until the stream
- * that read returns is released, or until the data writer is closed.
+ * that read returns is released, or until the data writer is closed. The
stream of a read may be
+ * released once the task ended, from another thread: an engine that runs a
query in native code
+ * may get the stream from Spark, read it in the task that opened it and
release it later, and
+ * Spark itself may read a stream on a thread that the task does not wait for,
for example the one
+ * of a PipedRDD. The task stays valid until then too, and for a read,
is_interrupted then also
+ * returns true once the task failed or completed, so the library may stop
reading then. Such an
+ * engine may also keep the arrays and the schemas of the stream after it
released the stream, so
+ * the release of an array or a schema must not use the stream or the task.
*/
typedef struct SparkDsTaskContext {
/* The size of this struct in the header that Spark was built with, as in
SparkDsHost. */
@@ -416,15 +424,23 @@ typedef struct SparkDsTaskContext {
/* The ID of the stage of the task, and the attempt number of the stage. */
int32_t stage_id;
int32_t stage_attempt_number;
- /* The partition that the task reads or writes, and the attempt number of
the task. */
+ /*
+ * The partition: for read, the index of the partition in the list that
partitions or
+ * stream_partitions returned, which is not the partition of the task when
Spark reads several
+ * scans or partitions in one task, for example after a union or a coalesce.
It is unique within
+ * a scan, but a task that reads several scans may get the same index for
each. For
+ * create_data_writer, the partition of the task that writes. Then the
attempt number of the
+ * task.
+ */
int32_t partition_id;
int32_t attempt_number;
/* The ID of the task attempt, unique in the Spark application. */
int64_t task_attempt_id;
/*
- * Returns whether the task was killed, for example because its query was
canceled. Spark cannot
- * interrupt native code, so a read or a write that runs for long, or
blocks, should check it
- * regularly, and fail once it is true.
+ * Returns whether the task was killed, for example because its query was
canceled, and for read,
+ * also whether the task failed or completed, since the stream of a read may
be read after its
+ * task ended. Spark cannot interrupt native code, so a read or a write that
runs for long, or
+ * blocks, should check it regularly, and fail once it is true.
*/
bool (*is_interrupted)(const struct SparkDsTaskContext* task);
} SparkDsTaskContext;
@@ -640,7 +656,8 @@ typedef struct SparkDsLibrary {
int (*prune_columns)(void* reader, const char* const* columns, int64_t
count);
/*
- * Plans the partitions of the scan. Spark reads each partition in a
separate task.
+ * Plans the partitions of the scan. Spark reads each partition with a call
of read, in a task
+ * that may read several partitions, for example after a coalesce or a union.
*
* info: nothing yet but its size, so that inputs can be added within this
version
* add_partition: called with each partition, serialized in a format defined
by the library
@@ -713,11 +730,13 @@ typedef struct SparkDsLibrary {
* out: the stream to initialize. Each array of the stream is a struct array
whose fields are the
* columns of the schema to read, info->schema. The callbacks of the
stream can be called
* from any thread, but never concurrently. get_next may block, for
example on network
- * I/O, but Spark cannot interrupt native code: when the task is
killed, Spark only stops
- * reading once get_next returns, and then releases the stream. So a
get_next that blocks
- * forever keeps the task running, unless the task reaper
(spark.task.reaper.enabled and
- * spark.task.reaper.killTimeout) kills the executor. Use timeouts for
blocking
- * operations, and check task->is_interrupted.
+ * I/O, but Spark cannot interrupt native code: when Spark reads the
stream itself and the
+ * task is killed, Spark only stops reading once get_next returns, and
then releases the
+ * stream. So a get_next that blocks forever keeps the task running,
unless the task reaper
+ * (spark.task.reaper.enabled and spark.task.reaper.killTimeout) kills
the executor. An
+ * engine that gets the stream from Spark reads and releases it itself
(see
+ * SparkDsTaskContext). Use timeouts for blocking operations, and check
+ * task->is_interrupted.
*/
int (*read)(const SparkDsTaskContext* task, const SparkDsReadInfo* info,
struct ArrowArrayStream* out);
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index d0e1fc430635..a98cb66a485a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -20,7 +20,7 @@ import java.util
import org.apache.spark.sql.connector.catalog.TableCapability
import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ,
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader,
PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.{InputPartition,
PartitionReaderFactory}
import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -111,10 +111,11 @@ case class ColumnarPlannedBatch(
/**
* Reads the data of a [[ColumnarDataSource]] for a micro-batch streaming
scan, with offsets in
- * JSON. Spark plans the partitions of each micro-batch on the driver, then
serializes the reader
- * to the executors, where `read` is called for each partition.
+ * JSON. Spark plans the partitions of each micro-batch on the driver with
this reader, which stays
+ * there, and serializes the reader factory that `readerFactory` returns to
the executors, which
+ * read the partitions with it.
*/
-trait ColumnarStreamReader extends Serializable {
+trait ColumnarStreamReader {
/**
* Returns the offset to start from when the streaming query has no
checkpoint. Spark keeps it in
* the checkpoint of the source, and only asks for it when the checkpoint
does not have it yet.
@@ -135,7 +136,12 @@ trait ColumnarStreamReader extends Serializable {
*/
def partitions(start: String, end: String): Array[InputPartition]
- def read(partition: InputPartition): PartitionReader[ColumnarBatch]
+ /**
+ * Returns the reader factory of a micro-batch, which Spark serializes to
the executors. It may
+ * also implement other interfaces of reader factories, such as
+ * `org.apache.spark.sql.connector.read.SupportsArrowStreamRead`.
+ */
+ def readerFactory(): PartitionReaderFactory
/** Tells the data source that Spark has processed all the data up to and
including `end`. */
def commit(end: String): Unit = {}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
index 21bbbeb2e5b1..6b4d8c0db3dc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
@@ -37,7 +37,6 @@ import
org.apache.spark.sql.execution.streaming.checkpointing.HDFSMetadataLog
import org.apache.spark.sql.internal.connector.SupportsMetadata
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-import org.apache.spark.sql.vectorized.ColumnarBatch
import org.apache.spark.util.Utils
/**
@@ -306,9 +305,7 @@ class ColumnarMicroBatchStream(reader:
ColumnarStreamReader, checkpointLocation:
}
}
- override def createReaderFactory(): PartitionReaderFactory = {
- new ColumnarStreamPartitionReaderFactory(reader)
- }
+ override def createReaderFactory(): PartitionReaderFactory =
reader.readerFactory()
override def deserializeOffset(json: String): Offset =
ColumnarStreamOffset(json)
@@ -368,10 +365,3 @@ abstract class ColumnarOnlyPartitionReaderFactory extends
PartitionReaderFactory
override def supportColumnarReads(partition: InputPartition): Boolean = true
}
-
-class ColumnarStreamPartitionReaderFactory(reader: ColumnarStreamReader)
- extends ColumnarOnlyPartitionReaderFactory {
- override def createColumnarReader(partition: InputPartition):
PartitionReader[ColumnarBatch] = {
- reader.read(partition)
- }
-}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 0e359ff7f72f..40e1f2d44d49 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -19,25 +19,31 @@ package org.apache.spark.sql.execution.datasources.v2.ffi
import java.lang.ref.{Cleaner, WeakReference}
import java.util
import java.util.concurrent.ConcurrentHashMap
-import java.util.concurrent.atomic.AtomicLong
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicLong}
+import java.util.function.BooleanSupplier
import javax.annotation.concurrent.GuardedBy
+import scala.collection.mutable
import scala.jdk.CollectionConverters._
+import scala.reflect.{classTag, ClassTag}
import scala.util.control.NonFatal
import com.fasterxml.jackson.core.JsonProcessingException
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.databind.json.JsonMapper
+import com.google.common.collect.MapMaker
import org.apache.arrow.c.{ArrowArray, ArrowArrayStream, ArrowSchema,
CDataDictionaryProvider, Data}
import org.apache.arrow.vector.{FieldVector, ValueVector, VectorSchemaRoot}
import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
import org.apache.arrow.vector.types.pojo.{ArrowType, Field, Schema}
-import org.apache.spark.{SparkException, SparkThrowable,
SparkUnsupportedOperationException, TaskContext}
+import org.apache.spark.{SparkContext, SparkException, SparkThrowable,
SparkUnsupportedOperationException, TaskContext}
+import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.Logging
+import org.apache.spark.sql.classic.SparkSession
import org.apache.spark.sql.connector.catalog.TableCapability
import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ,
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader,
PartitionReaderFactory, SupportsArrowStreamRead}
import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
import
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchPlanner,
ColumnarDataSource, ColumnarOnlyPartitionReaderFactory, ColumnarPlannedBatch,
ColumnarStreamReader, ColumnarStreamWriter, ColumnarWriter}
@@ -283,7 +289,7 @@ class NativeBatchPlanner(
try {
val planned = call(name, "plan a scan
of")(library.partitions(handle.get))
ColumnarPlannedBatch(
- NativeInputPartition.from(planned.partitions),
+ NativeInputPartition.from(planned.partitions).toArray[InputPartition],
new NativePartitionReaderFactory(executorLocation, name,
bytesOrEmpty(Option(planned.state)), readSchema, timeZoneId))
} finally {
@@ -293,17 +299,27 @@ class NativeBatchPlanner(
}
}
-/** Reads the partitions of a planned batch scan of a native data source, on
the executors. */
+/**
+ * Reads the partitions of a planned batch scan of a native data source, on
the executors. An
+ * engine can also get the Arrow C stream of a partition from it, see
[[SupportsArrowStreamRead]].
+ */
class NativePartitionReaderFactory(
location: NativeLibraryLocation,
name: String,
readerState: Array[Byte],
readSchema: StructType,
timeZoneId: String)
- extends ColumnarOnlyPartitionReaderFactory {
+ extends ColumnarOnlyPartitionReaderFactory with SupportsArrowStreamRead {
override def createColumnarReader(partition: InputPartition):
PartitionReader[ColumnarBatch] = {
new NativePartitionReader(location, name, NativeLibrary.READ_BATCH,
readerState,
- partition.asInstanceOf[NativeInputPartition].bytes, readSchema,
timeZoneId)
+ partition.asInstanceOf[NativeInputPartition], readSchema, timeZoneId)
+ }
+
+ override def openArrowStream(partition: InputPartition, streamAddress:
Long): Unit = {
+ NativePartitionReader.openEngineStream[NativeInputPartition](location,
name,
+ NativeLibrary.READ_BATCH, partition, readSchema, timeZoneId,
streamAddress) { partition =>
+ (readerState, partition)
+ }
}
}
@@ -317,17 +333,21 @@ class NativeDataSourceStreamReader(
name: String,
schema: StructType,
timeZoneId: String,
- @transient private val handle: NativeHandle)
+ handle: NativeHandle)
extends ColumnarStreamReader {
import NativeCalls._
- // Set on the driver whenever the partitions of a micro-batch are planned,
and serialized to
- // the executors with the partitions of that micro-batch. Spark plans each
range once (see
- // ColumnarMicroBatchStream), so the scans of a micro-batch get the
partitions of a single
- // stream_partitions call, whose state this is.
- private var readerState: Array[Byte] = Array.emptyByteArray
+ // The ends of the planned ranges that are not committed yet, in the order
in which they were
+ // planned, with the broadcast of the state of their micro-batch, if it was
broadcast. Spark plans
+ // each range once (see ColumnarMicroBatchStream), so the scans of a
micro-batch get the
+ // partitions of a single stream_partitions call, with the state that it
returned.
+ private val uncommitted = mutable.LinkedHashMap.empty[String,
Option[Broadcast[Array[Byte]]]]
+ // The context that broadcasts the states, taken on the driver when the
reader is created, so
+ // that a range can also be planned on a thread without an active session.
+ private val sparkContext =
+ SparkContext.getActive.getOrElse(SparkSession.active.sparkContext)
- @transient private lazy val library = NativeLibraries.get(location)
+ private lazy val library = NativeLibraries.get(location)
// The calls of the reader on the driver are synchronized, since calls on a
handle must not run
// at the same time: Spark may plan the scans of a micro-batch on several
threads, such as the
@@ -366,26 +386,89 @@ class NativeDataSourceStreamReader(
override def partitions(start: String, end: String): Array[InputPartition] =
synchronized {
val planned = call(name, "plan a scan
of")(library.streamPartitions(handle.get, start, end))
- readerState = bytesOrEmpty(Option(planned.state))
- NativeInputPartition.from(planned.partitions)
+ val partitions = NativeInputPartition.from(planned.partitions)
+ val bytes = bytesOrEmpty(Option(planned.state))
+ // A small state goes with each task, while the executors fetch a larger
one once for the
+ // micro-batch, as a broadcast. Spark only plans the next range once the
micro-batch of the
+ // previous one finished, so the copies of the previous broadcast on the
executors are removed
+ // then. The broadcast itself is destroyed once its micro-batch is
committed, whose partitions
+ // are not read anymore then, also by an engine.
+ uncommitted.values.lastOption.flatten.foreach(_.unpersist(blocking =
false))
+ val state = if (bytes.length >
NativeDataSourceStreamReader.maxInlineStateBytes &&
+ partitions.nonEmpty) {
+ Right(sparkContext.broadcast(bytes))
+ } else {
+ Left(bytes)
+ }
+ uncommitted.remove(end).flatten.foreach(_.destroy(blocking = false))
+ uncommitted.put(end, state.toOption)
+ partitions.map[InputPartition](NativeMicroBatchPartition(_, state))
}
- override def read(partition: InputPartition): PartitionReader[ColumnarBatch]
= {
- new NativePartitionReader(location, name, NativeLibrary.READ_MICRO_BATCH,
readerState,
- partition.asInstanceOf[NativeInputPartition].bytes, schema, timeZoneId)
+ override def readerFactory(): PartitionReaderFactory = {
+ new NativeMicroBatchPartitionReaderFactory(location, name, schema,
timeZoneId)
}
override def commit(end: String): Unit = synchronized {
+ // The micro-batches up to the committed one are committed too, also those
that Spark did not
+ // commit one by one, as with async progress tracking, and their
broadcasts are destroyed.
+ if (uncommitted.contains(end)) {
+ val committed = uncommitted.keys.toSeq.takeWhile(_ != end) :+ end
+
committed.foreach(uncommitted.remove(_).flatten.foreach(_.destroy(blocking =
false)))
+ }
call(name, "commit an offset
of")(optional(library.commitOffset(handle.get, end)))
}
- override def stop(): Unit = synchronized(handle.close())
+ // The broadcasts of the states of the planned micro-batches that are not
committed yet, for
+ // tests.
+ private[ffi] def uncommittedStates: Seq[Broadcast[Array[Byte]]] =
+ synchronized(uncommitted.values.flatten.toSeq)
+
+ override def stop(): Unit = synchronized {
+ Utils.tryWithSafeFinally {
+ uncommitted.values.flatten.foreach(_.destroy(blocking = false))
+ uncommitted.clear()
+ }(handle.close())
+ }
}
object NativeDataSourceStreamReader {
// Parses the offsets of the libraries to check them, and rejects anything
after the JSON value.
private val mapper =
JsonMapper.builder().enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS).build()
+
+ // The size of the largest state of a micro-batch that goes with each of its
tasks, rather than
+ // as a broadcast, which costs more than it saves for a state of a few
offsets, written to the
+ // block manager of the driver, fetched by each executor, and removed from
them for each
+ // micro-batch.
+ private[ffi] val maxInlineStateBytes: Int = 64 * 1024
+}
+
+/**
+ * Reads the partitions of a micro-batch of a native data source, on the
executors, each with the
+ * state of its micro-batch. An engine can also get the Arrow C stream of a
partition from it, see
+ * [[SupportsArrowStreamRead]].
+ *
+ * @param timeZoneId the time zone of the timestamps of the schema that the
stream reader got
+ */
+class NativeMicroBatchPartitionReaderFactory(
+ location: NativeLibraryLocation,
+ name: String,
+ schema: StructType,
+ timeZoneId: String)
+ extends ColumnarOnlyPartitionReaderFactory with SupportsArrowStreamRead {
+ override def createColumnarReader(partition: InputPartition):
PartitionReader[ColumnarBatch] = {
+ val microBatchPartition = partition.asInstanceOf[NativeMicroBatchPartition]
+ new NativePartitionReader(location, name, NativeLibrary.READ_MICRO_BATCH,
+ microBatchPartition.readerState, microBatchPartition.partition, schema,
timeZoneId)
+ }
+
+ override def openArrowStream(partition: InputPartition, streamAddress:
Long): Unit = {
+
NativePartitionReader.openEngineStream[NativeMicroBatchPartition](location,
name,
+ NativeLibrary.READ_MICRO_BATCH, partition, schema, timeZoneId,
streamAddress) { partition =>
+ (partition.readerState, partition.partition)
+ }
+ }
}
/**
@@ -472,13 +555,31 @@ class NativeDataSourceStreamWriter(
case class NativeInputPartition(index: Int, bytes: Array[Byte]) extends
InputPartition
object NativeInputPartition {
- def from(partitions: Array[Array[Byte]]): Array[InputPartition] = {
+ def from(partitions: Array[Array[Byte]]): Array[NativeInputPartition] = {
Option(partitions).getOrElse(Array.empty).zipWithIndex.map { case (bytes,
i) =>
NativeInputPartition(i, Option(bytes).getOrElse(Array.emptyByteArray))
}
}
}
+/**
+ * A partition of a micro-batch, with the state that the stream reader
returned with the
+ * partitions of the micro-batch, if any. The state changes with each
micro-batch, so it goes with
+ * the partitions rather than with the reader factory, which an engine may
keep while Spark plans
+ * the next micro-batch.
+ *
+ * @param state the bytes of the state, empty if the stream reader returned
none, or for a state
+ * larger than 64 KiB, their broadcast, so that each executor
fetches it once for the
+ * micro-batch rather than with each task
+ */
+case class NativeMicroBatchPartition(
+ partition: NativeInputPartition,
+ state: Either[Array[Byte], Broadcast[Array[Byte]]])
+ extends InputPartition {
+ /** The state to read the partition with, empty if the stream reader
returned none. */
+ def readerState: Array[Byte] = state.fold(identity, _.value)
+}
+
/** The message returned by the data writer of a task of a native library. */
case class NativeWriterCommitMessage(bytes: Array[Byte]) extends
WriterCommitMessage
@@ -505,6 +606,7 @@ object NativeWriterCommitMessage {
* library.
*
* @param kind the kind of the scan, `NativeLibrary.READ_BATCH` or
`READ_MICRO_BATCH`
+ * @param partition the partition, whose index in the scan the library gets as
`partition_id`
* @param schema the schema to read
* @param timeZoneId the time zone of the timestamps of the schema to read, as
the reader of the
* library got it on the driver
@@ -514,7 +616,7 @@ class NativePartitionReader(
name: String,
kind: Int,
readerState: Array[Byte],
- partition: Array[Byte],
+ partition: NativeInputPartition,
schema: StructType,
timeZoneId: String)
extends PartitionReader[ColumnarBatch] {
@@ -540,7 +642,8 @@ class NativePartitionReader(
// Synchronized so that a close on another thread sees the task and the
vectors.
synchronized {
try {
- openStream()
+ NativePartitionReader.openStream(location, name, kind, readerState,
partition, schema,
+ timeZoneId, stream.memoryAddress())(task = _)
root = VectorSchemaRoot.create(importSchema(), allocator)
checkSchema()
batch = new
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
@@ -551,25 +654,7 @@ class NativePartitionReader(
}
}
- private def openStream(): Unit = {
- val library = NativeLibraries.get(location)
- task = NativeTask.create()
- NativeArrow.exportSchema(schema, timeZoneId) { schemaAddress =>
- call(name, "read") {
- library.read(task, kind, schemaAddress, readerState, partition,
stream.memoryAddress())
- }
- }
- }
-
private def importSchema(): Schema = {
- // Arrow calls the callbacks of the stream without checking them, and
calling NULL would crash
- // the JVM.
- val snapshot = stream.snapshot()
- if (snapshot.release == 0 || snapshot.get_schema == 0 || snapshot.get_next
== 0 ||
- snapshot.get_last_error == 0) {
- throw error(name, "read", "The library did not initialize the stream, or
a callback of " +
- "the stream is NULL.")
- }
val arrowSchema = ArrowSchema.allocateNew(allocator)
try {
call(name, "read")(stream.getSchema(arrowSchema))
@@ -934,6 +1019,132 @@ class NativePartitionReader(
}
}
+object NativePartitionReader {
+ import NativeCalls._
+
+ /**
+ * Initializes the stream at `streamAddress` with the data of a partition
for an engine, see
+ * [[SupportsArrowStreamRead]]: the context of its task is closed once the
stream is released.
+ * It first checks the arguments of `openArrowStream`, before it looks at
the partition, which
+ * must be a `P`, the class of the partitions of the scan of the reader
factory. It then reads
+ * the partition with the state and the partition of the library that
`stateAndPartition`
+ * returns for it.
+ */
+ private[ffi] def openEngineStream[P <: InputPartition: ClassTag](
+ location: NativeLibraryLocation,
+ name: String,
+ kind: Int,
+ partition: InputPartition,
+ schema: StructType,
+ timeZoneId: String,
+ streamAddress: Long)(
+ stateAndPartition: P => (Array[Byte], NativeInputPartition)): Unit = {
+ val context = TaskContext.get()
+ if (context != null) {
+ // A task that was killed gets the error of its kill, as Spark reports
it.
+ context.killTaskIfInterrupted()
+ }
+ // A task that failed is over too, also while its listeners run.
+ if (context == null || context.isFailed() || context.isCompleted()) {
+ throw new IllegalStateException("openArrowStream must be called in the
running task that " +
+ "reads the partition, on a thread that has its TaskContext.")
+ }
+ if (streamAddress == 0 ||
ArrowArrayStream.wrap(streamAddress).snapshot().release != 0) {
+ throw new IllegalArgumentException(
+ "openArrowStream needs the address of a stream whose release callback
is NULL.")
+ }
+ val (readerState, nativePartition) = partition match {
+ case expected: P => stateAndPartition(expected)
+ case _ =>
+ throw new IllegalArgumentException("openArrowStream needs a partition
of the scan of " +
+ s"the reader factory, a ${classTag[P].runtimeClass.getSimpleName},
but got " +
+ s"${Option(partition).map(_.getClass.getName).orNull}.")
+ }
+ var task: NativeLibrary.Task = null
+ try {
+ openStream(location, name, kind, readerState, nativePartition, schema,
timeZoneId,
+ streamAddress)(task = _)
+ // As a failure of read, a failure to create the callbacks of the
streams of engines, such as
+ // a link error, is an error of the data source.
+ call(name, "read")(NativeLibrary.wrapEngineStream(streamAddress, task,
name))
+ } catch {
+ case e: Throwable =>
+ releaseOnFailure(e, streamAddress, Option(task))
+ throw e
+ }
+ }
+
+ /**
+ * Initializes the Arrow C stream at `streamAddress`, whose release callback
is NULL, with the
+ * data of a partition, by calling read of the library in the running task.
It gives the context
+ * of the task to `setTask` before read, since the library may use it until
the stream is
+ * released, also if this fails: the caller then releases the stream, and
closes the context
+ * after it, as the reader of Spark closes both.
+ *
+ * @param partition the partition, whose index in the scan the library gets
as `partition_id`,
+ * rather than the partition of the task: a task may read
several partitions of
+ * a scan, or of several scans, as after a coalesce or a
union
+ */
+ private def openStream(
+ location: NativeLibraryLocation,
+ name: String,
+ kind: Int,
+ readerState: Array[Byte],
+ partition: NativeInputPartition,
+ schema: StructType,
+ timeZoneId: String,
+ streamAddress: Long)(
+ setTask: NativeLibrary.Task => Unit): Unit = {
+ val library = NativeLibraries.get(location)
+ val stream = ArrowArrayStream.wrap(streamAddress)
+ val task = NativeTask.create(Some(partition.index), read = true)
+ setTask(task)
+ NativeArrow.exportSchema(schema, timeZoneId) { schemaAddress =>
+ call(name, "read") {
+ library.read(task, kind, schemaAddress, readerState, partition.bytes,
streamAddress)
+ }
+ }
+ // Arrow calls the callbacks of the stream without checking them, and
calling NULL would crash
+ // the JVM.
+ val snapshot = stream.snapshot()
+ if (snapshot.release == 0 || snapshot.get_schema == 0 || snapshot.get_next
== 0 ||
+ snapshot.get_last_error == 0) {
+ throw error(name, "read", "The library did not initialize the stream, or
a callback of the " +
+ "stream is NULL.")
+ }
+ }
+
+ /**
+ * After `failure`, releases the stream of an engine at `streamAddress`, if
the library
+ * initialized it, before the task that it may use, if any, and marks it
released in case its
+ * release callback did not. If the stream cannot be released, the task is
not closed either,
+ * since the library may still use it: both then leak, rather than letting
the release of the
+ * stream use a freed task. A failure to clean up is added to `failure`.
+ */
+ private def releaseOnFailure(
+ failure: Throwable,
+ streamAddress: Long,
+ task: Option[NativeLibrary.Task]): Unit = {
+ val released = try {
+ val stream = ArrowArrayStream.wrap(streamAddress)
+ stream.release()
+ stream.markReleased()
+ true
+ } catch {
+ case e: Throwable =>
+ failure.addSuppressed(e)
+ false
+ }
+ if (released) {
+ task.foreach { task =>
+ try task.close() catch {
+ case e: Throwable => failure.addSuppressed(e)
+ }
+ }
+ }
+ }
+}
+
/**
* Writes the batches of a task to the data writer of a native library. The
data writer gets the
* schema of the data once, when it is created, and then only the arrays of
the batches.
@@ -1037,22 +1248,70 @@ class NativeDataWriter(
/** Creates the `SparkDsTaskContext` of the running task. */
private[ffi] object NativeTask {
+ // The flag of each running task that reads, set once the task failed or
completed by a single
+ // pair of listeners for all its reads, which then also remove it: a task
that reads many
+ // partitions, as after a coalesce, registers them once. The contexts of the
reads keep their
+ // flag, so they still see it once it is removed. A task is the key rather
than its attempt ID,
+ // which is 0 for each context of `TaskContext.empty()`. The keys are weak,
compared by identity,
+ // so that a context that never ends, such as one that a test sets on its
thread, does not stay.
+ private val endedTasks = new MapMaker().weakKeys().makeMap[TaskContext,
AtomicBoolean]()
+
+ /** Returns the flag that is set once the given task failed or completed. */
+ private def endedFlag(taskContext: TaskContext): AtomicBoolean = {
+ Option(endedTasks.get(taskContext)).getOrElse {
+ val ended = new AtomicBoolean(false)
+ Option(endedTasks.putIfAbsent(taskContext, ended)).getOrElse {
+ // A listener runs right away if the task already failed or completed.
+ def end(): Unit = {
+ ended.set(true)
+ endedTasks.remove(taskContext, ended)
+ }
+ taskContext.addTaskFailureListener((_, _) => end())
+ taskContext.addTaskCompletionListener[Unit](_ => end())
+ ended
+ }
+ }
+ }
+
+ // The number of tasks whose reads have a flag, for tests.
+ private[ffi] def tasksWithEndedFlags: Int = endedTasks.size()
+
/**
- * @param partitionId the partition that the task reads or writes, by
default the partition of
- * the task
+ * @param partitionId the `partition_id` of the context: the index of the
partition in the scan
+ * for a read, and by default the partition of the task
* @param taskAttemptId the ID of the task attempt, by default the one of
the task
+ * @param read whether the context is the one of a read, which the library
may use after the
+ * task ended: an engine may read the stream of a partition
then, and a thread that
+ * the task does not wait for may be in get_next once it
completes, as with
+ * `PipedRDD`. `is_interrupted` then also reports that the task
failed or completed,
+ * and the context does not keep the task alive. A data writer
only runs within its
+ * task, whose kill `is_interrupted` reports. Outside of a task,
it reports nothing.
*/
def create(
partitionId: Option[Int] = None,
- taskAttemptId: Option[Long] = None): NativeLibrary.Task = {
+ taskAttemptId: Option[Long] = None,
+ read: Boolean = false): NativeLibrary.Task = {
val context = Option(TaskContext.get())
+ val interrupted: BooleanSupplier = context match {
+ case Some(taskContext) if read =>
+ // Set once the task failed or completed, by listeners, so that a
check only reads
+ // volatile fields, unlike `isFailed()` and `isCompleted()`, which
lock the task.
+ val ended = endedFlag(taskContext)
+ val task = new WeakReference(taskContext)
+ () => ended.get() || Option(task.get()).forall(_.isInterrupted())
+ case Some(taskContext) =>
+ () => taskContext.isInterrupted()
+ case None =>
+ // Outside of a task, as in tests, nothing interrupts the library.
+ () => false
+ }
new NativeLibrary.Task(
context.map(_.stageId()).getOrElse(-1),
context.map(_.stageAttemptNumber()).getOrElse(-1),
partitionId.orElse(context.map(_.partitionId())).getOrElse(-1),
context.map(_.attemptNumber()).getOrElse(0),
taskAttemptId.orElse(context.map(_.taskAttemptId())).getOrElse(-1L),
- () => context.exists(_.isInterrupted()))
+ interrupted)
}
}
diff --git
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index 4fcb530179cd..87551a695b31 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -39,16 +39,17 @@
// validity buffer, a null_count of -1 instead. With the
option
// "wait_for_interrupt", the stream of a partition creates
the file
// <wait_for_interrupt>.waiting, waits until its task is
killed, and then creates
-// the file <wait_for_interrupt> and fails. With the option
"slow_read", the
-// get_next that creates the file <slow_read>.reading, the
first one, then waits
-// until the file <slow_read>.continue exists, for at most a
minute; with the
-// option "fail" set to "slow_read", each get_next then
fails. After the wait,
-// it creates the file <slow_read>.interrupted with what
is_interrupted returns
-// for the task context, "true" for one that Spark freed
meanwhile. Releasing
-// the stream creates the file <slow_read>, which says
whether Spark released it
-// "during get_next", which it must not, or "after get_next"
otherwise. With
-// the option "ignore_read_schema", read returns all the
columns of the reader,
-// rather than only those of the schema to read.
+// the file <wait_for_interrupt> and fails. With the option
"released_path",
+// releasing the stream of a partition creates that file.
With the option
+// "slow_read", the get_next that creates the file
<slow_read>.reading, the first
+// one, then waits until the file <slow_read>.continue
exists, for at most a
+// minute; with the option "fail" set to "slow_read", each
get_next then fails.
+// After the wait, it creates the file
<slow_read>.interrupted with what
+// is_interrupted returns for the task context, "true" once
the task ended.
+// Releasing the stream creates the file <slow_read>, which
says
+// whether Spark released it "during get_next", which it
must not, or "after
+// get_next" otherwise. With the option
"ignore_read_schema", read returns all
+// the columns of the reader, rather than only those of the
schema to read.
// native_sink Writes the rows as CSV files under the directory in the
option "path". With
// the option "commit_messages", a data writer passes its
commit message that
// many times. Creating and closing a data writer create a
file named after
@@ -67,7 +68,8 @@
// read to that file, and with "ranges_path", each call to
stream_partitions
// appends the start and the end offsets that it got to that
file, as a line.
// The partitions and the state that stream_partitions
returns have the number of
-// the call, which read checks.
+// the call, which read checks. With the option
"state_padding", the state has
+// that many more bytes.
//
// native_range supports batch reads, native_counter micro-batch streaming
reads and native_sink
// batch and streaming writes, which may replace the existing data. With the
option
@@ -919,6 +921,7 @@ struct Stream {
const SparkDsTaskContext* task;
std::string wait_for_interrupt;
std::string last_error;
+ std::string released_path;
std::string slow_read;
bool fail_slow_read;
// task->is_interrupted, kept to call it without reading the task context,
which a regression
@@ -976,9 +979,8 @@ int streamGetNext(ArrowArrayStream* stream, ArrowArray*
out) {
for (int i = 0; i < 6000 && stat(go.c_str(), &info) != 0; i++) {
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
- // The task context must still be valid, as a library that checks it
here expects. Spark
- // only looks its address up, and answers true for a context that it
freed, unless a new
- // one got the same address, which no other read in these tests
creates meanwhile.
+ // The task context must still be valid, as a library that checks it
here expects, and
+ // it tells a read that its task ended.
bool interrupted = state->is_interrupted(state->task);
writeFile(state->slow_read + ".interrupted", interrupted ? "true" :
"false");
}
@@ -1002,6 +1004,11 @@ const char* streamGetLastError(ArrowArrayStream* stream)
{
void streamRelease(ArrowArrayStream* stream) {
auto* state = static_cast<Stream*>(stream->private_data);
+ try {
+ if (!state->released_path.empty()) writeFile(state->released_path, "");
+ } catch (...) {
+ // No exception may cross the interface, and release cannot fail.
+ }
stream->release = nullptr;
if (!state->slow_read.empty()) {
bool during = state->in_get_next;
@@ -1200,7 +1207,7 @@ int createReader(void* handle, struct ArrowSchema*
schema, const SparkDsReaderIn
reader->state["partitions"] = get(source->options, "partitions", "2");
reader->state["batch_size"] = get(source->options, "batch_size", "1000");
for (const char* option : {"fail", "slice", "short", "null_rows",
"wait_for_interrupt",
- "task_path", "states", "stateless", "slow_read",
+ "task_path", "states", "stateless",
"slow_read", "released_path",
"ignore_read_schema"}) {
reader->state[option] = get(source->options, option, "");
}
@@ -1246,9 +1253,11 @@ int createStreamReader(void* handle, struct ArrowSchema*
schema,
reader->state["batch_size"] = get(source->options, "batch_size", "1000");
for (const char* option :
{"commit_path", "bad_offset", "bad_initial_offset", "closed_path",
"starts_path",
- "initial_offsets_path", "ranges_path", "states", "stateless",
"fail"}) {
+ "initial_offsets_path", "ranges_path", "states", "stateless",
"task_path", "fail"}) {
reader->state[option] = get(source->options, option, "");
}
+ std::string padding = get(source->options, "state_padding", "");
+ if (!padding.empty()) reader->state["padding"] =
std::string(std::stoll(padding), 'x');
*out = reader;
});
}
@@ -1495,8 +1504,11 @@ int readPartition(const SparkDsTaskContext* task, const
SparkDsReadInfo* info,
get(state, "slice", "") == "true",
get(state, "short", ""),
get(state, "null_rows", ""), task,
get(state, "wait_for_interrupt", ""), "",
+ get(state, "released_path", ""),
get(state, "slow_read", ""),
get(state, "fail", "") == "slow_read",
task->is_interrupted};
+ // Initializes the stream without get_next, which Spark must reject, and
release.
+ if (get(state, "fail", "") == "null_get_next") stream->get_next = nullptr;
});
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 94e2b5a78b88..92ef0d87de92 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
import org.apache.spark.sql.connector.catalog.{Table, TableCapability,
TableProvider}
import org.apache.spark.sql.connector.expressions.Transform
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader,
PartitionReaderFactory}
import org.apache.spark.sql.connector.read.streaming.{ReadLimit,
SupportsAdmissionControl}
import org.apache.spark.sql.connector.write.{DataWriter,
PhysicalWriteInfoImpl, WriterCommitMessage}
import org.apache.spark.sql.execution.{ApplyColumnarRulesAndInsertTransitions,
ColumnarRule, ColumnarToRowExec, RowToColumnConverter, SparkPlan}
@@ -1110,10 +1110,8 @@ class TestRangeStreamReader(end: Long, schema:
StructType, table: String)
Array(TestRangePartition(offset(start), offset(end)))
}
- override def read(partition: InputPartition): PartitionReader[ColumnarBatch]
= {
- val range = partition.asInstanceOf[TestRangePartition]
- new TestBatchReader(range.lo, range.hi, schema)
- }
+ override def readerFactory(): PartitionReaderFactory =
+ new TestRangeReaderFactory(schema, arrowBatchSize = 0, ownedBatches =
false)
}
object TestRangeStreamReader {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index ad1982253cd2..1928bbac3722 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -16,7 +16,7 @@
*/
package org.apache.spark.sql.execution.datasources.v2.ffi
-import java.io.File
+import java.io.{ByteArrayOutputStream, File, ObjectOutputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.{AccessDeniedException, Files, NoSuchFileException, Paths}
import java.nio.file.StandardCopyOption.REPLACE_EXISTING
@@ -30,19 +30,22 @@ import java.util.zip.{ZipException, ZipFile}
import scala.collection.mutable.ArrayBuffer
import scala.jdk.CollectionConverters._
+import scala.reflect.ClassTag
-import org.apache.arrow.vector.BigIntVector
+import org.apache.arrow.c.{ArrowArray, ArrowArrayStream, ArrowSchema,
CDataDictionaryProvider, Data}
+import org.apache.arrow.memory.BufferAllocator
+import org.apache.arrow.vector.{BigIntVector, VectorSchemaRoot}
import org.scalatest.time.SpanSugar._
-import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException,
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable,
SparkUnsupportedOperationException, TestUtils}
+import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException,
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable,
SparkUnsupportedOperationException, TaskContext, TaskKilledException, TestUtils}
import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils,
QueryTest, Row}
import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
import org.apache.spark.sql.classic.{DataFrame, SparkSession}
import org.apache.spark.sql.connector.catalog.SupportsRead
-import org.apache.spark.sql.connector.read.SupportsPushDownRequiredColumns
-import org.apache.spark.sql.connector.read.streaming.{Offset, ReadLimit,
SupportsAdmissionControl}
+import org.apache.spark.sql.connector.read.{InputPartition,
SupportsArrowStreamRead, SupportsPushDownRequiredColumns}
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream,
Offset, ReadLimit, SupportsAdmissionControl}
import org.apache.spark.sql.connector.write.PhysicalWriteInfoImpl
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.execution.datasources.DataSource
@@ -58,6 +61,7 @@ import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types._
import org.apache.spark.sql.util.{ArrowUtils, CaseInsensitiveStringMap}
import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch}
+import org.apache.spark.unsafe.Platform
import org.apache.spark.util.{ThreadUtils, Utils}
/**
@@ -1669,6 +1673,497 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
.mode("append")
.save()
check(writeTasks)
+
+ // A read gets the index of its partition in the scan, also when a task
reads several
+ // partitions, or the partitions of several scans: the task attempt that
read each partition.
+ def readers(dir: File): Map[Int, Long] = dir.listFiles().map { file =>
+ file.getName.toInt -> Files.readString(file.toPath).split(",")(5).toLong
+ }.toMap
+ val (first, second) = (Utils.createTempDir(), Utils.createTempDir())
+ checkRows(
+ range(session, "end" -> "6", "task_path" -> first.getPath).select("id")
+ .union(range(session, "end" -> "6", "task_path" ->
second.getPath).select("id")),
+ (0L until 6L).flatMap(i => Seq(Row(i), Row(i))))
+ assert(readers(first).keySet == Set(0, 1) && readers(second).keySet ==
Set(0, 1))
+ val coalesced = Utils.createTempDir()
+ checkRows(range(session, "end" -> "6", "task_path" ->
coalesced.getPath).select("id")
+ .coalesce(1), (0L until 6L).map(Row(_)))
+ assert(readers(coalesced).keySet == Set(0, 1) &&
readers(coalesced).values.toSet.size == 1)
+ // So does a read of a micro-batch.
+ val (firstStream, secondStream) = (Utils.createTempDir(),
Utils.createTempDir())
+ def counter(dir: File): DataFrame =
session.readStream.format("native_counter")
+ .option("max_offset", "4").option("task_path", dir.getPath).load()
+ val query =
counter(firstStream).union(counter(secondStream)).writeStream.format("noop").start()
+ try query.processAllAvailable() finally query.stop()
+ assert(readers(firstStream).keySet == Set(0) &&
readers(secondStream).keySet == Set(0))
+ }
+
+ nativeTest("open the partitions of batch and micro-batch scans as Arrow C
streams") {
+ val session = newSession(defaultPackage)
+ // The reader factory and the partitions of the batch scan of a query,
which an engine gets
+ // from its plan.
+ def plannedScan(df: DataFrame): (SupportsArrowStreamRead,
Seq[InputPartition]) = {
+ val scan = df.queryExecution.executedPlan.collectFirst { case s:
BatchScanExec => s }.get
+ (scan.readerFactory.asInstanceOf[SupportsArrowStreamRead],
scan.inputPartitions)
+ }
+ // The batch scan of only some of the columns from a library that does not
prune the others
+ // still reads only them through its streams, since read returns exactly
the columns of its
+ // schema to read.
+ val readOnly = newSession(defaultPackage, createPackage(readOnlyLibrary,
dataSources("ro_")))
+ val (projectedFactory, projectedPartitions) =
+ plannedScan(readOnly.read.format("ro_native_range").option("end",
"4").load().select("id"))
+ val projectedReads = readOnly.artifactManager.withResources {
+ readOnly.sparkContext.parallelize(Seq(0), 1).flatMap { _ =>
+ projectedPartitions.map { partition =>
+ val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+ try {
+ projectedFactory.openArrowStream(partition, stream.memoryAddress())
+ try TestArrowEngine.readNamesAndIdsInPlace(stream) finally
stream.release()
+ } finally {
+ stream.close()
+ }
+ }
+ }.collect().toSeq
+ }
+ assert(projectedReads.map(_._1).distinct == Seq(Seq("id")))
+ assert(projectedReads.flatMap(_._2).sorted == (0L until 4L))
+ assert(NativeLibrary.openEngineStreams() == 0)
+ // Runs `f` in a task that only gets the files of the session. It is the
task of the partition
+ // 1, so that the partition of the scan that a read gets is not the
partition of its task.
+ def inTask[T: ClassTag](f: => T): T =
session.artifactManager.withResources {
+ session.sparkContext.parallelize(Seq(0, 1), 2).flatMap(i => if (i == 1)
Seq(f) else Nil)
+ .collect().head
+ }
+ import TestArrowEngine.{callbacks, readIds, readIdsInPlace}
+ // Reads the given partitions in a single task, as an engine does.
+ def readAll(factory: SupportsArrowStreamRead, partitions:
Seq[InputPartition]): Seq[Long] = {
+ inTask {
+ partitions.flatMap { partition =>
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine",
0, Long.MaxValue)
+ try {
+ val stream = ArrowArrayStream.allocateNew(allocator)
+ try factory.openArrowStream(partition, stream.memoryAddress())
catch {
+ case e: Throwable =>
+ stream.close()
+ throw e
+ }
+ readIds(stream, allocator)
+ } finally {
+ allocator.close()
+ }
+ }
+ }
+ }
+ // Checks that `f` opens at most `delta` more task contexts than it
closes. The count is global
+ // to the JVM, where the tasks of earlier tests may still close their
contexts, so it is read
+ // right before `f`, and may be lower afterwards.
+ def checkContexts[T](delta: Int)(f: => T): T = {
+ val before = NativeLibrary.openTaskContexts()
+ val result = f
+ assert(NativeLibrary.openTaskContexts() <= before + delta)
+ result
+ }
+ // The size of struct ArrowArrayStream, which the layout test compares
with the header.
+ val streamSize = NativeLibrary.layout().get("ArrowArrayStream").longValue
+
+ // An engine reads both partitions of the scan in a single task.
+ val taskDir = Utils.createTempDir()
+ val (factory, partitions) =
+ plannedScan(range(session, "end" -> "6", "task_path" ->
taskDir.getPath).select("id"))
+ assert(partitions.length == 2)
+ assert(readAll(factory, partitions).sorted == (0L until 6L))
+ // The library got the partition of the scan that it reads, not the one of
the task.
+ val tasks = taskDir.listFiles().map { file =>
+ file.getName.toInt ->
Files.readString(file.toPath).split(",").map(_.toLong).toSeq
+ }.toMap
+ assert(tasks.keySet == Set(0, 1) && tasks.values.map(_(5)).toSet.size ==
1, tasks)
+ assert(NativeLibrary.openEngineStreams() == 0)
+ // The reads of a task share the flag that tells them that the task ended,
with a single pair
+ // of listeners, which remove it once the task ended. The tasks of earlier
tests may still
+ // be ending.
+ eventually(timeout(1.minute))(assert(NativeTask.tasksWithEndedFlags == 0))
+ val flagsInTask = inTask {
+ val ids = partitions.flatMap { partition =>
+ val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+ try {
+ factory.openArrowStream(partition, stream.memoryAddress())
+ try readIdsInPlace(stream) finally stream.release()
+ } finally {
+ stream.close()
+ }
+ }
+ assert(ids.sorted == (0L until 6L))
+ NativeTask.tasksWithEndedFlags
+ }
+ assert(flagsInTask == 1 && NativeTask.tasksWithEndedFlags == 0)
+
+ // The engine may release the stream after its task ended, and on another
thread: Spark frees
+ // the context of the task once the library released the stream.
+ val released = new File(Utils.createTempDir(), "released")
+ val (laterFactory, laterPartitions) =
+ plannedScan(range(session, "end" -> "6", "released_path" ->
released.getPath).select("id"))
+ // Opens a stream in a task, and returns it unreleased, with what `read`
returned from it in
+ // the task.
+ def openInTask[T: ClassTag](factory: SupportsArrowStreamRead, partition:
InputPartition)(
+ read: ArrowArrayStream => T): (Long, T) = inTask {
+ val stream = Platform.allocateMemory(streamSize)
+ Platform.setMemory(stream, 0.toByte, streamSize)
+ factory.openArrowStream(partition, stream)
+ (stream, read(ArrowArrayStream.wrap(stream)))
+ }
+ // The engine reads the stream to its end in the task, but releases it
later.
+ val (address, readInTask) =
+ checkContexts(1)(openInTask(laterFactory,
laterPartitions.head)(readIdsInPlace))
+ // A copy of the stream, as an engine that moves the stream may leave
behind.
+ val copy = Platform.allocateMemory(streamSize)
+ Utils.tryWithSafeFinally {
+ Platform.copyMemory(null, address, null, copy, streamSize)
+ assert(readInTask == (0L until 3L))
+ assert(NativeLibrary.openEngineStreams() == 1 && !released.exists())
+ checkContexts(-1) {
+ ArrowArrayStream.wrap(address).release()
+ assert(NativeLibrary.openEngineStreams() == 0 && released.exists())
+ }
+ // Once the stream is released, the callbacks of the copy fail with
EINVAL, and its release
+ // only marks it released.
+ checkContexts(0) {
+ val array = ArrowArray.allocateNew(ArrowUtils.rootAllocator)
+ val stale = try
intercept[Exception](ArrowArrayStream.wrap(copy).getNext(array)) finally {
+ array.close()
+ }
+ assert(Utils.exceptionString(stale)
+ .contains("The stream was released, or is not a stream of Spark."))
+ ArrowArrayStream.wrap(copy).release()
+ assert(ArrowArrayStream.wrap(copy).snapshot().release == 0)
+ assert(NativeLibrary.openEngineStreams() == 0)
+ }
+ } {
+ // A failure above may leave the stream open, which would keep the
context of its task.
+ if (ArrowArrayStream.wrap(address).snapshot().release != 0) {
+ ArrowArrayStream.wrap(address).release()
+ }
+ Platform.freeMemory(address)
+ Platform.freeMemory(copy)
+ }
+
+ // Once the task ended, its context tells the library that it was
interrupted, so that the
+ // library stops waiting.
+ val marker = new File(Utils.createTempDir(), "interrupted")
+ val (waitingFactory, waitingPartitions) = plannedScan(
+ range(session, "end" -> "6", "wait_for_interrupt" ->
marker.getPath).select("id"))
+ checkContexts(0) {
+ val (waiting, _) = openInTask(waitingFactory, waitingPartitions.head)(_
=> ())
+ val error = intercept[Exception] {
+ readIds(ArrowArrayStream.wrap(waiting), ArrowUtils.rootAllocator)
+ }
+ assert(Utils.exceptionString(error).contains("the task was killed"))
+ assert(Files.readString(marker.toPath) == "interrupted")
+ assert(NativeLibrary.openEngineStreams() == 0)
+ Platform.freeMemory(waiting)
+ }
+
+ // So does the context of a stream that an engine reads in its task once
the task is killed,
+ // although the task did not end.
+ val killedMarker = new File(Utils.createTempDir(), "interrupted")
+ val (killedFactory, killedPartitions) = plannedScan(range(session, "end"
-> "6",
+ "partitions" -> "1", "wait_for_interrupt" ->
killedMarker.getPath).select("id"))
+ val contextsBeforeKill = NativeLibrary.openTaskContexts()
+ val jobGroup = s"native-data-source-${UUID.randomUUID()}"
+ val killError = new AtomicReference[Throwable]()
+ val engineThread = new Thread(() => {
+ session.sparkContext.setJobGroup(jobGroup, "an engine that waits",
interruptOnCancel = true)
+ try {
+ inTask {
+ // Reads the stream on the thread of the task, which waits in
get_next.
+ val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+ try {
+ killedFactory.openArrowStream(killedPartitions.head,
stream.memoryAddress())
+ try readIdsInPlace(stream) finally stream.release()
+ } finally {
+ stream.close()
+ }
+ }
+ } catch {
+ case e: Throwable => killError.set(e)
+ }
+ })
+ engineThread.start()
+ eventually(timeout(1.minute)) {
+ assert(new File(killedMarker.getPath + ".waiting").isFile)
+ }
+ session.sparkContext.cancelJobGroup(jobGroup)
+ engineThread.join(TimeUnit.MINUTES.toMillis(1))
+ assert(!engineThread.isAlive && killError.get() != null)
+ eventually(timeout(1.minute)) {
+ assert(killedMarker.isFile && Files.readString(killedMarker.toPath) ==
"interrupted")
+ assert(NativeLibrary.openEngineStreams() == 0)
+ assert(NativeLibrary.openTaskContexts() <= contextsBeforeKill)
+ }
+
+ // So does the reader factory of a micro-batch scan, with the state of its
micro-batch.
+ val streamTaskDir = Utils.createTempDir()
+ val query =
session.readStream.format("native_counter").option("max_offset", "4")
+ .option("task_path", streamTaskDir.getPath).load()
+ .writeStream.format("noop").start()
+ val microBatchScan = try {
+ query.processAllAvailable()
+
query.asInstanceOf[StreamingQueryWrapper].streamingQuery.lastExecution.executedPlan
+ .collectFirst { case s: MicroBatchScanExec => s }.get
+ } finally {
+ query.stop()
+ }
+ // Only the reads of the engine record their tasks from now on.
+ streamTaskDir.listFiles().foreach(_.delete())
+ val streamFactory =
microBatchScan.readerFactory.asInstanceOf[SupportsArrowStreamRead]
+ val streamPartitions = microBatchScan.inputPartitions.toSeq
+ checkContexts(0) {
+ assert(readAll(streamFactory, streamPartitions).sorted == (0L until 4L))
+ }
+ assert(NativeLibrary.openEngineStreams() == 0)
+ // The library got the partition of the scan, the only one of the
micro-batch.
+ assert(streamTaskDir.listFiles().map(_.getName.toInt).toSet == Set(0))
+ // The partitions of a micro-batch keep the state that the stream reader
returned with them:
+ // a reader factory that an engine keeps reads them with it, also once
Spark planned the next
+ // micro-batch, and so does the reader of Spark.
+ def microBatchStream(options: (String, String)*): MicroBatchStream = {
+ val relation =
session.readStream.format("native_counter").options(options.toMap).load()
+ .queryExecution.analyzed.collectFirst { case relation:
StreamingRelationV2 => relation }.get
+
relation.table.asInstanceOf[SupportsRead].newScanBuilder(relation.extraOptions)
+ .build().toMicroBatchStream(Utils.createTempDir().getPath)
+ }
+ def plan(stream: MicroBatchStream, start: Int, end: Int):
Seq[InputPartition] = {
+ def offset(i: Int): Offset =
stream.deserializeOffset(s"""{"offset":$i}""")
+ stream.planInputPartitions(offset(start), offset(end)).toSeq
+ }
+ def serialized(partition: NativeMicroBatchPartition): Seq[Byte] = {
+ val bytes = new ByteArrayOutputStream()
+ Utils.tryWithResource(new
ObjectOutputStream(bytes))(_.writeObject(partition))
+ bytes.toByteArray.toSeq
+ }
+ val padding = (NativeDataSourceStreamReader.maxInlineStateBytes +
1).toString
+ val stream = microBatchStream("max_offset" -> "4", "state_padding" ->
padding)
+ try {
+ val first = plan(stream, 0, 2)
+ val keptFactory = stream.createReaderFactory()
+ val second = plan(stream, 2, 4)
+ // The executors get a state larger than 64 KiB once for the
micro-batch, as a broadcast that
+ // its partitions keep, rather than with the partition of each task.
+ Seq(first, second).foreach { partitions =>
+ val partition = partitions.head.asInstanceOf[NativeMicroBatchPartition]
+ assert(partition.state.isRight && partition.readerState.length >
padding.toInt)
+
assert(!serialized(partition).containsSlice(partition.readerState.toSeq))
+ }
+ checkContexts(0) {
+ assert(readAll(keptFactory.asInstanceOf[SupportsArrowStreamRead],
first) == Seq(0L, 1L))
+ assert(readAll(keptFactory.asInstanceOf[SupportsArrowStreamRead],
second) == Seq(2L, 3L))
+ val read = inTask {
+ first.flatMap { partition =>
+ val reader = keptFactory.createColumnarReader(partition)
+ try {
+ val ids = ArrayBuffer.empty[Long]
+ while (reader.next()) {
+ val batch = reader.get()
+ (0 until batch.numRows()).foreach(i => ids +=
batch.column(0).getLong(i))
+ }
+ ids.toSeq
+ } finally {
+ reader.close()
+ }
+ }
+ }
+ assert(read == Seq(0L, 1L))
+ }
+ } finally {
+ stream.stop()
+ }
+ // A smaller state, such as a few offsets, goes with the partition of each
task, which costs
+ // less than a broadcast.
+ val small = microBatchStream("max_offset" -> "4")
+ try {
+ val partition = plan(small, 0,
2).head.asInstanceOf[NativeMicroBatchPartition]
+ assert(partition.state.isLeft && partition.readerState.nonEmpty)
+ assert(serialized(partition).containsSlice(partition.readerState.toSeq))
+ checkContexts(0) {
+
assert(readAll(small.createReaderFactory().asInstanceOf[SupportsArrowStreamRead],
+ Seq(partition)) == Seq(0L, 1L))
+ }
+ } finally {
+ small.stop()
+ }
+ // A micro-batch without partitions broadcasts no state, although the
stream reader returned
+ // a large one.
+ val (counterName, counterLocation) =
+ NativeDataSourceRegistry.lookup("native_counter",
session.sessionState.conf).get
+ val streamReader = new NativeDataSource(counterLocation, counterName,
+ new CaseInsensitiveStringMap(Map("max_offset" -> "4", "state_padding"
-> padding).asJava))
+ .streamReader(StructType.fromDDL("id BIGINT"),
Utils.createTempDir().getPath)
+ .asInstanceOf[NativeDataSourceStreamReader]
+ try {
+ def offset(i: Int): String = s"""{"offset":$i}"""
+ assert(streamReader.partitions(offset(0), offset(0)).isEmpty)
+ assert(streamReader.uncommittedStates.isEmpty)
+ // The broadcast of a micro-batch is destroyed once the micro-batch is
committed, and those
+ // of the micro-batches after it stay.
+ assert(streamReader.partitions(offset(0), offset(2)).nonEmpty)
+ assert(streamReader.partitions(offset(2), offset(4)).nonEmpty)
+ assert(streamReader.partitions(offset(4), offset(6)).nonEmpty)
+ val states = streamReader.uncommittedStates
+ assert(states.length == 3 && states.forall(_.isValid))
+ streamReader.commit(offset(2))
+ assert(states.map(_.isValid) == Seq(false, true, true))
+ assert(streamReader.uncommittedStates == states.tail)
+ // Spark may not commit every micro-batch, as with async progress
tracking, so a commit also
+ // destroys the broadcasts of the micro-batches before it, also when the
committed one has no
+ // broadcast, as a micro-batch without partitions, here at an end not
planned before.
+ assert(streamReader.partitions(offset(7), offset(7)).isEmpty)
+ streamReader.commit(offset(7))
+ assert(states.forall(!_.isValid) &&
streamReader.uncommittedStates.isEmpty)
+ // stop() destroys the broadcasts of the micro-batches that are not
committed.
+ assert(streamReader.partitions(offset(7), offset(9)).nonEmpty)
+ val lastState = streamReader.uncommittedStates
+ assert(lastState.length == 1 && lastState.head.isValid)
+ streamReader.stop()
+ assert(!lastState.head.isValid && streamReader.uncommittedStates.isEmpty)
+ } finally {
+ streamReader.stop()
+ }
+ // A micro-batch without a state has no bytes to read the partitions with.
+ val stateless = microBatchStream("max_offset" -> "4", "states" -> "0")
+ try {
+ val partitions = plan(stateless, 0,
2).map(_.asInstanceOf[NativeMicroBatchPartition])
+ assert(partitions.map(_.state.isLeft) == Seq(true) &&
partitions.head.readerState.isEmpty)
+ } finally {
+ stateless.stop()
+ }
+
+ // It runs in the running task, with a stream whose release callback is
NULL.
+ val outside = intercept[IllegalStateException] {
+ factory.openArrowStream(partitions.head, 0)
+ }
+ assert(outside.getMessage.contains("must be called in the running task"))
+ val completed = intercept[Exception] {
+ session.artifactManager.withResources {
+ session.sparkContext.parallelize(Seq(0), 1).foreach { _ =>
+ TaskContext.get().addTaskCompletionListener[Unit] { _ =>
+ factory.openArrowStream(partitions.head, 0)
+ }
+ }
+ }
+ }
+ assert(Utils.exceptionString(completed).contains("must be called in the
running task"))
+ // Nor in a task that failed, while its failure listeners run.
+ val failed = intercept[Exception] {
+ session.artifactManager.withResources {
+ session.sparkContext.parallelize(Seq(0), 1).foreach { _ =>
+ TaskContext.get().addTaskFailureListener { (_, _) =>
+ factory.openArrowStream(partitions.head, 0)
+ }
+ throw new IllegalStateException("injected failure of the task")
+ }
+ }
+ }
+ assert(Utils.exceptionString(failed).contains("injected failure of the
task"))
+ assert(Utils.exceptionString(failed).contains("must be called in the
running task"))
+ // A task that was killed gets the error of its kill.
+ val killedContext = TaskContext.empty()
+ killedContext.markInterrupted("killed for the test")
+ TaskContext.setTaskContext(killedContext)
+ val killed = try intercept[TaskKilledException] {
+ factory.openArrowStream(partitions.head, 0)
+ } finally {
+ TaskContext.unset()
+ }
+ assert(killed.reason == "killed for the test")
+ val noStream =
intercept[Exception](inTask(factory.openArrowStream(partitions.head, 0)))
+ assert(Utils.exceptionString(noStream).contains("whose release callback is
NULL"))
+ // The arguments are checked before the partition, which must be of the
kind of the scan of the
+ // factory: a partition of a micro-batch fails for a batch scan, and the
other way around.
+ val outsideWithOtherKind = intercept[IllegalStateException] {
+ streamFactory.openArrowStream(partitions.head, 0)
+ }
+ assert(outsideWithOtherKind.getMessage.contains("must be called in the
running task"))
+ val otherKinds = checkContexts(0) {
+ inTask {
+ Seq(streamFactory -> partitions.head, factory ->
streamPartitions.head).map {
+ case (kindFactory, partition) =>
+ val stream = Platform.allocateMemory(streamSize)
+ try {
+ Platform.setMemory(stream, 0.toByte, streamSize)
+ kindFactory.openArrowStream(partition, stream)
+ None
+ } catch {
+ case e: IllegalArgumentException => Some(e.getMessage)
+ } finally {
+ Platform.freeMemory(stream)
+ }
+ }
+ }
+ }
+ assert(otherKinds == Seq(
+ Some("openArrowStream needs a partition of the scan of the reader
factory, a " +
+ s"NativeMicroBatchPartition, but got
${classOf[NativeInputPartition].getName}."),
+ Some("openArrowStream needs a partition of the scan of the reader
factory, a " +
+ s"NativeInputPartition, but got
${classOf[NativeMicroBatchPartition].getName}.")))
+ // A stream whose release callback is not NULL, such as an open one, is
left as it is.
+ val (rejected, unchanged, ids) = checkContexts(0) {
+ inTask {
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine",
0, Long.MaxValue)
+ try {
+ val stream = ArrowArrayStream.allocateNew(allocator)
+ factory.openArrowStream(partitions.head, stream.memoryAddress())
+ val before = callbacks(stream)
+ val rejected = try {
+ factory.openArrowStream(partitions.head, stream.memoryAddress())
+ None
+ } catch {
+ case e: IllegalArgumentException => Some(e.getMessage)
+ }
+ (rejected, callbacks(stream) == before, readIds(stream, allocator))
+ } finally {
+ allocator.close()
+ }
+ }
+ }
+ assert(rejected.exists(_.contains("whose release callback is NULL")),
rejected)
+ assert(unchanged && ids == (0L until 3L))
+
+ // If it fails, the stream is released, including the stream of the
library that Spark rejects.
+ Seq(
+ "open" -> "injected failure in open",
+ "uninitialized" ->
+ "The library did not initialize the stream, or a callback of the
stream is NULL.",
+ "null_get_next" ->
+ "The library did not initialize the stream, or a callback of the
stream is NULL."
+ ).foreach { case (fail, msg) =>
+ val failReleased = new File(Utils.createTempDir(), "released")
+ val (failing, failingPartitions) =
+ plannedScan(range(session, "fail" -> fail, "released_path" ->
failReleased.getPath))
+ val e = checkContexts(0) {
+ intercept[Exception] {
+ inTask {
+ val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+ try {
+ failing.openArrowStream(failingPartitions.head,
stream.memoryAddress())
+ } catch {
+ // Keeps the failure, but not as the cause, which findError
would find.
+ case e: Throwable if stream.snapshot().release != 0 =>
+ val notReleased = new IllegalStateException(
+ s"The stream is not released after the failure $fail.")
+ notReleased.addSuppressed(e)
+ throw notReleased
+ } finally {
+ stream.close()
+ }
+ }
+ }
+ }
+ checkError(
+ exception = findError(e, "NATIVE_DATA_SOURCE_ERROR"),
+ condition = "NATIVE_DATA_SOURCE_ERROR",
+ parameters = Map("action" -> "read", "name" -> "native_range", "msg"
-> msg))
+ assert(failReleased.exists() == (fail == "null_get_next"), fail)
+ assert(NativeLibrary.openEngineStreams() == 0)
+ }
}
nativeTest("tell a read that its task was killed") {
@@ -1678,6 +2173,7 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
"wait_for_interrupt" -> marker.getPath)
val jobGroup = s"native-data-source-${UUID.randomUUID()}"
val error = new AtomicReference[Throwable]()
+ val contexts = NativeLibrary.openTaskContexts()
val thread = new Thread(() => {
session.sparkContext.setJobGroup(jobGroup, "a read that waits",
interruptOnCancel = true)
try df.collect() catch {
@@ -1694,9 +2190,12 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
thread.join(TimeUnit.MINUTES.toMillis(1))
assert(!thread.isAlive)
assert(error.get() != null)
- // The job fails right away, and its task ends once the library sees that
it was killed.
+ // The job fails right away, and its task ends once the library sees that
it was killed. The
+ // test also waits until the task closed the context of its read, which
the tests that count
+ // the contexts would otherwise count.
eventually(timeout(1.minute)) {
assert(marker.isFile && Files.readString(marker.toPath) == "interrupted")
+ assert(NativeLibrary.openTaskContexts() <= contexts)
}
}
@@ -1706,6 +2205,10 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
val marker = new File(Utils.createTempDir(), "released")
val reading = marker.getPath + ".reading"
val df = range(session, "end" -> "10", "partitions" -> "1", "slow_read" ->
marker.getPath)
+ // The contexts of the tasks of earlier tests are closed first, so that
the counts below are
+ // exactly those of this test, also the one of the context that stays open.
+ eventually(timeout(1.minute))(assert(NativeLibrary.openTaskContexts() ==
0))
+ val contexts = 0
try {
// PipedRDD reads the scan on a thread of its own, which the task does
not wait for. The
// task completes without reading the output of the command, once that
thread is in
@@ -1723,16 +2226,18 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
assert(count == 0)
assert(new File(reading).isFile, "get_next was not called.")
if (marker.isFile) fail(s"The stream was released:
${Files.readString(marker.toPath)}.")
+ // The task completed, but its context stays valid until the stream is
released.
+ assert(NativeLibrary.openTaskContexts() == contexts + 1)
} finally {
new File(marker.getPath + ".continue").createNewFile()
}
- // The thread in get_next releases the stream once get_next returns.
+ // The thread in get_next releases the stream once get_next returns, and
then the context.
eventually(timeout(1.minute)) {
assert(marker.isFile && Files.readString(marker.toPath) == "after
get_next")
+ assert(NativeLibrary.openTaskContexts() == contexts)
}
- // The task context was still valid once get_next continued: Spark answers
that a context
- // that it freed is interrupted, and this task was not.
- assert(Files.readString(new File(marker.getPath + ".interrupted").toPath)
== "false")
+ // The read learned that its task ended, so that it may stop reading.
+ assert(Files.readString(new File(marker.getPath + ".interrupted").toPath)
== "true")
}
nativeTest("close a reader while another thread reads it") {
@@ -1761,7 +2266,8 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
case e: ExecutionException => Left(e.getCause)
}
assert(Files.readString(marker.toPath) == "after get_next")
- // The task context was still valid once get_next continued, as in the
test above.
+ // The context was still valid once get_next continued: Spark answers
that a context that
+ // it freed is interrupted, and a read outside of a task never is.
assert(Files.readString(new File(marker.getPath +
".interrupted").toPath) == "false")
// A reader that was closed reads nothing more, and closing it again
does not fail.
assert(!reader.next())
@@ -2556,3 +3062,79 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
check(noInitLibrary, "ni_", "The library does not export the function
spark_ds_init.")
}
}
+
+/** What an engine does with the Arrow C stream of a partition of a native
scan. */
+object TestArrowEngine {
+ /** The callbacks and the private data of a stream. */
+ def callbacks(stream: ArrowArrayStream): Seq[Long] = {
+ val snapshot = stream.snapshot()
+ Seq(snapshot.get_schema, snapshot.get_next, snapshot.get_last_error,
snapshot.release,
+ snapshot.private_data)
+ }
+
+ /**
+ * Reads the ids of a stream to its end on the calling thread, without
moving or releasing it, as
+ * an engine that releases the stream later does.
+ */
+ def readIdsInPlace(stream: ArrowArrayStream): Seq[Long] =
readNamesAndIdsInPlace(stream)._2
+
+ /** The names of the columns of a stream and its ids, read as
`readIdsInPlace` reads them. */
+ def readNamesAndIdsInPlace(stream: ArrowArrayStream): (Seq[String],
Seq[Long]) = {
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine", 0,
Long.MaxValue)
+ val dictionaries = new CDataDictionaryProvider()
+ try {
+ val schema = ArrowSchema.allocateNew(allocator)
+ val root = try {
+ stream.getSchema(schema)
+ VectorSchemaRoot.create(Data.importSchema(allocator, schema,
dictionaries), allocator)
+ } finally {
+ schema.close()
+ }
+ try {
+ val ids = ArrayBuffer.empty[Long]
+ var more = true
+ while (more) {
+ // A new array for each batch, since the import closes it, as in
NativePartitionReader.
+ val array = ArrowArray.allocateNew(allocator)
+ try {
+ stream.getNext(array)
+ // The stream ends with a released array.
+ more = array.snapshot().release != 0
+ if (more) {
+ Data.importIntoVectorSchemaRoot(allocator, array, root,
dictionaries)
+ val vector = root.getVector("id").asInstanceOf[BigIntVector]
+ (0 until root.getRowCount).foreach(i => ids += vector.get(i))
+ }
+ } finally {
+ array.close()
+ }
+ }
+ (root.getSchema.getFields.asScala.map(_.getName).toSeq, ids.toSeq)
+ } finally {
+ root.close()
+ }
+ } finally {
+ dictionaries.close()
+ allocator.close()
+ }
+ }
+
+ /** Reads the ids of a stream on another thread, as an engine may, and
releases the stream. */
+ def readIds(stream: ArrowArrayStream, allocator: BufferAllocator): Seq[Long]
= {
+ // Moves the stream into a reader, which releases it once closed.
+ val reader = Data.importArrayStream(allocator, stream)
+ try {
+ ThreadUtils.runInNewThread("engine") {
+ val ids = ArrayBuffer.empty[Long]
+ while (reader.loadNextBatch()) {
+ val root = reader.getVectorSchemaRoot
+ val vector = root.getVector("id").asInstanceOf[BigIntVector]
+ (0 until root.getRowCount).foreach(i => ids += vector.get(i))
+ }
+ ids.toSeq
+ }
+ } finally {
+ reader.close()
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]