This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 8f1ce6974c8b [SPARK-60726][SQL] Add an API to read a partition of a 
native data source as an Arrow C stream
8f1ce6974c8b is described below

commit 8f1ce6974c8b7410073e14f7fb82d88b63c24dea
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:43:02 2026 +0900

    [SPARK-60726][SQL] Add an API to read a partition of a native data source 
as an Arrow C stream
    
    ### What changes were proposed in this pull request?
    
    A follow-up to SPARK-60101 (#59424), which zhouyuan asked for in 
https://github.com/apache/spark/pull/59424#issuecomment-6196064644. The design 
questions were listed in the reply 
https://github.com/apache/spark/pull/59424#issuecomment-6204557249. The change 
is an `Evolving` API, like the other mix-ins of 
`org.apache.spark.sql.connector.read`, to read a partition of a native scan as 
an Arrow C stream:
    
    ```java
    Evolving
    public interface SupportsArrowStreamRead extends PartitionReaderFactory {
      void openArrowStream(InputPartition partition, long streamAddress);
    }
    ```
    
    The reader factories of the batch and micro-batch scans of native data 
sources implement it, with the stream that `read` of the library initializes. 
Its arrays have exactly the columns of the read schema of the scan, since 
`read` returns exactly the columns of its schema to read (#59931), also for a 
library that does not prune the others. This PR is based on #59931 (version 1 
of the interface), whose commit it includes until that PR is merged. The 
questions of the reply are settled as [...]
    
    - **Which object.** The engine gets the reader factory of the scan from its 
plan, `BatchScanExec.readerFactory`, and the partitions with it. The factory of 
a batch scan holds the location of the library, the name of the data source, 
the read schema, its time zone and the state of the reader.
    - **Who owns the task context.** `openArrowStream` is called on the thread 
of the running task that reads the partition, and fails otherwise. The engine 
owns the stream, which it may move, and releases it, also after the task ended 
and on another thread. Spark frees the `SparkDsTaskContext` once the library 
released its stream: it wraps the four callbacks of the stream with its own, 
which call those of the library, keyed by a `private_data` of Spark. Once the 
task failed or completed, [...]
    - **What the engine checks.** Spark does not check the arrays of such a 
stream, while it checks those that it imports itself, because what to check 
depends on the importer. Arrow Java ignores the offsets of the arrays, for 
example, while Arrow C++ and arrow-rs do not. Spark still checks that the 
library initialized the stream, with non-NULL callbacks. The callbacks can be 
called from any thread, but never concurrently, as the header already says, and 
they call into the JVM, so the eng [...]
    
    `NativePartitionReader` opens its streams with the same code. The release 
of an engine's stream closes its `NativeLibrary.Task` on whichever thread the 
engine releases it, and only the release that removes the stream from the 
wrapped streams does, so the task is closed once. The docs describe the API 
under "Native Execution Engines".
    
    The reader factories of micro-batch scans implement it too: 
`ColumnarStreamReader.readerFactory()` lets a stream reader return its own 
factory, `NativeMicroBatchPartitionReaderFactory` for native data sources. The 
partitions of each micro-batch hold the state that the stream reader returned 
with them, so that a factory that an engine keeps reads them with that state, 
also once Spark planned the next micro-batch. A state larger than 64 KiB is 
broadcast, so that each executor fetches it [...]
    
    ### Why are the changes needed?
    
    Without it, a query on a native data source under a native engine such as 
Gluten imports the data into Arrow Java. A `ColumnarToRowExec` then converts it 
to rows, which the engine converts back to its own columnar format:
    
    ```
    read -> ArrowArrayStream -> Arrow Java -> ArrowColumnVector -> 
ColumnarToRow -> RowToVeloxColumnar
    ```
    
    With the API, the engine imports the stream of the library directly. 
Library loading, the checksums of the packages and `spark_ds_init` stay in 
Spark, so the same library runs on Spark alone and under the engine.
    
    ### Does this PR introduce _any_ user-facing change?
    
    It adds an API for engines, `SupportsArrowStreamRead`, in 4.4.0. Nothing 
changes for queries. For a library, the `partition_id` of a read with Spark's 
own reader is now the index of the partition in the scan, rather than the 
partition of the task, which differs after a union or a coalesce; native data 
sources are new in the unreleased 4.4.0.
    
    ### How was this patch tested?
    
    "open the partitions of batch and micro-batch scans as Arrow C streams" in 
`NativeDataSourceSuite` acts as an engine: in a single task, it opens the 
streams of both partitions, reads them on another thread, and checks the data 
and the partitions that the library got. It also reads a stream to its end in 
its task and releases it after the task ended, from the driver, and checks that 
the context stays alive until then. It checks the preconditions of 
`openArrowStream`, including a comple [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59934 from HyukjinKwon/native-ds-arrow-stream-read.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
    (cherry picked from commit e06d5d3c6990898f2b935d5e90dcd2b19b982863)
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 docs/sql-native-data-sources.md                    |  47 +-
 .../connector/read/SupportsArrowStreamRead.java    |  62 +++
 .../datasources/v2/ffi/NativeLibrary.java          | 349 +++++++++++-
 .../apache/spark/sql/datasource/spark_datasource.h |  45 +-
 .../v2/columnar/ColumnarDataSource.scala           |  16 +-
 .../datasources/v2/columnar/ColumnarTable.scala    |  12 +-
 .../datasources/v2/ffi/NativeDataSource.scala      | 349 ++++++++++--
 .../native-datasource/test_native_datasource.cc    |  44 +-
 .../v2/columnar/ColumnarDataSourceSuite.scala      |   8 +-
 .../datasources/v2/ffi/NativeDataSourceSuite.scala | 604 ++++++++++++++++++++-
 10 files changed, 1395 insertions(+), 141 deletions(-)

diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index 03b5eae4ab0c..049f49902e7c 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -116,11 +116,12 @@ Spark calls the functions of the table as follows:
   not specify a schema, Spark asks for it with `schema`.
 - **Batch reads**: on the driver, Spark creates a reader with `create_reader`, 
pushes operations
   down with `push_limit` and `prune_columns`, and plans the partitions with 
`partitions`, which
-  also returns the state of the reader. Then it calls `read` on the executors, 
in one task for each
-  partition, with a `SparkDsReadInfo` that has the partition, the schema to 
read and the state of
-  the reader, and `read` exports an `ArrowArrayStream`. Its callbacks can be 
called from any
-  thread, but never concurrently. The schema to read has only the columns that 
the query needs,
-  which `prune_columns` got, and the arrays of the stream have exactly these 
columns, also when the
+  also returns the state of the reader. Then it calls `read` on the executors 
for each partition,
+  in tasks that may each read several partitions, for example after a coalesce 
or a union, with a
+  `SparkDsReadInfo` that has the partition, the schema to read and the state 
of the reader, and
+  `read` exports an `ArrowArrayStream`. Its callbacks can be called from any 
thread, but never
+  concurrently. The schema to read has only the columns that the query needs, 
which
+  `prune_columns` got, and the arrays of the stream have exactly these 
columns, also when the
   library does not implement `prune_columns`, which only lets it plan to read 
them.
 - **Streaming reads**: on the driver, Spark creates a stream reader with 
`create_stream_reader`,
   which gets the columns to read, which may be only those that the query 
needs, and the location of
@@ -201,13 +202,16 @@ to, but it cannot check that the buffers of the arrays 
hold those values, becaus
 interface does not carry the sizes of buffers: a library that exports a buffer 
that is too short
 makes Spark read past its end.
 
-`read` and `create_data_writer` get a `SparkDsTaskContext`, with the IDs of 
the task, and its
-partition and attempt, and an `is_interrupted` function that returns whether 
the task was killed,
-for example because its query was canceled. Spark cannot interrupt native 
code: when a task is
-killed, Spark stops reading only once the `get_next` function of the stream 
returns. So a read or
-a write that runs for long, or blocks, for example on network I/O, should 
check `is_interrupted`
-regularly and use timeouts. The task reaper, `spark.task.reaper.enabled` and
-`spark.task.reaper.killTimeout`, kills an executor whose killed task does not 
end.
+`read` and `create_data_writer` get a `SparkDsTaskContext`, with the IDs of 
the task, its
+attempt, and the partition: for `read`, the index of the partition in the list 
that the planning
+returned, which is unique within a scan but not across the scans that a task 
reads, and for
+`create_data_writer`, the partition of the task. It also has an 
`is_interrupted` function that
+returns whether the task was killed, for example because its query was 
canceled. Spark cannot
+interrupt native code: when a task is killed, Spark stops reading only once 
the `get_next` function
+of the stream returns. So a read or a write that runs for long, or blocks, for 
example on network
+I/O, should check `is_interrupted` regularly and use timeouts. The task reaper,
+`spark.task.reaper.enabled` and `spark.task.reaper.killTimeout`, kills an 
executor whose killed
+task does not end.
 
 ### Compatibility
 
@@ -269,6 +273,25 @@ Increase it for libraries that use much memory, and bound 
the memory that a libr
 example with the size of its batches: on YARN or Kubernetes, an executor that 
uses more memory than
 it requested is killed.
 
+### Native Execution Engines
+
+An engine that runs queries in native code within Spark, such as Apache Gluten 
or Apache DataFusion
+Comet, can read a native data source without Spark importing its data on the 
JVM. The reader factory
+of a batch or micro-batch scan of a native data source implements
+`org.apache.spark.sql.connector.read.SupportsArrowStreamRead`: in the task 
that reads a partition,
+`openArrowStream` initializes an Arrow C stream that the engine allocated with 
the stream that
+`read` returns. The engine reads the stream in the task that opened it, and 
owns and releases the
+stream, which it may do after the task ended: Spark frees the 
`SparkDsTaskContext` of the read once
+the library released its stream. Once the task ended, `is_interrupted` returns 
true, so the library
+may stop reading then. The callbacks of the stream can be called from any 
thread, but never
+concurrently. Each call goes through the JVM, which then calls the callback of 
the stream of the
+library: this adds tens of nanoseconds to the call, while the import of a 
batch on the JVM takes
+microseconds. The first call on a native thread also attaches the thread to 
the JVM, which takes
+tens of microseconds, and the thread stays attached until it exits. Since the 
callbacks call into
+the JVM, the engine releases the stream before the JVM exits. Spark does not 
check the arrays of
+such a stream, unlike the streams that it reads itself, so the engine checks 
what its import of the
+arrays requires.
+
 ## Finding Native Data Sources
 
 When no Java or Python data source has the name that a query uses, Spark looks 
for a native data
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
new file mode 100644
index 000000000000..7b772a6d9951
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsArrowStreamRead.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * A mix-in interface for a {@link PartitionReaderFactory} that also reads a 
partition as an
+ * <a href="https://arrow.apache.org/docs/format/CStreamInterface.html";>Arrow 
C stream</a>, for an
+ * engine that processes Arrow data in native code: the engine imports the 
arrays of the stream
+ * itself, instead of reading the {@link 
org.apache.spark.sql.vectorized.ColumnarBatch}es of
+ * {@link #createColumnarReader(InputPartition)}. The reader factories of the 
batch and
+ * micro-batch scans of native data sources implement it, with the stream that 
the library
+ * returns.
+ * <p>
+ * Each array of the stream is a struct array whose fields are the columns of 
the read schema of
+ * the scan. Spark only checks that the source initialized the callbacks of 
the stream, not its
+ * arrays, while it checks the arrays of the streams that it reads itself, so 
the engine checks
+ * what its import of the arrays requires, such as the schema of the stream 
against the read
+ * schema, and the offsets and the lengths of the arrays. The callbacks of the 
stream can be called
+ * from any thread, but never concurrently. Each call goes through the JVM, 
which then calls the
+ * callback of the stream of the source, so the engine releases the stream 
before the JVM exits.
+ * The first call on a native thread attaches the thread to the JVM, until the 
thread exits.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface SupportsArrowStreamRead extends PartitionReaderFactory {
+
+  /**
+   * Initializes an Arrow C stream with the data of a partition. It is called 
in the running task
+   * that reads the partition, on a thread that has its {@code TaskContext}, 
and fails otherwise.
+   * The caller reads the stream in that task: once the task ended, the source 
may stop returning
+   * data and fail. The caller owns
+   * the stream, which it may move, and releases it, which may happen after 
the task ended: Spark
+   * frees what the stream uses, such as the context of the task, once the 
stream is released. A
+   * stream that is never released leaks.
+   *
+   * @param partition a partition of the scan, as the scan planned it
+   * @param streamAddress the address of a {@code struct ArrowArrayStream} 
that the caller
+   *                      allocated, and whose release callback is NULL, as it 
is in a
+   *                      zero-initialized struct. If this method fails after 
it checked its
+   *                      arguments, the stream is released. A stream whose 
release callback
+   *                      is not NULL is left as it is.
+   */
+  void openArrowStream(InputPartition partition, long streamAddress);
+}
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index 2185e8d73c12..ba0e2e683916 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -189,6 +189,13 @@ final class NativeLibrary {
   private static final long FINISH_WRITE_INFO_NUM_MESSAGES = 32;
   private static final long FINISH_WRITE_INFO_REJECTED = 40;
   private static final long FINISH_DATA_WRITER_INFO_SIZE = 8;
+  // struct ArrowArrayStream of the Arrow C stream interface, which Spark 
wraps for engines.
+  private static final long STREAM_GET_SCHEMA = 0;
+  private static final long STREAM_GET_NEXT = 8;
+  private static final long STREAM_GET_LAST_ERROR = 16;
+  private static final long STREAM_RELEASE = 24;
+  private static final long STREAM_PRIVATE_DATA = 32;
+  private static final long STREAM_SIZE = 40;
 
   // The structs that only hold their size, which the library gets as const 
pointers. Until fields
   // are added to them, a single struct of each kind is allocated once, and 
never freed.
@@ -283,6 +290,13 @@ final class NativeLibrary {
     layout.put("SparkDsFinishWriteInfo.rejected", FINISH_WRITE_INFO_REJECTED);
     layout.put("SparkDsFinishDataWriterInfo", FINISH_DATA_WRITER_INFO_SIZE);
     layout.put("SparkDsFinishDataWriterInfo.size", 0L);
+    // The streams of engines, which Spark wraps, see EngineStream.
+    layout.put("ArrowArrayStream", STREAM_SIZE);
+    layout.put("ArrowArrayStream.get_schema", STREAM_GET_SCHEMA);
+    layout.put("ArrowArrayStream.get_next", STREAM_GET_NEXT);
+    layout.put("ArrowArrayStream.get_last_error", STREAM_GET_LAST_ERROR);
+    layout.put("ArrowArrayStream.release", STREAM_RELEASE);
+    layout.put("ArrowArrayStream.private_data", STREAM_PRIVATE_DATA);
     return layout;
   }
 
@@ -492,15 +506,7 @@ final class NativeLibrary {
       MethodHandle handle = (MethodHandle) DOWNCALL.invoke(
         LINKER, address, descriptor(result, parameters), NO_OPTIONS);
       // Passes and returns the pointers, which are MemorySegments in the FFM 
API, as addresses.
-      for (int i = 0; i < parameters.length; i++) {
-        if (parameters[i] == C.PTR) {
-          handle = MethodHandles.filterArguments(handle, i, OF_ADDRESS);
-        }
-      }
-      if (result == C.PTR) {
-        handle = MethodHandles.filterReturnValue(handle, ADDRESS_OF);
-      }
-      return handle;
+      return convertPointers(handle, 0, OF_ADDRESS, ADDRESS_OF, result, 
parameters);
     }
 
     /**
@@ -509,14 +515,33 @@ final class NativeLibrary {
      * function lives as long as the JVM.
      */
     static long upcall(MethodHandle target, C result, C... parameters) throws 
Throwable {
+      target = convertPointers(target, 0, ADDRESS_OF, OF_ADDRESS, result, 
parameters);
+      Object stub = UPCALL.invoke(
+        LINKER, target, descriptor(result, parameters), GLOBAL_ARENA, 
NO_OPTIONS);
+      return (long) ADDRESS_OF.invoke(stub);
+    }
+
+    /**
+     * Applies `parameterFilter` to the parameters of `handle` that are 
pointers, the C parameters
+     * starting at the parameter `first`, and `resultFilter` to a result that 
is a pointer: a
+     * downcall converts the addresses to MemorySegments and back, and an 
upcall the other way.
+     */
+    static MethodHandle convertPointers(
+        MethodHandle handle,
+        int first,
+        MethodHandle parameterFilter,
+        MethodHandle resultFilter,
+        C result,
+        C... parameters) {
       for (int i = 0; i < parameters.length; i++) {
         if (parameters[i] == C.PTR) {
-          target = MethodHandles.filterArguments(target, i, ADDRESS_OF);
+          handle = MethodHandles.filterArguments(handle, first + i, 
parameterFilter);
         }
       }
-      Object stub = UPCALL.invoke(
-        LINKER, target, descriptor(result, parameters), GLOBAL_ARENA, 
NO_OPTIONS);
-      return (long) ADDRESS_OF.invoke(stub);
+      if (result == C.PTR) {
+        handle = MethodHandles.filterReturnValue(handle, resultFilter);
+      }
+      return handle;
     }
 
     /**
@@ -525,11 +550,7 @@ final class NativeLibrary {
      */
     static long host(String path, String sparkVersion) throws Throwable {
       long log = upcall(MethodHandles.insertArguments(LOG, 0, path), C.VOID, 
C.I32, C.PTR);
-      byte[] version = sparkVersion.getBytes(StandardCharsets.UTF_8);
-      long versionAddress = Platform.allocateMemory(version.length + 1L);
-      Platform.copyMemory(
-        version, Platform.BYTE_ARRAY_OFFSET, null, versionAddress, 
version.length);
-      Platform.putByte(null, versionAddress + version.length, (byte) 0);
+      long versionAddress = toCString(sparkVersion);
       long host = Platform.allocateMemory(HOST_SIZE);
       Platform.setMemory(host, (byte) 0, HOST_SIZE);
       Platform.putLong(null, host, HOST_SIZE);
@@ -1063,9 +1084,8 @@ final class NativeLibrary {
 
     /** Copies the string as a NUL-terminated UTF-8 string. */
     long string(String value) {
-      byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
-      long address = zeroed(bytes.length + 1L);
-      Platform.copyMemory(bytes, Platform.BYTE_ARRAY_OFFSET, null, address, 
bytes.length);
+      long address = toCString(value);
+      allocations.add(address);
       return address;
     }
 
@@ -1084,6 +1104,15 @@ final class NativeLibrary {
     }
   }
 
+  /** Copies a string to native memory, as a NUL-terminated UTF-8 string, 
which the caller frees. */
+  private static long toCString(String value) {
+    byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
+    long address = Platform.allocateMemory(bytes.length + 1L);
+    Platform.copyMemory(bytes, Platform.BYTE_ARRAY_OFFSET, null, address, 
bytes.length);
+    Platform.putByte(null, address + bytes.length, (byte) 0);
+    return address;
+  }
+
   private static byte[] readCString(long address) {
     int length = 0;
     while (Platform.getByte(null, address + length) != 0) {
@@ -1098,12 +1127,16 @@ final class NativeLibrary {
   // Task contexts
   // 
---------------------------------------------------------------------------------------------
 
-  // Whether the task of each SparkDsTaskContext was killed, by the address of 
the struct.
+  // Whether the library should stop reading or writing with each 
SparkDsTaskContext, which
+  // is_interrupted answers, by the address of the struct.
   private static final ConcurrentHashMap<Long, BooleanSupplier> tasks = new 
ConcurrentHashMap<>();
 
   /**
    * The {@code SparkDsTaskContext} of a task that reads a partition or writes 
data. The reader or
-   * the data writer closes it once the library no longer uses it, which frees 
it.
+   * the data writer closes it once the library no longer uses it, which frees 
it. For the stream of
+   * a read that an engine got, the release of the stream closes it, on 
whichever thread the engine
+   * releases the stream: only the release that removes the stream from {@code 
engineStreams} does,
+   * so it is closed once (see {@link EngineStream}).
    */
   static final class Task implements AutoCloseable {
     final long address;
@@ -1153,6 +1186,276 @@ final class NativeLibrary {
     }
   }
 
+  /** The number of task contexts that are not closed yet, for tests. */
+  static int openTaskContexts() {
+    return tasks.size();
+  }
+
+  // 
---------------------------------------------------------------------------------------------
+  // Streams of engines
+  // 
---------------------------------------------------------------------------------------------
+
+  // The errno values that the callbacks of a stream return.
+  private static final int EINVAL = 22;
+  private static final int EIO = 5;
+
+  /**
+   * The callbacks of the streams that Spark wraps for engines, see {@link 
EngineStream}, the
+   * method handles that call the callbacks of the wrapped streams, and what 
the callbacks return
+   * for a stream that Spark does not know. They are created when the first 
stream is wrapped, so
+   * that a JVM that wraps no stream does not create them, and a failure to 
create them only fails
+   * the streams of engines.
+   */
+  private static final class EngineStreamCallbacks {
+    // The addresses of the C functions that call Java, which live as long as 
the JVM.
+    static final long GET_SCHEMA;
+    static final long GET_NEXT;
+    static final long GET_LAST_ERROR;
+    static final long RELEASE;
+    // The callbacks of struct ArrowArrayStream, called with the address of 
the function first:
+    // get_schema and get_next, get_last_error, and release.
+    static final MethodHandle INT_CALLBACK;
+    static final MethodHandle POINTER_CALLBACK;
+    static final MethodHandle VOID_CALLBACK;
+    // The message of get_last_error for a stream that Spark does not know, 
which lives as long as
+    // the JVM.
+    static final long UNKNOWN_STREAM_ERROR;
+
+    static {
+      try {
+        MethodHandles.Lookup lookup = MethodHandles.lookup();
+        // Linker.downcallHandle(FunctionDescriptor, Linker.Option...), whose 
method handles take
+        // the address of the function to call as their first argument.
+        MethodHandle unboundDowncall = lookup.findVirtual(
+          Class.forName("java.lang.foreign.Linker"), "downcallHandle",
+          MethodType.methodType(MethodHandle.class,
+            Class.forName("java.lang.foreign.FunctionDescriptor"), 
Ffm.NO_OPTIONS.getClass()))
+          .asFixedArity();
+        GET_SCHEMA = Ffm.upcall(
+          lookup.findStatic(NativeLibrary.class, "engineStreamGetSchema",
+            MethodType.methodType(int.class, long.class, long.class)),
+          C.I32, C.PTR, C.PTR);
+        GET_NEXT = Ffm.upcall(
+          lookup.findStatic(NativeLibrary.class, "engineStreamGetNext",
+            MethodType.methodType(int.class, long.class, long.class)),
+          C.I32, C.PTR, C.PTR);
+        GET_LAST_ERROR = Ffm.upcall(
+          lookup.findStatic(NativeLibrary.class, "engineStreamGetLastError",
+            MethodType.methodType(long.class, long.class)),
+          C.PTR, C.PTR);
+        RELEASE = Ffm.upcall(
+          lookup.findStatic(NativeLibrary.class, "engineStreamRelease",
+            MethodType.methodType(void.class, long.class)),
+          C.VOID, C.PTR);
+        INT_CALLBACK = callback(unboundDowncall, C.I32, C.PTR, C.PTR);
+        POINTER_CALLBACK = callback(unboundDowncall, C.PTR, C.PTR);
+        VOID_CALLBACK = callback(unboundDowncall, C.VOID, C.PTR);
+        UNKNOWN_STREAM_ERROR = toCString("The stream was released, or is not a 
stream of Spark.");
+      } catch (Throwable e) {
+        // The message names the cause, because the JVM only keeps this 
message for the later
+        // uses of the class.
+        throw new IllegalStateException(
+          "Cannot create the callbacks of the streams of engines: " + 
rootCause(e), e);
+      }
+    }
+
+    /**
+     * Returns a method handle that calls the C function at the address of its 
first argument, a
+     * {@code long}, with the given types, as {@link Ffm#downcall} does for a 
fixed address.
+     */
+    private static MethodHandle callback(MethodHandle unboundDowncall, C 
result, C... parameters)
+        throws Throwable {
+      MethodHandle handle = (MethodHandle) unboundDowncall.invoke(
+        Ffm.LINKER, Ffm.descriptor(result, parameters), Ffm.NO_OPTIONS);
+      handle = MethodHandles.filterArguments(handle, 0, Ffm.OF_ADDRESS);
+      return Ffm.convertPointers(handle, 1, Ffm.OF_ADDRESS, Ffm.ADDRESS_OF, 
result, parameters);
+    }
+  }
+
+  private static final AtomicLong nextEngineStreamId = new AtomicLong(1);
+  // The streams wrapped for engines that are not released yet, by their 
private_data.
+  private static final ConcurrentHashMap<Long, EngineStream> engineStreams =
+    new ConcurrentHashMap<>();
+
+  /**
+   * The stream of a read that an engine got with {@code 
SupportsArrowStreamRead}, which ties the
+   * task context of the read to the stream: the context is closed once the 
library released the
+   * stream, whenever and on whichever thread the engine releases it, also 
after the task ended.
+   * The stream of the library is moved to memory of Spark, and the stream of 
the engine gets
+   * callbacks of Spark, which call the current callbacks of the library's 
stream, and a
+   * private_data that identifies the wrapped stream, so that the engine can 
move the stream too.
+   */
+  private static final class EngineStream {
+    final long inner;
+    final Task task;
+    // The name of the data source, for the errors.
+    final String name;
+    // The message of the last error of Spark, rather than of the library, or 
0 if the last call
+    // returned an error of the library, or none. The callbacks never run 
concurrently, but may
+    // run on different threads one after the other.
+    volatile long sparkError;
+
+    EngineStream(long inner, Task task, String name) {
+      this.inner = inner;
+      this.task = task;
+      this.name = name;
+    }
+
+    long callback(long offset) {
+      return Platform.getLong(null, inner + offset);
+    }
+
+    /**
+     * Records an error of Spark in the given callback, which get_last_error 
then returns, and logs
+     * it. The message is built here, since building it can throw too, as a 
StackOverflowError on
+     * the small stack of a thread of an engine can.
+     */
+    int fail(String callback, Throwable e) {
+      try {
+        clearError();
+        sparkError = toCString("The " + callback + " of the stream of the 
native data source " +
+          name + " failed: " + rootCause(e));
+        LOGGER.error("The {} of the stream of the native data source {} 
failed", e,
+          MDC.of(LogKeys.FUNCTION_NAME, callback), MDC.of(LogKeys.DATA_SOURCE, 
name));
+      } catch (Throwable ignored) {
+        // An upcall must not throw.
+      }
+      return EIO;
+    }
+
+    void clearError() {
+      if (sparkError != 0) {
+        Platform.freeMemory(sparkError);
+        sparkError = 0;
+      }
+    }
+  }
+
+  /**
+   * Wraps the stream at the given address, which read initialized with 
non-NULL callbacks, so that
+   * releasing it closes the given task context (see {@link EngineStream}). If 
it fails, the stream
+   * is left as it is.
+   *
+   * @param name the name of the data source, for the errors of the stream
+   */
+  static void wrapEngineStream(long stream, Task task, String name) {
+    // Creates the callbacks first, so that the stream is left as it is if 
that fails.
+    long getSchema;
+    long getNext;
+    long getLastError;
+    long release;
+    try {
+      getSchema = EngineStreamCallbacks.GET_SCHEMA;
+      getNext = EngineStreamCallbacks.GET_NEXT;
+      getLastError = EngineStreamCallbacks.GET_LAST_ERROR;
+      release = EngineStreamCallbacks.RELEASE;
+    } catch (ExceptionInInitializerError | NoClassDefFoundError e) {
+      // The first use of a class that failed to initialize gets the error, 
and the later ones a
+      // NoClassDefFoundError, whose cause has its message: both name the root 
cause here, as the
+      // errors of load do.
+      throw new IllegalStateException(
+        "Cannot create the callbacks of the streams of engines: " + 
rootCause(e), e);
+    }
+    long inner = Platform.allocateMemory(STREAM_SIZE);
+    long id = nextEngineStreamId.getAndIncrement();
+    try {
+      Platform.copyMemory(null, stream, null, inner, STREAM_SIZE);
+      engineStreams.put(id, new EngineStream(inner, task, name));
+    } catch (Throwable e) {
+      engineStreams.remove(id);
+      Platform.freeMemory(inner);
+      throw e;
+    }
+    Platform.putLong(null, stream + STREAM_GET_SCHEMA, getSchema);
+    Platform.putLong(null, stream + STREAM_GET_NEXT, getNext);
+    Platform.putLong(null, stream + STREAM_GET_LAST_ERROR, getLastError);
+    Platform.putLong(null, stream + STREAM_RELEASE, release);
+    Platform.putLong(null, stream + STREAM_PRIVATE_DATA, id);
+  }
+
+  /** The number of streams wrapped for engines that are not released yet, for 
tests. */
+  static int openEngineStreams() {
+    return engineStreams.size();
+  }
+
+  private static EngineStream engineStream(long stream) {
+    return engineStreams.get(Platform.getLong(null, stream + 
STREAM_PRIVATE_DATA));
+  }
+
+  // The callbacks of the streams wrapped for engines. They must not throw: an 
exception thrown by
+  // an upcall crashes the JVM.
+
+  private static int engineStreamGetSchema(long stream, long out) {
+    return engineStreamCall(stream, out, STREAM_GET_SCHEMA, "get_schema");
+  }
+
+  private static int engineStreamGetNext(long stream, long out) {
+    return engineStreamCall(stream, out, STREAM_GET_NEXT, "get_next");
+  }
+
+  private static int engineStreamCall(long stream, long out, long offset, 
String callback) {
+    EngineStream wrapped = null;
+    try {
+      wrapped = engineStream(stream);
+      if (wrapped == null) {
+        return EINVAL;
+      }
+      wrapped.clearError();
+      return (int) EngineStreamCallbacks.INT_CALLBACK.invokeExact(
+        wrapped.callback(offset), wrapped.inner, out);
+    } catch (Throwable e) {
+      return wrapped == null ? EIO : wrapped.fail(callback, e);
+    }
+  }
+
+  private static long engineStreamGetLastError(long stream) {
+    try {
+      EngineStream wrapped = engineStream(stream);
+      if (wrapped == null) {
+        return EngineStreamCallbacks.UNKNOWN_STREAM_ERROR;
+      }
+      if (wrapped.sparkError != 0) {
+        return wrapped.sparkError;
+      }
+      return (long) EngineStreamCallbacks.POINTER_CALLBACK.invokeExact(
+        wrapped.callback(STREAM_GET_LAST_ERROR), wrapped.inner);
+    } catch (Throwable e) {
+      return 0;
+    }
+  }
+
+  private static void engineStreamRelease(long stream) {
+    EngineStream wrapped = null;
+    try {
+      wrapped = engineStreams.remove(Platform.getLong(null, stream + 
STREAM_PRIVATE_DATA));
+      if (wrapped != null) {
+        try {
+          EngineStreamCallbacks.VOID_CALLBACK.invokeExact(
+            wrapped.callback(STREAM_RELEASE), wrapped.inner);
+        } finally {
+          // The library no longer uses the task once it released its stream.
+          wrapped.clearError();
+          Platform.freeMemory(wrapped.inner);
+          wrapped.task.close();
+        }
+      }
+    } catch (Throwable e) {
+      try {
+        if (wrapped == null) {
+          LOGGER.error("The release of the stream of a native data source 
failed", e);
+        } else {
+          LOGGER.error("The {} of the stream of the native data source {} 
failed", e,
+            MDC.of(LogKeys.FUNCTION_NAME, "release"), 
MDC.of(LogKeys.DATA_SOURCE, wrapped.name));
+        }
+      } catch (Throwable ignored) {
+        // An upcall must not throw.
+      }
+    } finally {
+      // Marks the stream released, as the C stream interface requires of 
release.
+      Platform.putLong(null, stream + STREAM_RELEASE, 0);
+    }
+  }
+
   // 
---------------------------------------------------------------------------------------------
   // Sink
   // 
---------------------------------------------------------------------------------------------
diff --git 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index 1aa4eb346885..a8d7785ee1b1 100644
--- 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++ 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -57,8 +57,9 @@
  * - Batch reads. On the driver, Spark creates a reader with create_reader, 
pushes operations down
  *   by calling, each at most once and in this order, push_limit and 
prune_columns, plans the
  *   partitions with partitions, which also returns the state of the reader, 
and closes it. Then it
- *   calls read on the executors, in one task for each partition, with the 
schema to read and the
- *   state of the reader. If Spark never plans the partitions, for example to 
only explain the plan
+ *   calls read on the executors for each partition, with the schema to read 
and the state of the
+ *   reader, in tasks that may each read several partitions, for example after 
a coalesce or a
+ *   union. If Spark never plans the partitions, for example to only explain 
the plan
  *   of the query, it closes the reader once the reader is no longer 
referenced.
  * - Streaming reads. On the driver, Spark creates a stream reader with 
create_stream_reader, with
  *   the columns to read and the location of the checkpoint of the source, and 
keeps it until the
@@ -408,7 +409,14 @@ typedef struct SparkDsCommitOffsetInfo {
 
 /*
  * The task that calls read or create_data_writer, on an executor. It stays 
valid until the stream
- * that read returns is released, or until the data writer is closed.
+ * that read returns is released, or until the data writer is closed. The 
stream of a read may be
+ * released once the task ended, from another thread: an engine that runs a 
query in native code
+ * may get the stream from Spark, read it in the task that opened it and 
release it later, and
+ * Spark itself may read a stream on a thread that the task does not wait for, 
for example the one
+ * of a PipedRDD. The task stays valid until then too, and for a read, 
is_interrupted then also
+ * returns true once the task failed or completed, so the library may stop 
reading then. Such an
+ * engine may also keep the arrays and the schemas of the stream after it 
released the stream, so
+ * the release of an array or a schema must not use the stream or the task.
  */
 typedef struct SparkDsTaskContext {
   /* The size of this struct in the header that Spark was built with, as in 
SparkDsHost. */
@@ -416,15 +424,23 @@ typedef struct SparkDsTaskContext {
   /* The ID of the stage of the task, and the attempt number of the stage. */
   int32_t stage_id;
   int32_t stage_attempt_number;
-  /* The partition that the task reads or writes, and the attempt number of 
the task. */
+  /*
+   * The partition: for read, the index of the partition in the list that 
partitions or
+   * stream_partitions returned, which is not the partition of the task when 
Spark reads several
+   * scans or partitions in one task, for example after a union or a coalesce. 
It is unique within
+   * a scan, but a task that reads several scans may get the same index for 
each. For
+   * create_data_writer, the partition of the task that writes. Then the 
attempt number of the
+   * task.
+   */
   int32_t partition_id;
   int32_t attempt_number;
   /* The ID of the task attempt, unique in the Spark application. */
   int64_t task_attempt_id;
   /*
-   * Returns whether the task was killed, for example because its query was 
canceled. Spark cannot
-   * interrupt native code, so a read or a write that runs for long, or 
blocks, should check it
-   * regularly, and fail once it is true.
+   * Returns whether the task was killed, for example because its query was 
canceled, and for read,
+   * also whether the task failed or completed, since the stream of a read may 
be read after its
+   * task ended. Spark cannot interrupt native code, so a read or a write that 
runs for long, or
+   * blocks, should check it regularly, and fail once it is true.
    */
   bool (*is_interrupted)(const struct SparkDsTaskContext* task);
 } SparkDsTaskContext;
@@ -640,7 +656,8 @@ typedef struct SparkDsLibrary {
   int (*prune_columns)(void* reader, const char* const* columns, int64_t 
count);
 
   /*
-   * Plans the partitions of the scan. Spark reads each partition in a 
separate task.
+   * Plans the partitions of the scan. Spark reads each partition with a call 
of read, in a task
+   * that may read several partitions, for example after a coalesce or a union.
    *
    * info: nothing yet but its size, so that inputs can be added within this 
version
    * add_partition: called with each partition, serialized in a format defined 
by the library
@@ -713,11 +730,13 @@ typedef struct SparkDsLibrary {
    * out: the stream to initialize. Each array of the stream is a struct array 
whose fields are the
    *      columns of the schema to read, info->schema. The callbacks of the 
stream can be called
    *      from any thread, but never concurrently. get_next may block, for 
example on network
-   *      I/O, but Spark cannot interrupt native code: when the task is 
killed, Spark only stops
-   *      reading once get_next returns, and then releases the stream. So a 
get_next that blocks
-   *      forever keeps the task running, unless the task reaper 
(spark.task.reaper.enabled and
-   *      spark.task.reaper.killTimeout) kills the executor. Use timeouts for 
blocking
-   *      operations, and check task->is_interrupted.
+   *      I/O, but Spark cannot interrupt native code: when Spark reads the 
stream itself and the
+   *      task is killed, Spark only stops reading once get_next returns, and 
then releases the
+   *      stream. So a get_next that blocks forever keeps the task running, 
unless the task reaper
+   *      (spark.task.reaper.enabled and spark.task.reaper.killTimeout) kills 
the executor. An
+   *      engine that gets the stream from Spark reads and releases it itself 
(see
+   *      SparkDsTaskContext). Use timeouts for blocking operations, and check
+   *      task->is_interrupted.
    */
   int (*read)(const SparkDsTaskContext* task, const SparkDsReadInfo* info,
               struct ArrowArrayStream* out);
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index d0e1fc430635..a98cb66a485a 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -20,7 +20,7 @@ import java.util
 
 import org.apache.spark.sql.connector.catalog.TableCapability
 import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, 
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, 
PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.{InputPartition, 
PartitionReaderFactory}
 import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -111,10 +111,11 @@ case class ColumnarPlannedBatch(
 
 /**
  * Reads the data of a [[ColumnarDataSource]] for a micro-batch streaming 
scan, with offsets in
- * JSON. Spark plans the partitions of each micro-batch on the driver, then 
serializes the reader
- * to the executors, where `read` is called for each partition.
+ * JSON. Spark plans the partitions of each micro-batch on the driver with 
this reader, which stays
+ * there, and serializes the reader factory that `readerFactory` returns to 
the executors, which
+ * read the partitions with it.
  */
-trait ColumnarStreamReader extends Serializable {
+trait ColumnarStreamReader {
   /**
    * Returns the offset to start from when the streaming query has no 
checkpoint. Spark keeps it in
    * the checkpoint of the source, and only asks for it when the checkpoint 
does not have it yet.
@@ -135,7 +136,12 @@ trait ColumnarStreamReader extends Serializable {
    */
   def partitions(start: String, end: String): Array[InputPartition]
 
-  def read(partition: InputPartition): PartitionReader[ColumnarBatch]
+  /**
+   * Returns the reader factory of a micro-batch, which Spark serializes to 
the executors. It may
+   * also implement other interfaces of reader factories, such as
+   * `org.apache.spark.sql.connector.read.SupportsArrowStreamRead`.
+   */
+  def readerFactory(): PartitionReaderFactory
 
   /** Tells the data source that Spark has processed all the data up to and 
including `end`. */
   def commit(end: String): Unit = {}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
index 21bbbeb2e5b1..6b4d8c0db3dc 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
@@ -37,7 +37,6 @@ import 
org.apache.spark.sql.execution.streaming.checkpointing.HDFSMetadataLog
 import org.apache.spark.sql.internal.connector.SupportsMetadata
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.util.CaseInsensitiveStringMap
-import org.apache.spark.sql.vectorized.ColumnarBatch
 import org.apache.spark.util.Utils
 
 /**
@@ -306,9 +305,7 @@ class ColumnarMicroBatchStream(reader: 
ColumnarStreamReader, checkpointLocation:
     }
   }
 
-  override def createReaderFactory(): PartitionReaderFactory = {
-    new ColumnarStreamPartitionReaderFactory(reader)
-  }
+  override def createReaderFactory(): PartitionReaderFactory = 
reader.readerFactory()
 
   override def deserializeOffset(json: String): Offset = 
ColumnarStreamOffset(json)
 
@@ -368,10 +365,3 @@ abstract class ColumnarOnlyPartitionReaderFactory extends 
PartitionReaderFactory
 
   override def supportColumnarReads(partition: InputPartition): Boolean = true
 }
-
-class ColumnarStreamPartitionReaderFactory(reader: ColumnarStreamReader)
-  extends ColumnarOnlyPartitionReaderFactory {
-  override def createColumnarReader(partition: InputPartition): 
PartitionReader[ColumnarBatch] = {
-    reader.read(partition)
-  }
-}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 0e359ff7f72f..40e1f2d44d49 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -19,25 +19,31 @@ package org.apache.spark.sql.execution.datasources.v2.ffi
 import java.lang.ref.{Cleaner, WeakReference}
 import java.util
 import java.util.concurrent.ConcurrentHashMap
-import java.util.concurrent.atomic.AtomicLong
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicLong}
+import java.util.function.BooleanSupplier
 import javax.annotation.concurrent.GuardedBy
 
+import scala.collection.mutable
 import scala.jdk.CollectionConverters._
+import scala.reflect.{classTag, ClassTag}
 import scala.util.control.NonFatal
 
 import com.fasterxml.jackson.core.JsonProcessingException
 import com.fasterxml.jackson.databind.DeserializationFeature
 import com.fasterxml.jackson.databind.json.JsonMapper
+import com.google.common.collect.MapMaker
 import org.apache.arrow.c.{ArrowArray, ArrowArrayStream, ArrowSchema, 
CDataDictionaryProvider, Data}
 import org.apache.arrow.vector.{FieldVector, ValueVector, VectorSchemaRoot}
 import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
 import org.apache.arrow.vector.types.pojo.{ArrowType, Field, Schema}
 
-import org.apache.spark.{SparkException, SparkThrowable, 
SparkUnsupportedOperationException, TaskContext}
+import org.apache.spark.{SparkContext, SparkException, SparkThrowable, 
SparkUnsupportedOperationException, TaskContext}
+import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.internal.Logging
+import org.apache.spark.sql.classic.SparkSession
 import org.apache.spark.sql.connector.catalog.TableCapability
 import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, 
BATCH_WRITE, MICRO_BATCH_READ, STREAMING_WRITE, TRUNCATE}
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, 
PartitionReaderFactory, SupportsArrowStreamRead}
 import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
 import org.apache.spark.sql.errors.{QueryCompilationErrors, 
QueryExecutionErrors}
 import 
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchPlanner, 
ColumnarDataSource, ColumnarOnlyPartitionReaderFactory, ColumnarPlannedBatch, 
ColumnarStreamReader, ColumnarStreamWriter, ColumnarWriter}
@@ -283,7 +289,7 @@ class NativeBatchPlanner(
     try {
       val planned = call(name, "plan a scan 
of")(library.partitions(handle.get))
       ColumnarPlannedBatch(
-        NativeInputPartition.from(planned.partitions),
+        NativeInputPartition.from(planned.partitions).toArray[InputPartition],
         new NativePartitionReaderFactory(executorLocation, name,
           bytesOrEmpty(Option(planned.state)), readSchema, timeZoneId))
     } finally {
@@ -293,17 +299,27 @@ class NativeBatchPlanner(
   }
 }
 
-/** Reads the partitions of a planned batch scan of a native data source, on 
the executors. */
+/**
+ * Reads the partitions of a planned batch scan of a native data source, on 
the executors. An
+ * engine can also get the Arrow C stream of a partition from it, see 
[[SupportsArrowStreamRead]].
+ */
 class NativePartitionReaderFactory(
     location: NativeLibraryLocation,
     name: String,
     readerState: Array[Byte],
     readSchema: StructType,
     timeZoneId: String)
-  extends ColumnarOnlyPartitionReaderFactory {
+  extends ColumnarOnlyPartitionReaderFactory with SupportsArrowStreamRead {
   override def createColumnarReader(partition: InputPartition): 
PartitionReader[ColumnarBatch] = {
     new NativePartitionReader(location, name, NativeLibrary.READ_BATCH, 
readerState,
-      partition.asInstanceOf[NativeInputPartition].bytes, readSchema, 
timeZoneId)
+      partition.asInstanceOf[NativeInputPartition], readSchema, timeZoneId)
+  }
+
+  override def openArrowStream(partition: InputPartition, streamAddress: 
Long): Unit = {
+    NativePartitionReader.openEngineStream[NativeInputPartition](location, 
name,
+      NativeLibrary.READ_BATCH, partition, readSchema, timeZoneId, 
streamAddress) { partition =>
+      (readerState, partition)
+    }
   }
 }
 
@@ -317,17 +333,21 @@ class NativeDataSourceStreamReader(
     name: String,
     schema: StructType,
     timeZoneId: String,
-    @transient private val handle: NativeHandle)
+    handle: NativeHandle)
   extends ColumnarStreamReader {
   import NativeCalls._
 
-  // Set on the driver whenever the partitions of a micro-batch are planned, 
and serialized to
-  // the executors with the partitions of that micro-batch. Spark plans each 
range once (see
-  // ColumnarMicroBatchStream), so the scans of a micro-batch get the 
partitions of a single
-  // stream_partitions call, whose state this is.
-  private var readerState: Array[Byte] = Array.emptyByteArray
+  // The ends of the planned ranges that are not committed yet, in the order 
in which they were
+  // planned, with the broadcast of the state of their micro-batch, if it was 
broadcast. Spark plans
+  // each range once (see ColumnarMicroBatchStream), so the scans of a 
micro-batch get the
+  // partitions of a single stream_partitions call, with the state that it 
returned.
+  private val uncommitted = mutable.LinkedHashMap.empty[String, 
Option[Broadcast[Array[Byte]]]]
+  // The context that broadcasts the states, taken on the driver when the 
reader is created, so
+  // that a range can also be planned on a thread without an active session.
+  private val sparkContext =
+    SparkContext.getActive.getOrElse(SparkSession.active.sparkContext)
 
-  @transient private lazy val library = NativeLibraries.get(location)
+  private lazy val library = NativeLibraries.get(location)
 
   // The calls of the reader on the driver are synchronized, since calls on a 
handle must not run
   // at the same time: Spark may plan the scans of a micro-batch on several 
threads, such as the
@@ -366,26 +386,89 @@ class NativeDataSourceStreamReader(
 
   override def partitions(start: String, end: String): Array[InputPartition] = 
synchronized {
     val planned = call(name, "plan a scan 
of")(library.streamPartitions(handle.get, start, end))
-    readerState = bytesOrEmpty(Option(planned.state))
-    NativeInputPartition.from(planned.partitions)
+    val partitions = NativeInputPartition.from(planned.partitions)
+    val bytes = bytesOrEmpty(Option(planned.state))
+    // A small state goes with each task, while the executors fetch a larger 
one once for the
+    // micro-batch, as a broadcast. Spark only plans the next range once the 
micro-batch of the
+    // previous one finished, so the copies of the previous broadcast on the 
executors are removed
+    // then. The broadcast itself is destroyed once its micro-batch is 
committed, whose partitions
+    // are not read anymore then, also by an engine.
+    uncommitted.values.lastOption.flatten.foreach(_.unpersist(blocking = 
false))
+    val state = if (bytes.length > 
NativeDataSourceStreamReader.maxInlineStateBytes &&
+        partitions.nonEmpty) {
+      Right(sparkContext.broadcast(bytes))
+    } else {
+      Left(bytes)
+    }
+    uncommitted.remove(end).flatten.foreach(_.destroy(blocking = false))
+    uncommitted.put(end, state.toOption)
+    partitions.map[InputPartition](NativeMicroBatchPartition(_, state))
   }
 
-  override def read(partition: InputPartition): PartitionReader[ColumnarBatch] 
= {
-    new NativePartitionReader(location, name, NativeLibrary.READ_MICRO_BATCH, 
readerState,
-      partition.asInstanceOf[NativeInputPartition].bytes, schema, timeZoneId)
+  override def readerFactory(): PartitionReaderFactory = {
+    new NativeMicroBatchPartitionReaderFactory(location, name, schema, 
timeZoneId)
   }
 
   override def commit(end: String): Unit = synchronized {
+    // The micro-batches up to the committed one are committed too, also those 
that Spark did not
+    // commit one by one, as with async progress tracking, and their 
broadcasts are destroyed.
+    if (uncommitted.contains(end)) {
+      val committed = uncommitted.keys.toSeq.takeWhile(_ != end) :+ end
+      
committed.foreach(uncommitted.remove(_).flatten.foreach(_.destroy(blocking = 
false)))
+    }
     call(name, "commit an offset 
of")(optional(library.commitOffset(handle.get, end)))
   }
 
-  override def stop(): Unit = synchronized(handle.close())
+  // The broadcasts of the states of the planned micro-batches that are not 
committed yet, for
+  // tests.
+  private[ffi] def uncommittedStates: Seq[Broadcast[Array[Byte]]] =
+    synchronized(uncommitted.values.flatten.toSeq)
+
+  override def stop(): Unit = synchronized {
+    Utils.tryWithSafeFinally {
+      uncommitted.values.flatten.foreach(_.destroy(blocking = false))
+      uncommitted.clear()
+    }(handle.close())
+  }
 }
 
 object NativeDataSourceStreamReader {
   // Parses the offsets of the libraries to check them, and rejects anything 
after the JSON value.
   private val mapper =
     
JsonMapper.builder().enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS).build()
+
+  // The size of the largest state of a micro-batch that goes with each of its 
tasks, rather than
+  // as a broadcast, which costs more than it saves for a state of a few 
offsets, written to the
+  // block manager of the driver, fetched by each executor, and removed from 
them for each
+  // micro-batch.
+  private[ffi] val maxInlineStateBytes: Int = 64 * 1024
+}
+
+/**
+ * Reads the partitions of a micro-batch of a native data source, on the 
executors, each with the
+ * state of its micro-batch. An engine can also get the Arrow C stream of a 
partition from it, see
+ * [[SupportsArrowStreamRead]].
+ *
+ * @param timeZoneId the time zone of the timestamps of the schema that the 
stream reader got
+ */
+class NativeMicroBatchPartitionReaderFactory(
+    location: NativeLibraryLocation,
+    name: String,
+    schema: StructType,
+    timeZoneId: String)
+  extends ColumnarOnlyPartitionReaderFactory with SupportsArrowStreamRead {
+  override def createColumnarReader(partition: InputPartition): 
PartitionReader[ColumnarBatch] = {
+    val microBatchPartition = partition.asInstanceOf[NativeMicroBatchPartition]
+    new NativePartitionReader(location, name, NativeLibrary.READ_MICRO_BATCH,
+      microBatchPartition.readerState, microBatchPartition.partition, schema, 
timeZoneId)
+  }
+
+  override def openArrowStream(partition: InputPartition, streamAddress: 
Long): Unit = {
+    
NativePartitionReader.openEngineStream[NativeMicroBatchPartition](location, 
name,
+      NativeLibrary.READ_MICRO_BATCH, partition, schema, timeZoneId, 
streamAddress) { partition =>
+      (partition.readerState, partition.partition)
+    }
+  }
 }
 
 /**
@@ -472,13 +555,31 @@ class NativeDataSourceStreamWriter(
 case class NativeInputPartition(index: Int, bytes: Array[Byte]) extends 
InputPartition
 
 object NativeInputPartition {
-  def from(partitions: Array[Array[Byte]]): Array[InputPartition] = {
+  def from(partitions: Array[Array[Byte]]): Array[NativeInputPartition] = {
     Option(partitions).getOrElse(Array.empty).zipWithIndex.map { case (bytes, 
i) =>
       NativeInputPartition(i, Option(bytes).getOrElse(Array.emptyByteArray))
     }
   }
 }
 
+/**
+ * A partition of a micro-batch, with the state that the stream reader 
returned with the
+ * partitions of the micro-batch, if any. The state changes with each 
micro-batch, so it goes with
+ * the partitions rather than with the reader factory, which an engine may 
keep while Spark plans
+ * the next micro-batch.
+ *
+ * @param state the bytes of the state, empty if the stream reader returned 
none, or for a state
+ *              larger than 64 KiB, their broadcast, so that each executor 
fetches it once for the
+ *              micro-batch rather than with each task
+ */
+case class NativeMicroBatchPartition(
+    partition: NativeInputPartition,
+    state: Either[Array[Byte], Broadcast[Array[Byte]]])
+  extends InputPartition {
+  /** The state to read the partition with, empty if the stream reader 
returned none. */
+  def readerState: Array[Byte] = state.fold(identity, _.value)
+}
+
 /** The message returned by the data writer of a task of a native library. */
 case class NativeWriterCommitMessage(bytes: Array[Byte]) extends 
WriterCommitMessage
 
@@ -505,6 +606,7 @@ object NativeWriterCommitMessage {
  * library.
  *
  * @param kind the kind of the scan, `NativeLibrary.READ_BATCH` or 
`READ_MICRO_BATCH`
+ * @param partition the partition, whose index in the scan the library gets as 
`partition_id`
  * @param schema the schema to read
  * @param timeZoneId the time zone of the timestamps of the schema to read, as 
the reader of the
  *                   library got it on the driver
@@ -514,7 +616,7 @@ class NativePartitionReader(
     name: String,
     kind: Int,
     readerState: Array[Byte],
-    partition: Array[Byte],
+    partition: NativeInputPartition,
     schema: StructType,
     timeZoneId: String)
   extends PartitionReader[ColumnarBatch] {
@@ -540,7 +642,8 @@ class NativePartitionReader(
   // Synchronized so that a close on another thread sees the task and the 
vectors.
   synchronized {
     try {
-      openStream()
+      NativePartitionReader.openStream(location, name, kind, readerState, 
partition, schema,
+        timeZoneId, stream.memoryAddress())(task = _)
       root = VectorSchemaRoot.create(importSchema(), allocator)
       checkSchema()
       batch = new 
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
@@ -551,25 +654,7 @@ class NativePartitionReader(
     }
   }
 
-  private def openStream(): Unit = {
-    val library = NativeLibraries.get(location)
-    task = NativeTask.create()
-    NativeArrow.exportSchema(schema, timeZoneId) { schemaAddress =>
-      call(name, "read") {
-        library.read(task, kind, schemaAddress, readerState, partition, 
stream.memoryAddress())
-      }
-    }
-  }
-
   private def importSchema(): Schema = {
-    // Arrow calls the callbacks of the stream without checking them, and 
calling NULL would crash
-    // the JVM.
-    val snapshot = stream.snapshot()
-    if (snapshot.release == 0 || snapshot.get_schema == 0 || snapshot.get_next 
== 0 ||
-        snapshot.get_last_error == 0) {
-      throw error(name, "read", "The library did not initialize the stream, or 
a callback of " +
-        "the stream is NULL.")
-    }
     val arrowSchema = ArrowSchema.allocateNew(allocator)
     try {
       call(name, "read")(stream.getSchema(arrowSchema))
@@ -934,6 +1019,132 @@ class NativePartitionReader(
   }
 }
 
+object NativePartitionReader {
+  import NativeCalls._
+
+  /**
+   * Initializes the stream at `streamAddress` with the data of a partition 
for an engine, see
+   * [[SupportsArrowStreamRead]]: the context of its task is closed once the 
stream is released.
+   * It first checks the arguments of `openArrowStream`, before it looks at 
the partition, which
+   * must be a `P`, the class of the partitions of the scan of the reader 
factory. It then reads
+   * the partition with the state and the partition of the library that 
`stateAndPartition`
+   * returns for it.
+   */
+  private[ffi] def openEngineStream[P <: InputPartition: ClassTag](
+      location: NativeLibraryLocation,
+      name: String,
+      kind: Int,
+      partition: InputPartition,
+      schema: StructType,
+      timeZoneId: String,
+      streamAddress: Long)(
+      stateAndPartition: P => (Array[Byte], NativeInputPartition)): Unit = {
+    val context = TaskContext.get()
+    if (context != null) {
+      // A task that was killed gets the error of its kill, as Spark reports 
it.
+      context.killTaskIfInterrupted()
+    }
+    // A task that failed is over too, also while its listeners run.
+    if (context == null || context.isFailed() || context.isCompleted()) {
+      throw new IllegalStateException("openArrowStream must be called in the 
running task that " +
+        "reads the partition, on a thread that has its TaskContext.")
+    }
+    if (streamAddress == 0 || 
ArrowArrayStream.wrap(streamAddress).snapshot().release != 0) {
+      throw new IllegalArgumentException(
+        "openArrowStream needs the address of a stream whose release callback 
is NULL.")
+    }
+    val (readerState, nativePartition) = partition match {
+      case expected: P => stateAndPartition(expected)
+      case _ =>
+        throw new IllegalArgumentException("openArrowStream needs a partition 
of the scan of " +
+          s"the reader factory, a ${classTag[P].runtimeClass.getSimpleName}, 
but got " +
+          s"${Option(partition).map(_.getClass.getName).orNull}.")
+    }
+    var task: NativeLibrary.Task = null
+    try {
+      openStream(location, name, kind, readerState, nativePartition, schema, 
timeZoneId,
+        streamAddress)(task = _)
+      // As a failure of read, a failure to create the callbacks of the 
streams of engines, such as
+      // a link error, is an error of the data source.
+      call(name, "read")(NativeLibrary.wrapEngineStream(streamAddress, task, 
name))
+    } catch {
+      case e: Throwable =>
+        releaseOnFailure(e, streamAddress, Option(task))
+        throw e
+    }
+  }
+
+  /**
+   * Initializes the Arrow C stream at `streamAddress`, whose release callback 
is NULL, with the
+   * data of a partition, by calling read of the library in the running task. 
It gives the context
+   * of the task to `setTask` before read, since the library may use it until 
the stream is
+   * released, also if this fails: the caller then releases the stream, and 
closes the context
+   * after it, as the reader of Spark closes both.
+   *
+   * @param partition the partition, whose index in the scan the library gets 
as `partition_id`,
+   *                  rather than the partition of the task: a task may read 
several partitions of
+   *                  a scan, or of several scans, as after a coalesce or a 
union
+   */
+  private def openStream(
+      location: NativeLibraryLocation,
+      name: String,
+      kind: Int,
+      readerState: Array[Byte],
+      partition: NativeInputPartition,
+      schema: StructType,
+      timeZoneId: String,
+      streamAddress: Long)(
+      setTask: NativeLibrary.Task => Unit): Unit = {
+    val library = NativeLibraries.get(location)
+    val stream = ArrowArrayStream.wrap(streamAddress)
+    val task = NativeTask.create(Some(partition.index), read = true)
+    setTask(task)
+    NativeArrow.exportSchema(schema, timeZoneId) { schemaAddress =>
+      call(name, "read") {
+        library.read(task, kind, schemaAddress, readerState, partition.bytes, 
streamAddress)
+      }
+    }
+    // Arrow calls the callbacks of the stream without checking them, and 
calling NULL would crash
+    // the JVM.
+    val snapshot = stream.snapshot()
+    if (snapshot.release == 0 || snapshot.get_schema == 0 || snapshot.get_next 
== 0 ||
+        snapshot.get_last_error == 0) {
+      throw error(name, "read", "The library did not initialize the stream, or 
a callback of the " +
+        "stream is NULL.")
+    }
+  }
+
+  /**
+   * After `failure`, releases the stream of an engine at `streamAddress`, if 
the library
+   * initialized it, before the task that it may use, if any, and marks it 
released in case its
+   * release callback did not. If the stream cannot be released, the task is 
not closed either,
+   * since the library may still use it: both then leak, rather than letting 
the release of the
+   * stream use a freed task. A failure to clean up is added to `failure`.
+   */
+  private def releaseOnFailure(
+      failure: Throwable,
+      streamAddress: Long,
+      task: Option[NativeLibrary.Task]): Unit = {
+    val released = try {
+      val stream = ArrowArrayStream.wrap(streamAddress)
+      stream.release()
+      stream.markReleased()
+      true
+    } catch {
+      case e: Throwable =>
+        failure.addSuppressed(e)
+        false
+    }
+    if (released) {
+      task.foreach { task =>
+        try task.close() catch {
+          case e: Throwable => failure.addSuppressed(e)
+        }
+      }
+    }
+  }
+}
+
 /**
  * Writes the batches of a task to the data writer of a native library. The 
data writer gets the
  * schema of the data once, when it is created, and then only the arrays of 
the batches.
@@ -1037,22 +1248,70 @@ class NativeDataWriter(
 
 /** Creates the `SparkDsTaskContext` of the running task. */
 private[ffi] object NativeTask {
+  // The flag of each running task that reads, set once the task failed or 
completed by a single
+  // pair of listeners for all its reads, which then also remove it: a task 
that reads many
+  // partitions, as after a coalesce, registers them once. The contexts of the 
reads keep their
+  // flag, so they still see it once it is removed. A task is the key rather 
than its attempt ID,
+  // which is 0 for each context of `TaskContext.empty()`. The keys are weak, 
compared by identity,
+  // so that a context that never ends, such as one that a test sets on its 
thread, does not stay.
+  private val endedTasks = new MapMaker().weakKeys().makeMap[TaskContext, 
AtomicBoolean]()
+
+  /** Returns the flag that is set once the given task failed or completed. */
+  private def endedFlag(taskContext: TaskContext): AtomicBoolean = {
+    Option(endedTasks.get(taskContext)).getOrElse {
+      val ended = new AtomicBoolean(false)
+      Option(endedTasks.putIfAbsent(taskContext, ended)).getOrElse {
+        // A listener runs right away if the task already failed or completed.
+        def end(): Unit = {
+          ended.set(true)
+          endedTasks.remove(taskContext, ended)
+        }
+        taskContext.addTaskFailureListener((_, _) => end())
+        taskContext.addTaskCompletionListener[Unit](_ => end())
+        ended
+      }
+    }
+  }
+
+  // The number of tasks whose reads have a flag, for tests.
+  private[ffi] def tasksWithEndedFlags: Int = endedTasks.size()
+
   /**
-   * @param partitionId the partition that the task reads or writes, by 
default the partition of
-   *                    the task
+   * @param partitionId the `partition_id` of the context: the index of the 
partition in the scan
+   *                    for a read, and by default the partition of the task
    * @param taskAttemptId the ID of the task attempt, by default the one of 
the task
+   * @param read whether the context is the one of a read, which the library 
may use after the
+   *             task ended: an engine may read the stream of a partition 
then, and a thread that
+   *             the task does not wait for may be in get_next once it 
completes, as with
+   *             `PipedRDD`. `is_interrupted` then also reports that the task 
failed or completed,
+   *             and the context does not keep the task alive. A data writer 
only runs within its
+   *             task, whose kill `is_interrupted` reports. Outside of a task, 
it reports nothing.
    */
   def create(
       partitionId: Option[Int] = None,
-      taskAttemptId: Option[Long] = None): NativeLibrary.Task = {
+      taskAttemptId: Option[Long] = None,
+      read: Boolean = false): NativeLibrary.Task = {
     val context = Option(TaskContext.get())
+    val interrupted: BooleanSupplier = context match {
+      case Some(taskContext) if read =>
+        // Set once the task failed or completed, by listeners, so that a 
check only reads
+        // volatile fields, unlike `isFailed()` and `isCompleted()`, which 
lock the task.
+        val ended = endedFlag(taskContext)
+        val task = new WeakReference(taskContext)
+        () => ended.get() || Option(task.get()).forall(_.isInterrupted())
+      case Some(taskContext) =>
+        () => taskContext.isInterrupted()
+      case None =>
+        // Outside of a task, as in tests, nothing interrupts the library.
+        () => false
+    }
     new NativeLibrary.Task(
       context.map(_.stageId()).getOrElse(-1),
       context.map(_.stageAttemptNumber()).getOrElse(-1),
       partitionId.orElse(context.map(_.partitionId())).getOrElse(-1),
       context.map(_.attemptNumber()).getOrElse(0),
       taskAttemptId.orElse(context.map(_.taskAttemptId())).getOrElse(-1L),
-      () => context.exists(_.isInterrupted()))
+      interrupted)
   }
 }
 
diff --git 
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc 
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index 4fcb530179cd..87551a695b31 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -39,16 +39,17 @@
 //                   validity buffer, a null_count of -1 instead. With the 
option
 //                   "wait_for_interrupt", the stream of a partition creates 
the file
 //                   <wait_for_interrupt>.waiting, waits until its task is 
killed, and then creates
-//                   the file <wait_for_interrupt> and fails. With the option 
"slow_read", the
-//                   get_next that creates the file <slow_read>.reading, the 
first one, then waits
-//                   until the file <slow_read>.continue exists, for at most a 
minute; with the
-//                   option "fail" set to "slow_read", each get_next then 
fails. After the wait,
-//                   it creates the file <slow_read>.interrupted with what 
is_interrupted returns
-//                   for the task context, "true" for one that Spark freed 
meanwhile. Releasing
-//                   the stream creates the file <slow_read>, which says 
whether Spark released it
-//                   "during get_next", which it must not, or "after get_next" 
otherwise. With
-//                   the option "ignore_read_schema", read returns all the 
columns of the reader,
-//                   rather than only those of the schema to read.
+//                   the file <wait_for_interrupt> and fails. With the option 
"released_path",
+//                   releasing the stream of a partition creates that file. 
With the option
+//                   "slow_read", the get_next that creates the file 
<slow_read>.reading, the first
+//                   one, then waits until the file <slow_read>.continue 
exists, for at most a
+//                   minute; with the option "fail" set to "slow_read", each 
get_next then fails.
+//                   After the wait, it creates the file 
<slow_read>.interrupted with what
+//                   is_interrupted returns for the task context, "true" once 
the task ended.
+//                   Releasing the stream creates the file <slow_read>, which 
says
+//                   whether Spark released it "during get_next", which it 
must not, or "after
+//                   get_next" otherwise. With the option 
"ignore_read_schema", read returns all
+//                   the columns of the reader, rather than only those of the 
schema to read.
 //   native_sink     Writes the rows as CSV files under the directory in the 
option "path". With
 //                   the option "commit_messages", a data writer passes its 
commit message that
 //                   many times. Creating and closing a data writer create a 
file named after
@@ -67,7 +68,8 @@
 //                   read to that file, and with "ranges_path", each call to 
stream_partitions
 //                   appends the start and the end offsets that it got to that 
file, as a line.
 //                   The partitions and the state that stream_partitions 
returns have the number of
-//                   the call, which read checks.
+//                   the call, which read checks. With the option 
"state_padding", the state has
+//                   that many more bytes.
 //
 // native_range supports batch reads, native_counter micro-batch streaming 
reads and native_sink
 // batch and streaming writes, which may replace the existing data. With the 
option
@@ -919,6 +921,7 @@ struct Stream {
   const SparkDsTaskContext* task;
   std::string wait_for_interrupt;
   std::string last_error;
+  std::string released_path;
   std::string slow_read;
   bool fail_slow_read;
   // task->is_interrupted, kept to call it without reading the task context, 
which a regression
@@ -976,9 +979,8 @@ int streamGetNext(ArrowArrayStream* stream, ArrowArray* 
out) {
         for (int i = 0; i < 6000 && stat(go.c_str(), &info) != 0; i++) {
           std::this_thread::sleep_for(std::chrono::milliseconds(10));
         }
-        // The task context must still be valid, as a library that checks it 
here expects. Spark
-        // only looks its address up, and answers true for a context that it 
freed, unless a new
-        // one got the same address, which no other read in these tests 
creates meanwhile.
+        // The task context must still be valid, as a library that checks it 
here expects, and
+        // it tells a read that its task ended.
         bool interrupted = state->is_interrupted(state->task);
         writeFile(state->slow_read + ".interrupted", interrupted ? "true" : 
"false");
       }
@@ -1002,6 +1004,11 @@ const char* streamGetLastError(ArrowArrayStream* stream) 
{
 
 void streamRelease(ArrowArrayStream* stream) {
   auto* state = static_cast<Stream*>(stream->private_data);
+  try {
+    if (!state->released_path.empty()) writeFile(state->released_path, "");
+  } catch (...) {
+    // No exception may cross the interface, and release cannot fail.
+  }
   stream->release = nullptr;
   if (!state->slow_read.empty()) {
     bool during = state->in_get_next;
@@ -1200,7 +1207,7 @@ int createReader(void* handle, struct ArrowSchema* 
schema, const SparkDsReaderIn
     reader->state["partitions"] = get(source->options, "partitions", "2");
     reader->state["batch_size"] = get(source->options, "batch_size", "1000");
     for (const char* option : {"fail", "slice", "short", "null_rows", 
"wait_for_interrupt",
-                               "task_path", "states", "stateless", "slow_read",
+                               "task_path", "states", "stateless", 
"slow_read", "released_path",
                                "ignore_read_schema"}) {
       reader->state[option] = get(source->options, option, "");
     }
@@ -1246,9 +1253,11 @@ int createStreamReader(void* handle, struct ArrowSchema* 
schema,
     reader->state["batch_size"] = get(source->options, "batch_size", "1000");
     for (const char* option :
          {"commit_path", "bad_offset", "bad_initial_offset", "closed_path", 
"starts_path",
-          "initial_offsets_path", "ranges_path", "states", "stateless", 
"fail"}) {
+          "initial_offsets_path", "ranges_path", "states", "stateless", 
"task_path", "fail"}) {
       reader->state[option] = get(source->options, option, "");
     }
+    std::string padding = get(source->options, "state_padding", "");
+    if (!padding.empty()) reader->state["padding"] = 
std::string(std::stoll(padding), 'x');
     *out = reader;
   });
 }
@@ -1495,8 +1504,11 @@ int readPartition(const SparkDsTaskContext* task, const 
SparkDsReadInfo* info,
                                       get(state, "slice", "") == "true", 
get(state, "short", ""),
                                       get(state, "null_rows", ""), task,
                                       get(state, "wait_for_interrupt", ""), "",
+                                      get(state, "released_path", ""),
                                       get(state, "slow_read", ""),
                                       get(state, "fail", "") == "slow_read", 
task->is_interrupted};
+    // Initializes the stream without get_next, which Spark must reject, and 
release.
+    if (get(state, "fail", "") == "null_get_next") stream->get_next = nullptr;
   });
 }
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 94e2b5a78b88..92ef0d87de92 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
 import org.apache.spark.sql.connector.catalog.{Table, TableCapability, 
TableProvider}
 import org.apache.spark.sql.connector.expressions.Transform
-import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, 
PartitionReaderFactory}
 import org.apache.spark.sql.connector.read.streaming.{ReadLimit, 
SupportsAdmissionControl}
 import org.apache.spark.sql.connector.write.{DataWriter, 
PhysicalWriteInfoImpl, WriterCommitMessage}
 import org.apache.spark.sql.execution.{ApplyColumnarRulesAndInsertTransitions, 
ColumnarRule, ColumnarToRowExec, RowToColumnConverter, SparkPlan}
@@ -1110,10 +1110,8 @@ class TestRangeStreamReader(end: Long, schema: 
StructType, table: String)
     Array(TestRangePartition(offset(start), offset(end)))
   }
 
-  override def read(partition: InputPartition): PartitionReader[ColumnarBatch] 
= {
-    val range = partition.asInstanceOf[TestRangePartition]
-    new TestBatchReader(range.lo, range.hi, schema)
-  }
+  override def readerFactory(): PartitionReaderFactory =
+    new TestRangeReaderFactory(schema, arrowBatchSize = 0, ownedBatches = 
false)
 }
 
 object TestRangeStreamReader {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index ad1982253cd2..1928bbac3722 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -16,7 +16,7 @@
  */
 package org.apache.spark.sql.execution.datasources.v2.ffi
 
-import java.io.File
+import java.io.{ByteArrayOutputStream, File, ObjectOutputStream}
 import java.nio.charset.StandardCharsets.UTF_8
 import java.nio.file.{AccessDeniedException, Files, NoSuchFileException, Paths}
 import java.nio.file.StandardCopyOption.REPLACE_EXISTING
@@ -30,19 +30,22 @@ import java.util.zip.{ZipException, ZipFile}
 
 import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
+import scala.reflect.ClassTag
 
-import org.apache.arrow.vector.BigIntVector
+import org.apache.arrow.c.{ArrowArray, ArrowArrayStream, ArrowSchema, 
CDataDictionaryProvider, Data}
+import org.apache.arrow.memory.BufferAllocator
+import org.apache.arrow.vector.{BigIntVector, VectorSchemaRoot}
 import org.scalatest.time.SpanSugar._
 
-import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TestUtils}
+import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TaskContext, TaskKilledException, TestUtils}
 import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
 import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, 
QueryTest, Row}
 import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
 import org.apache.spark.sql.classic.{DataFrame, SparkSession}
 import org.apache.spark.sql.connector.catalog.SupportsRead
-import org.apache.spark.sql.connector.read.SupportsPushDownRequiredColumns
-import org.apache.spark.sql.connector.read.streaming.{Offset, ReadLimit, 
SupportsAdmissionControl}
+import org.apache.spark.sql.connector.read.{InputPartition, 
SupportsArrowStreamRead, SupportsPushDownRequiredColumns}
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, 
Offset, ReadLimit, SupportsAdmissionControl}
 import org.apache.spark.sql.connector.write.PhysicalWriteInfoImpl
 import org.apache.spark.sql.execution.WholeStageCodegenExec
 import org.apache.spark.sql.execution.datasources.DataSource
@@ -58,6 +61,7 @@ import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.{ArrowUtils, CaseInsensitiveStringMap}
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch}
+import org.apache.spark.unsafe.Platform
 import org.apache.spark.util.{ThreadUtils, Utils}
 
 /**
@@ -1669,6 +1673,497 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
       .mode("append")
       .save()
     check(writeTasks)
+
+    // A read gets the index of its partition in the scan, also when a task 
reads several
+    // partitions, or the partitions of several scans: the task attempt that 
read each partition.
+    def readers(dir: File): Map[Int, Long] = dir.listFiles().map { file =>
+      file.getName.toInt -> Files.readString(file.toPath).split(",")(5).toLong
+    }.toMap
+    val (first, second) = (Utils.createTempDir(), Utils.createTempDir())
+    checkRows(
+      range(session, "end" -> "6", "task_path" -> first.getPath).select("id")
+        .union(range(session, "end" -> "6", "task_path" -> 
second.getPath).select("id")),
+      (0L until 6L).flatMap(i => Seq(Row(i), Row(i))))
+    assert(readers(first).keySet == Set(0, 1) && readers(second).keySet == 
Set(0, 1))
+    val coalesced = Utils.createTempDir()
+    checkRows(range(session, "end" -> "6", "task_path" -> 
coalesced.getPath).select("id")
+      .coalesce(1), (0L until 6L).map(Row(_)))
+    assert(readers(coalesced).keySet == Set(0, 1) && 
readers(coalesced).values.toSet.size == 1)
+    // So does a read of a micro-batch.
+    val (firstStream, secondStream) = (Utils.createTempDir(), 
Utils.createTempDir())
+    def counter(dir: File): DataFrame = 
session.readStream.format("native_counter")
+      .option("max_offset", "4").option("task_path", dir.getPath).load()
+    val query = 
counter(firstStream).union(counter(secondStream)).writeStream.format("noop").start()
+    try query.processAllAvailable() finally query.stop()
+    assert(readers(firstStream).keySet == Set(0) && 
readers(secondStream).keySet == Set(0))
+  }
+
+  nativeTest("open the partitions of batch and micro-batch scans as Arrow C 
streams") {
+    val session = newSession(defaultPackage)
+    // The reader factory and the partitions of the batch scan of a query, 
which an engine gets
+    // from its plan.
+    def plannedScan(df: DataFrame): (SupportsArrowStreamRead, 
Seq[InputPartition]) = {
+      val scan = df.queryExecution.executedPlan.collectFirst { case s: 
BatchScanExec => s }.get
+      (scan.readerFactory.asInstanceOf[SupportsArrowStreamRead], 
scan.inputPartitions)
+    }
+    // The batch scan of only some of the columns from a library that does not 
prune the others
+    // still reads only them through its streams, since read returns exactly 
the columns of its
+    // schema to read.
+    val readOnly = newSession(defaultPackage, createPackage(readOnlyLibrary, 
dataSources("ro_")))
+    val (projectedFactory, projectedPartitions) =
+      plannedScan(readOnly.read.format("ro_native_range").option("end", 
"4").load().select("id"))
+    val projectedReads = readOnly.artifactManager.withResources {
+      readOnly.sparkContext.parallelize(Seq(0), 1).flatMap { _ =>
+        projectedPartitions.map { partition =>
+          val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+          try {
+            projectedFactory.openArrowStream(partition, stream.memoryAddress())
+            try TestArrowEngine.readNamesAndIdsInPlace(stream) finally 
stream.release()
+          } finally {
+            stream.close()
+          }
+        }
+      }.collect().toSeq
+    }
+    assert(projectedReads.map(_._1).distinct == Seq(Seq("id")))
+    assert(projectedReads.flatMap(_._2).sorted == (0L until 4L))
+    assert(NativeLibrary.openEngineStreams() == 0)
+    // Runs `f` in a task that only gets the files of the session. It is the 
task of the partition
+    // 1, so that the partition of the scan that a read gets is not the 
partition of its task.
+    def inTask[T: ClassTag](f: => T): T = 
session.artifactManager.withResources {
+      session.sparkContext.parallelize(Seq(0, 1), 2).flatMap(i => if (i == 1) 
Seq(f) else Nil)
+        .collect().head
+    }
+    import TestArrowEngine.{callbacks, readIds, readIdsInPlace}
+    // Reads the given partitions in a single task, as an engine does.
+    def readAll(factory: SupportsArrowStreamRead, partitions: 
Seq[InputPartition]): Seq[Long] = {
+      inTask {
+        partitions.flatMap { partition =>
+          val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine", 
0, Long.MaxValue)
+          try {
+            val stream = ArrowArrayStream.allocateNew(allocator)
+            try factory.openArrowStream(partition, stream.memoryAddress()) 
catch {
+              case e: Throwable =>
+                stream.close()
+                throw e
+            }
+            readIds(stream, allocator)
+          } finally {
+            allocator.close()
+          }
+        }
+      }
+    }
+    // Checks that `f` opens at most `delta` more task contexts than it 
closes. The count is global
+    // to the JVM, where the tasks of earlier tests may still close their 
contexts, so it is read
+    // right before `f`, and may be lower afterwards.
+    def checkContexts[T](delta: Int)(f: => T): T = {
+      val before = NativeLibrary.openTaskContexts()
+      val result = f
+      assert(NativeLibrary.openTaskContexts() <= before + delta)
+      result
+    }
+    // The size of struct ArrowArrayStream, which the layout test compares 
with the header.
+    val streamSize = NativeLibrary.layout().get("ArrowArrayStream").longValue
+
+    // An engine reads both partitions of the scan in a single task.
+    val taskDir = Utils.createTempDir()
+    val (factory, partitions) =
+      plannedScan(range(session, "end" -> "6", "task_path" -> 
taskDir.getPath).select("id"))
+    assert(partitions.length == 2)
+    assert(readAll(factory, partitions).sorted == (0L until 6L))
+    // The library got the partition of the scan that it reads, not the one of 
the task.
+    val tasks = taskDir.listFiles().map { file =>
+      file.getName.toInt -> 
Files.readString(file.toPath).split(",").map(_.toLong).toSeq
+    }.toMap
+    assert(tasks.keySet == Set(0, 1) && tasks.values.map(_(5)).toSet.size == 
1, tasks)
+    assert(NativeLibrary.openEngineStreams() == 0)
+    // The reads of a task share the flag that tells them that the task ended, 
with a single pair
+    // of listeners, which remove it once the task ended. The tasks of earlier 
tests may still
+    // be ending.
+    eventually(timeout(1.minute))(assert(NativeTask.tasksWithEndedFlags == 0))
+    val flagsInTask = inTask {
+      val ids = partitions.flatMap { partition =>
+        val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+        try {
+          factory.openArrowStream(partition, stream.memoryAddress())
+          try readIdsInPlace(stream) finally stream.release()
+        } finally {
+          stream.close()
+        }
+      }
+      assert(ids.sorted == (0L until 6L))
+      NativeTask.tasksWithEndedFlags
+    }
+    assert(flagsInTask == 1 && NativeTask.tasksWithEndedFlags == 0)
+
+    // The engine may release the stream after its task ended, and on another 
thread: Spark frees
+    // the context of the task once the library released the stream.
+    val released = new File(Utils.createTempDir(), "released")
+    val (laterFactory, laterPartitions) =
+      plannedScan(range(session, "end" -> "6", "released_path" -> 
released.getPath).select("id"))
+    // Opens a stream in a task, and returns it unreleased, with what `read` 
returned from it in
+    // the task.
+    def openInTask[T: ClassTag](factory: SupportsArrowStreamRead, partition: 
InputPartition)(
+        read: ArrowArrayStream => T): (Long, T) = inTask {
+      val stream = Platform.allocateMemory(streamSize)
+      Platform.setMemory(stream, 0.toByte, streamSize)
+      factory.openArrowStream(partition, stream)
+      (stream, read(ArrowArrayStream.wrap(stream)))
+    }
+    // The engine reads the stream to its end in the task, but releases it 
later.
+    val (address, readInTask) =
+      checkContexts(1)(openInTask(laterFactory, 
laterPartitions.head)(readIdsInPlace))
+    // A copy of the stream, as an engine that moves the stream may leave 
behind.
+    val copy = Platform.allocateMemory(streamSize)
+    Utils.tryWithSafeFinally {
+      Platform.copyMemory(null, address, null, copy, streamSize)
+      assert(readInTask == (0L until 3L))
+      assert(NativeLibrary.openEngineStreams() == 1 && !released.exists())
+      checkContexts(-1) {
+        ArrowArrayStream.wrap(address).release()
+        assert(NativeLibrary.openEngineStreams() == 0 && released.exists())
+      }
+      // Once the stream is released, the callbacks of the copy fail with 
EINVAL, and its release
+      // only marks it released.
+      checkContexts(0) {
+        val array = ArrowArray.allocateNew(ArrowUtils.rootAllocator)
+        val stale = try 
intercept[Exception](ArrowArrayStream.wrap(copy).getNext(array)) finally {
+          array.close()
+        }
+        assert(Utils.exceptionString(stale)
+          .contains("The stream was released, or is not a stream of Spark."))
+        ArrowArrayStream.wrap(copy).release()
+        assert(ArrowArrayStream.wrap(copy).snapshot().release == 0)
+        assert(NativeLibrary.openEngineStreams() == 0)
+      }
+    } {
+      // A failure above may leave the stream open, which would keep the 
context of its task.
+      if (ArrowArrayStream.wrap(address).snapshot().release != 0) {
+        ArrowArrayStream.wrap(address).release()
+      }
+      Platform.freeMemory(address)
+      Platform.freeMemory(copy)
+    }
+
+    // Once the task ended, its context tells the library that it was 
interrupted, so that the
+    // library stops waiting.
+    val marker = new File(Utils.createTempDir(), "interrupted")
+    val (waitingFactory, waitingPartitions) = plannedScan(
+      range(session, "end" -> "6", "wait_for_interrupt" -> 
marker.getPath).select("id"))
+    checkContexts(0) {
+      val (waiting, _) = openInTask(waitingFactory, waitingPartitions.head)(_ 
=> ())
+      val error = intercept[Exception] {
+        readIds(ArrowArrayStream.wrap(waiting), ArrowUtils.rootAllocator)
+      }
+      assert(Utils.exceptionString(error).contains("the task was killed"))
+      assert(Files.readString(marker.toPath) == "interrupted")
+      assert(NativeLibrary.openEngineStreams() == 0)
+      Platform.freeMemory(waiting)
+    }
+
+    // So does the context of a stream that an engine reads in its task once 
the task is killed,
+    // although the task did not end.
+    val killedMarker = new File(Utils.createTempDir(), "interrupted")
+    val (killedFactory, killedPartitions) = plannedScan(range(session, "end" 
-> "6",
+      "partitions" -> "1", "wait_for_interrupt" -> 
killedMarker.getPath).select("id"))
+    val contextsBeforeKill = NativeLibrary.openTaskContexts()
+    val jobGroup = s"native-data-source-${UUID.randomUUID()}"
+    val killError = new AtomicReference[Throwable]()
+    val engineThread = new Thread(() => {
+      session.sparkContext.setJobGroup(jobGroup, "an engine that waits", 
interruptOnCancel = true)
+      try {
+        inTask {
+          // Reads the stream on the thread of the task, which waits in 
get_next.
+          val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+          try {
+            killedFactory.openArrowStream(killedPartitions.head, 
stream.memoryAddress())
+            try readIdsInPlace(stream) finally stream.release()
+          } finally {
+            stream.close()
+          }
+        }
+      } catch {
+        case e: Throwable => killError.set(e)
+      }
+    })
+    engineThread.start()
+    eventually(timeout(1.minute)) {
+      assert(new File(killedMarker.getPath + ".waiting").isFile)
+    }
+    session.sparkContext.cancelJobGroup(jobGroup)
+    engineThread.join(TimeUnit.MINUTES.toMillis(1))
+    assert(!engineThread.isAlive && killError.get() != null)
+    eventually(timeout(1.minute)) {
+      assert(killedMarker.isFile && Files.readString(killedMarker.toPath) == 
"interrupted")
+      assert(NativeLibrary.openEngineStreams() == 0)
+      assert(NativeLibrary.openTaskContexts() <= contextsBeforeKill)
+    }
+
+    // So does the reader factory of a micro-batch scan, with the state of its 
micro-batch.
+    val streamTaskDir = Utils.createTempDir()
+    val query = 
session.readStream.format("native_counter").option("max_offset", "4")
+      .option("task_path", streamTaskDir.getPath).load()
+      .writeStream.format("noop").start()
+    val microBatchScan = try {
+      query.processAllAvailable()
+      
query.asInstanceOf[StreamingQueryWrapper].streamingQuery.lastExecution.executedPlan
+        .collectFirst { case s: MicroBatchScanExec => s }.get
+    } finally {
+      query.stop()
+    }
+    // Only the reads of the engine record their tasks from now on.
+    streamTaskDir.listFiles().foreach(_.delete())
+    val streamFactory = 
microBatchScan.readerFactory.asInstanceOf[SupportsArrowStreamRead]
+    val streamPartitions = microBatchScan.inputPartitions.toSeq
+    checkContexts(0) {
+      assert(readAll(streamFactory, streamPartitions).sorted == (0L until 4L))
+    }
+    assert(NativeLibrary.openEngineStreams() == 0)
+    // The library got the partition of the scan, the only one of the 
micro-batch.
+    assert(streamTaskDir.listFiles().map(_.getName.toInt).toSet == Set(0))
+    // The partitions of a micro-batch keep the state that the stream reader 
returned with them:
+    // a reader factory that an engine keeps reads them with it, also once 
Spark planned the next
+    // micro-batch, and so does the reader of Spark.
+    def microBatchStream(options: (String, String)*): MicroBatchStream = {
+      val relation = 
session.readStream.format("native_counter").options(options.toMap).load()
+        .queryExecution.analyzed.collectFirst { case relation: 
StreamingRelationV2 => relation }.get
+      
relation.table.asInstanceOf[SupportsRead].newScanBuilder(relation.extraOptions)
+        .build().toMicroBatchStream(Utils.createTempDir().getPath)
+    }
+    def plan(stream: MicroBatchStream, start: Int, end: Int): 
Seq[InputPartition] = {
+      def offset(i: Int): Offset = 
stream.deserializeOffset(s"""{"offset":$i}""")
+      stream.planInputPartitions(offset(start), offset(end)).toSeq
+    }
+    def serialized(partition: NativeMicroBatchPartition): Seq[Byte] = {
+      val bytes = new ByteArrayOutputStream()
+      Utils.tryWithResource(new 
ObjectOutputStream(bytes))(_.writeObject(partition))
+      bytes.toByteArray.toSeq
+    }
+    val padding = (NativeDataSourceStreamReader.maxInlineStateBytes + 
1).toString
+    val stream = microBatchStream("max_offset" -> "4", "state_padding" -> 
padding)
+    try {
+      val first = plan(stream, 0, 2)
+      val keptFactory = stream.createReaderFactory()
+      val second = plan(stream, 2, 4)
+      // The executors get a state larger than 64 KiB once for the 
micro-batch, as a broadcast that
+      // its partitions keep, rather than with the partition of each task.
+      Seq(first, second).foreach { partitions =>
+        val partition = partitions.head.asInstanceOf[NativeMicroBatchPartition]
+        assert(partition.state.isRight && partition.readerState.length > 
padding.toInt)
+        
assert(!serialized(partition).containsSlice(partition.readerState.toSeq))
+      }
+      checkContexts(0) {
+        assert(readAll(keptFactory.asInstanceOf[SupportsArrowStreamRead], 
first) == Seq(0L, 1L))
+        assert(readAll(keptFactory.asInstanceOf[SupportsArrowStreamRead], 
second) == Seq(2L, 3L))
+        val read = inTask {
+          first.flatMap { partition =>
+            val reader = keptFactory.createColumnarReader(partition)
+            try {
+              val ids = ArrayBuffer.empty[Long]
+              while (reader.next()) {
+                val batch = reader.get()
+                (0 until batch.numRows()).foreach(i => ids += 
batch.column(0).getLong(i))
+              }
+              ids.toSeq
+            } finally {
+              reader.close()
+            }
+          }
+        }
+        assert(read == Seq(0L, 1L))
+      }
+    } finally {
+      stream.stop()
+    }
+    // A smaller state, such as a few offsets, goes with the partition of each 
task, which costs
+    // less than a broadcast.
+    val small = microBatchStream("max_offset" -> "4")
+    try {
+      val partition = plan(small, 0, 
2).head.asInstanceOf[NativeMicroBatchPartition]
+      assert(partition.state.isLeft && partition.readerState.nonEmpty)
+      assert(serialized(partition).containsSlice(partition.readerState.toSeq))
+      checkContexts(0) {
+        
assert(readAll(small.createReaderFactory().asInstanceOf[SupportsArrowStreamRead],
+          Seq(partition)) == Seq(0L, 1L))
+      }
+    } finally {
+      small.stop()
+    }
+    // A micro-batch without partitions broadcasts no state, although the 
stream reader returned
+    // a large one.
+    val (counterName, counterLocation) =
+      NativeDataSourceRegistry.lookup("native_counter", 
session.sessionState.conf).get
+    val streamReader = new NativeDataSource(counterLocation, counterName,
+        new CaseInsensitiveStringMap(Map("max_offset" -> "4", "state_padding" 
-> padding).asJava))
+      .streamReader(StructType.fromDDL("id BIGINT"), 
Utils.createTempDir().getPath)
+      .asInstanceOf[NativeDataSourceStreamReader]
+    try {
+      def offset(i: Int): String = s"""{"offset":$i}"""
+      assert(streamReader.partitions(offset(0), offset(0)).isEmpty)
+      assert(streamReader.uncommittedStates.isEmpty)
+      // The broadcast of a micro-batch is destroyed once the micro-batch is 
committed, and those
+      // of the micro-batches after it stay.
+      assert(streamReader.partitions(offset(0), offset(2)).nonEmpty)
+      assert(streamReader.partitions(offset(2), offset(4)).nonEmpty)
+      assert(streamReader.partitions(offset(4), offset(6)).nonEmpty)
+      val states = streamReader.uncommittedStates
+      assert(states.length == 3 && states.forall(_.isValid))
+      streamReader.commit(offset(2))
+      assert(states.map(_.isValid) == Seq(false, true, true))
+      assert(streamReader.uncommittedStates == states.tail)
+      // Spark may not commit every micro-batch, as with async progress 
tracking, so a commit also
+      // destroys the broadcasts of the micro-batches before it, also when the 
committed one has no
+      // broadcast, as a micro-batch without partitions, here at an end not 
planned before.
+      assert(streamReader.partitions(offset(7), offset(7)).isEmpty)
+      streamReader.commit(offset(7))
+      assert(states.forall(!_.isValid) && 
streamReader.uncommittedStates.isEmpty)
+      // stop() destroys the broadcasts of the micro-batches that are not 
committed.
+      assert(streamReader.partitions(offset(7), offset(9)).nonEmpty)
+      val lastState = streamReader.uncommittedStates
+      assert(lastState.length == 1 && lastState.head.isValid)
+      streamReader.stop()
+      assert(!lastState.head.isValid && streamReader.uncommittedStates.isEmpty)
+    } finally {
+      streamReader.stop()
+    }
+    // A micro-batch without a state has no bytes to read the partitions with.
+    val stateless = microBatchStream("max_offset" -> "4", "states" -> "0")
+    try {
+      val partitions = plan(stateless, 0, 
2).map(_.asInstanceOf[NativeMicroBatchPartition])
+      assert(partitions.map(_.state.isLeft) == Seq(true) && 
partitions.head.readerState.isEmpty)
+    } finally {
+      stateless.stop()
+    }
+
+    // It runs in the running task, with a stream whose release callback is 
NULL.
+    val outside = intercept[IllegalStateException] {
+      factory.openArrowStream(partitions.head, 0)
+    }
+    assert(outside.getMessage.contains("must be called in the running task"))
+    val completed = intercept[Exception] {
+      session.artifactManager.withResources {
+        session.sparkContext.parallelize(Seq(0), 1).foreach { _ =>
+          TaskContext.get().addTaskCompletionListener[Unit] { _ =>
+            factory.openArrowStream(partitions.head, 0)
+          }
+        }
+      }
+    }
+    assert(Utils.exceptionString(completed).contains("must be called in the 
running task"))
+    // Nor in a task that failed, while its failure listeners run.
+    val failed = intercept[Exception] {
+      session.artifactManager.withResources {
+        session.sparkContext.parallelize(Seq(0), 1).foreach { _ =>
+          TaskContext.get().addTaskFailureListener { (_, _) =>
+            factory.openArrowStream(partitions.head, 0)
+          }
+          throw new IllegalStateException("injected failure of the task")
+        }
+      }
+    }
+    assert(Utils.exceptionString(failed).contains("injected failure of the 
task"))
+    assert(Utils.exceptionString(failed).contains("must be called in the 
running task"))
+    // A task that was killed gets the error of its kill.
+    val killedContext = TaskContext.empty()
+    killedContext.markInterrupted("killed for the test")
+    TaskContext.setTaskContext(killedContext)
+    val killed = try intercept[TaskKilledException] {
+      factory.openArrowStream(partitions.head, 0)
+    } finally {
+      TaskContext.unset()
+    }
+    assert(killed.reason == "killed for the test")
+    val noStream = 
intercept[Exception](inTask(factory.openArrowStream(partitions.head, 0)))
+    assert(Utils.exceptionString(noStream).contains("whose release callback is 
NULL"))
+    // The arguments are checked before the partition, which must be of the 
kind of the scan of the
+    // factory: a partition of a micro-batch fails for a batch scan, and the 
other way around.
+    val outsideWithOtherKind = intercept[IllegalStateException] {
+      streamFactory.openArrowStream(partitions.head, 0)
+    }
+    assert(outsideWithOtherKind.getMessage.contains("must be called in the 
running task"))
+    val otherKinds = checkContexts(0) {
+      inTask {
+        Seq(streamFactory -> partitions.head, factory -> 
streamPartitions.head).map {
+          case (kindFactory, partition) =>
+            val stream = Platform.allocateMemory(streamSize)
+            try {
+              Platform.setMemory(stream, 0.toByte, streamSize)
+              kindFactory.openArrowStream(partition, stream)
+              None
+            } catch {
+              case e: IllegalArgumentException => Some(e.getMessage)
+            } finally {
+              Platform.freeMemory(stream)
+            }
+        }
+      }
+    }
+    assert(otherKinds == Seq(
+      Some("openArrowStream needs a partition of the scan of the reader 
factory, a " +
+        s"NativeMicroBatchPartition, but got 
${classOf[NativeInputPartition].getName}."),
+      Some("openArrowStream needs a partition of the scan of the reader 
factory, a " +
+        s"NativeInputPartition, but got 
${classOf[NativeMicroBatchPartition].getName}.")))
+    // A stream whose release callback is not NULL, such as an open one, is 
left as it is.
+    val (rejected, unchanged, ids) = checkContexts(0) {
+      inTask {
+        val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine", 
0, Long.MaxValue)
+        try {
+          val stream = ArrowArrayStream.allocateNew(allocator)
+          factory.openArrowStream(partitions.head, stream.memoryAddress())
+          val before = callbacks(stream)
+          val rejected = try {
+            factory.openArrowStream(partitions.head, stream.memoryAddress())
+            None
+          } catch {
+            case e: IllegalArgumentException => Some(e.getMessage)
+          }
+          (rejected, callbacks(stream) == before, readIds(stream, allocator))
+        } finally {
+          allocator.close()
+        }
+      }
+    }
+    assert(rejected.exists(_.contains("whose release callback is NULL")), 
rejected)
+    assert(unchanged && ids == (0L until 3L))
+
+    // If it fails, the stream is released, including the stream of the 
library that Spark rejects.
+    Seq(
+      "open" -> "injected failure in open",
+      "uninitialized" ->
+        "The library did not initialize the stream, or a callback of the 
stream is NULL.",
+      "null_get_next" ->
+        "The library did not initialize the stream, or a callback of the 
stream is NULL."
+    ).foreach { case (fail, msg) =>
+      val failReleased = new File(Utils.createTempDir(), "released")
+      val (failing, failingPartitions) =
+        plannedScan(range(session, "fail" -> fail, "released_path" -> 
failReleased.getPath))
+      val e = checkContexts(0) {
+        intercept[Exception] {
+          inTask {
+            val stream = ArrowArrayStream.allocateNew(ArrowUtils.rootAllocator)
+            try {
+              failing.openArrowStream(failingPartitions.head, 
stream.memoryAddress())
+            } catch {
+              // Keeps the failure, but not as the cause, which findError 
would find.
+              case e: Throwable if stream.snapshot().release != 0 =>
+                val notReleased = new IllegalStateException(
+                  s"The stream is not released after the failure $fail.")
+                notReleased.addSuppressed(e)
+                throw notReleased
+            } finally {
+              stream.close()
+            }
+          }
+        }
+      }
+      checkError(
+        exception = findError(e, "NATIVE_DATA_SOURCE_ERROR"),
+        condition = "NATIVE_DATA_SOURCE_ERROR",
+        parameters = Map("action" -> "read", "name" -> "native_range", "msg" 
-> msg))
+      assert(failReleased.exists() == (fail == "null_get_next"), fail)
+      assert(NativeLibrary.openEngineStreams() == 0)
+    }
   }
 
   nativeTest("tell a read that its task was killed") {
@@ -1678,6 +2173,7 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
       "wait_for_interrupt" -> marker.getPath)
     val jobGroup = s"native-data-source-${UUID.randomUUID()}"
     val error = new AtomicReference[Throwable]()
+    val contexts = NativeLibrary.openTaskContexts()
     val thread = new Thread(() => {
       session.sparkContext.setJobGroup(jobGroup, "a read that waits", 
interruptOnCancel = true)
       try df.collect() catch {
@@ -1694,9 +2190,12 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     thread.join(TimeUnit.MINUTES.toMillis(1))
     assert(!thread.isAlive)
     assert(error.get() != null)
-    // The job fails right away, and its task ends once the library sees that 
it was killed.
+    // The job fails right away, and its task ends once the library sees that 
it was killed. The
+    // test also waits until the task closed the context of its read, which 
the tests that count
+    // the contexts would otherwise count.
     eventually(timeout(1.minute)) {
       assert(marker.isFile && Files.readString(marker.toPath) == "interrupted")
+      assert(NativeLibrary.openTaskContexts() <= contexts)
     }
   }
 
@@ -1706,6 +2205,10 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     val marker = new File(Utils.createTempDir(), "released")
     val reading = marker.getPath + ".reading"
     val df = range(session, "end" -> "10", "partitions" -> "1", "slow_read" -> 
marker.getPath)
+    // The contexts of the tasks of earlier tests are closed first, so that 
the counts below are
+    // exactly those of this test, also the one of the context that stays open.
+    eventually(timeout(1.minute))(assert(NativeLibrary.openTaskContexts() == 
0))
+    val contexts = 0
     try {
       // PipedRDD reads the scan on a thread of its own, which the task does 
not wait for. The
       // task completes without reading the output of the command, once that 
thread is in
@@ -1723,16 +2226,18 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
       assert(count == 0)
       assert(new File(reading).isFile, "get_next was not called.")
       if (marker.isFile) fail(s"The stream was released: 
${Files.readString(marker.toPath)}.")
+      // The task completed, but its context stays valid until the stream is 
released.
+      assert(NativeLibrary.openTaskContexts() == contexts + 1)
     } finally {
       new File(marker.getPath + ".continue").createNewFile()
     }
-    // The thread in get_next releases the stream once get_next returns.
+    // The thread in get_next releases the stream once get_next returns, and 
then the context.
     eventually(timeout(1.minute)) {
       assert(marker.isFile && Files.readString(marker.toPath) == "after 
get_next")
+      assert(NativeLibrary.openTaskContexts() == contexts)
     }
-    // The task context was still valid once get_next continued: Spark answers 
that a context
-    // that it freed is interrupted, and this task was not.
-    assert(Files.readString(new File(marker.getPath + ".interrupted").toPath) 
== "false")
+    // The read learned that its task ended, so that it may stop reading.
+    assert(Files.readString(new File(marker.getPath + ".interrupted").toPath) 
== "true")
   }
 
   nativeTest("close a reader while another thread reads it") {
@@ -1761,7 +2266,8 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
           case e: ExecutionException => Left(e.getCause)
         }
         assert(Files.readString(marker.toPath) == "after get_next")
-        // The task context was still valid once get_next continued, as in the 
test above.
+        // The context was still valid once get_next continued: Spark answers 
that a context that
+        // it freed is interrupted, and a read outside of a task never is.
         assert(Files.readString(new File(marker.getPath + 
".interrupted").toPath) == "false")
         // A reader that was closed reads nothing more, and closing it again 
does not fail.
         assert(!reader.next())
@@ -2556,3 +3062,79 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     check(noInitLibrary, "ni_", "The library does not export the function 
spark_ds_init.")
   }
 }
+
+/** What an engine does with the Arrow C stream of a partition of a native 
scan. */
+object TestArrowEngine {
+  /** The callbacks and the private data of a stream. */
+  def callbacks(stream: ArrowArrayStream): Seq[Long] = {
+    val snapshot = stream.snapshot()
+    Seq(snapshot.get_schema, snapshot.get_next, snapshot.get_last_error, 
snapshot.release,
+      snapshot.private_data)
+  }
+
+  /**
+   * Reads the ids of a stream to its end on the calling thread, without 
moving or releasing it, as
+   * an engine that releases the stream later does.
+   */
+  def readIdsInPlace(stream: ArrowArrayStream): Seq[Long] = 
readNamesAndIdsInPlace(stream)._2
+
+  /** The names of the columns of a stream and its ids, read as 
`readIdsInPlace` reads them. */
+  def readNamesAndIdsInPlace(stream: ArrowArrayStream): (Seq[String], 
Seq[Long]) = {
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("engine", 0, 
Long.MaxValue)
+    val dictionaries = new CDataDictionaryProvider()
+    try {
+      val schema = ArrowSchema.allocateNew(allocator)
+      val root = try {
+        stream.getSchema(schema)
+        VectorSchemaRoot.create(Data.importSchema(allocator, schema, 
dictionaries), allocator)
+      } finally {
+        schema.close()
+      }
+      try {
+        val ids = ArrayBuffer.empty[Long]
+        var more = true
+        while (more) {
+          // A new array for each batch, since the import closes it, as in 
NativePartitionReader.
+          val array = ArrowArray.allocateNew(allocator)
+          try {
+            stream.getNext(array)
+            // The stream ends with a released array.
+            more = array.snapshot().release != 0
+            if (more) {
+              Data.importIntoVectorSchemaRoot(allocator, array, root, 
dictionaries)
+              val vector = root.getVector("id").asInstanceOf[BigIntVector]
+              (0 until root.getRowCount).foreach(i => ids += vector.get(i))
+            }
+          } finally {
+            array.close()
+          }
+        }
+        (root.getSchema.getFields.asScala.map(_.getName).toSeq, ids.toSeq)
+      } finally {
+        root.close()
+      }
+    } finally {
+      dictionaries.close()
+      allocator.close()
+    }
+  }
+
+  /** Reads the ids of a stream on another thread, as an engine may, and 
releases the stream. */
+  def readIds(stream: ArrowArrayStream, allocator: BufferAllocator): Seq[Long] 
= {
+    // Moves the stream into a reader, which releases it once closed.
+    val reader = Data.importArrayStream(allocator, stream)
+    try {
+      ThreadUtils.runInNewThread("engine") {
+        val ids = ArrayBuffer.empty[Long]
+        while (reader.loadNextBatch()) {
+          val root = reader.getVectorSchemaRoot
+          val vector = root.getVector("id").asInstanceOf[BigIntVector]
+          (0 until root.getRowCount).foreach(i => ids += vector.get(i))
+        }
+        ids.toSeq
+      }
+    } finally {
+      reader.close()
+    }
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to