This is an automated email from the ASF dual-hosted git repository.

sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 69743391a88d [SPARK-60705][SQL] Do not release the stream of a native 
data source while get_next runs
69743391a88d is described below

commit 69743391a88df5220bd8e981236dfe27e8f65779
Author: Kousuke Saruta <[email protected]>
AuthorDate: Fri Oct 30 20:36:45 2026 +0900

    [SPARK-60705][SQL] Do not release the stream of a native data source while 
get_next runs
    
    ### What changes were proposed in this pull request?
    
    This PR makes `NativePartitionReader`, the reader of a partition of a 
native data source (SPARK-60101), release the Arrow stream of the library only 
once `get_next` returns, also when another thread closes the reader while 
`get_next` runs.
    
    - `close()` no longer releases the stream, the vectors, the task context 
and the allocator while a thread is in `next()`. It marks the reader as closed, 
and that `next()` releases them once it is done with the batch that `get_next` 
returned: it then returns false, or throws the error of the read. A failure to 
release them there is logged, since the thread that closed the reader no longer 
gets it. `close()` does not wait, so it does not keep the task from completing.
    - `next()` returns false once the reader is closed, so the reader never 
calls a released stream, and `close()` does nothing when it is called again. 
The reading thread may then close the reader again: `DataSourceRDD` closes a 
reader whose `next()` returns false, as well as from the completion listener. 
Before, a second `close()` failed with `NullPointerException: ArrowArrayStream 
is already closed`.
    - The constructor runs under the same lock, so that a `close()` on another 
thread sees the task and the vectors that it created.
    
    The test library gets an option `slow_read`: the first `get_next` waits 
until the test lets it continue, then records what `is_interrupted` returns for 
the task context, and releasing the stream records whether Spark released it 
during `get_next` or not. With the option `fail` set to `slow_read`, `get_next` 
then fails.
    
    ### Why are the changes needed?
    
    `spark_datasource.h` promises a library that the callbacks of the stream 
never run concurrently, and that the task context passed to `read` stays valid 
until the stream is released. When a task is killed, it also says that Spark 
only releases the stream once `get_next` returns.
    
    Spark did not keep these promises when another thread reads a partition. 
`DataSourceRDD` closes its readers from a task completion listener, and some 
operators read their input on a thread of their own that the task does not wait 
for. For example, `PipedRDD` reads its parent on its stdin writer thread. On 
SPARK-60101 as merged (99a1bf9a837, the same reader as its last PR commit 
1f4048404be), with a library whose `get_next` takes 2 seconds, 
`spark.read.format(...).load().rdd.pipe("true [...]
    
    `close()` hands the release to the thread in `next()` rather than waiting 
for it, since that thread can be in a `get_next` that waits for data: 
`task->is_interrupted` stays false for a task that completed, so waiting could 
keep the task from completing for as long as `get_next` blocks. Such a 
`get_next` still runs until it returns on its own, and keeps its thread and the 
resources of the read until then; SPARK-60733 is to tell the library to stop.
    
    Besides when the library gets the release, a closed reader now reads 
nothing more and can be closed again. As before, a batch that `next()` returned 
is only valid until the reader is closed, so a thread that still uses it after 
the task completed reads closed vectors.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Native data sources are not released yet (4.4.0).
    
    ### How was this patch tested?
    
    New tests in `NativeDataSourceSuite`, which hold the library in its first 
`get_next` until they let it continue, so that they do not depend on timing:
    - `release a stream only once get_next returns` reads a native data source 
through `rdd.pipe("cat")`, and completes the task while the stdin writer thread 
of `PipedRDD` is in `get_next`. It checks that the stream is not released when 
the task completes, that the task context is still valid when `get_next` 
continues, and that the library gets the release after `get_next` returns.
    - `close a reader while another thread reads it` closes a reader while 
another thread is in `next()`, and checks the same, that `next()` then returns 
false, or throws when `get_next` fails, and that a closed reader reads nothing 
more and can be closed again.
    
    The task context is checked through `is_interrupted`, which Spark answers 
with true for a context that it freed. Without the change in 
`NativePartitionReader`, both fail with `The stream was released: during 
get_next.` Each of the other parts of the change, except running the 
constructor under the lock and logging a failure to release the resources in 
`next()`, also makes a test fail when it is reverted.
    
    `NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass locally on Java 
22 on macOS, except two tests that are canceled without pandas, and the new 
tests passed in 5 runs, and in 5 more after the check of the task context was 
added.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Kiro CLI
    
    Closes #59914 from sarutak/native-ds-reader-close-race.
    
    Authored-by: Kousuke Saruta <[email protected]>
    Signed-off-by: Kousuke Saruta <[email protected]>
---
 .../datasources/v2/ffi/NativeDataSource.scala      | 69 ++++++++++++++---
 .../native-datasource/test_native_datasource.cc    | 87 +++++++++++++++++++---
 .../datasources/v2/ffi/NativeDataSourceSuite.scala | 86 ++++++++++++++++++++-
 3 files changed, 217 insertions(+), 25 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 9f5c6adec3b9..8c68b24602f7 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -20,6 +20,7 @@ import java.lang.ref.{Cleaner, WeakReference}
 import java.util
 import java.util.concurrent.ConcurrentHashMap
 import java.util.concurrent.atomic.AtomicLong
+import javax.annotation.concurrent.GuardedBy
 
 import scala.jdk.CollectionConverters._
 import scala.util.control.NonFatal
@@ -468,6 +469,15 @@ object NativeWriterCommitMessage {
 /**
  * Reads a partition of a native data source: each array of the Arrow stream 
exported by the
  * library is checked, and imported into the Arrow vectors that back the 
returned batch.
+ *
+ * A reader can be closed while another thread is in [[next]]: some operators 
read their input on
+ * a thread of their own that the task does not wait for, such as the stdin 
writer thread of
+ * `PipedRDD`, while `DataSourceRDD` closes the reader once the task 
completes. The library must
+ * not get the release of the stream during get_next, and the task context 
must stay valid until
+ * then, so [[close]] then leaves the release to [[next]], once get_next 
returns, rather than wait
+ * for it and keep the task from completing. That [[next]] then returns false, 
or throws the error
+ * of its read. A [[next]] called once the reader is closed returns false 
without calling the
+ * library.
  */
 class NativePartitionReader(
     location: NativeLibraryLocation,
@@ -487,16 +497,26 @@ class NativePartitionReader(
   private var task: NativeLibrary.Task = _
   private var root: VectorSchemaRoot = _
   private var batch: ColumnarBatch = _
-
-  try {
-    openStream()
-    root = VectorSchemaRoot.create(importSchema(), allocator)
-    checkSchema()
-    batch = new 
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
-  } catch {
-    case e: Throwable =>
-      close()
-      throw e
+  @GuardedBy("this")
+  private var closed = false
+  // Whether a thread is in next. While it is, only that thread uses the 
stream and the vectors,
+  // and it releases them if the reader was closed meanwhile. Otherwise, they 
are only released
+  // under the lock.
+  @GuardedBy("this")
+  private var reading = false
+
+  // Synchronized so that a close on another thread sees the task and the 
vectors.
+  synchronized {
+    try {
+      openStream()
+      root = VectorSchemaRoot.create(importSchema(), allocator)
+      checkSchema()
+      batch = new 
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
+    } catch {
+      case e: Throwable =>
+        close()
+        throw e
+    }
   }
 
   private def openStream(): Unit = {
@@ -667,6 +687,24 @@ class NativePartitionReader(
   }
 
   override def next(): Boolean = {
+    // A reader that was closed while it read has released the batch that it 
read.
+    startReading() && Utils.tryWithSafeFinally(readNext())(stopReading()) && 
synchronized(!closed)
+  }
+
+  private def startReading(): Boolean = synchronized {
+    reading = !closed
+    reading
+  }
+
+  private def stopReading(): Unit = synchronized {
+    reading = false
+    if (closed) {
+      // The reader was closed by another thread, which no longer gets a 
failure from here.
+      Utils.tryLogNonFatalError(closeResources())
+    }
+  }
+
+  private def readNext(): Boolean = {
     val array = ArrowArray.allocateNew(allocator)
     try {
       call(name, "read")(stream.getNext(array))
@@ -817,7 +855,16 @@ class NativePartitionReader(
 
   override def get(): ColumnarBatch = batch
 
-  override def close(): Unit = {
+  override def close(): Unit = synchronized {
+    if (!closed) {
+      closed = true
+      if (!reading) {
+        closeResources()
+      }
+    }
+  }
+
+  private def closeResources(): Unit = {
     Utils.tryWithSafeFinally {
       Utils.tryWithSafeFinally {
         if (root != null) {
diff --git 
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc 
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index cb9eba8c56e4..80b2e249381d 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -37,7 +37,14 @@
 //                   which has no validity buffer, a null_count of -1 instead. 
With the option
 //                   "wait_for_interrupt", the stream of a partition creates 
the file
 //                   <wait_for_interrupt>.waiting, waits until its task is 
killed, and then creates
-//                   the file <wait_for_interrupt> and fails.
+//                   the file <wait_for_interrupt> and fails. With the option 
"slow_read", the
+//                   get_next that creates the file <slow_read>.reading, the 
first one, then waits
+//                   until the file <slow_read>.continue exists, for at most a 
minute; with the
+//                   option "fail" set to "slow_read", each get_next then 
fails. After the wait,
+//                   it creates the file <slow_read>.interrupted with what 
is_interrupted returns
+//                   for the task context, "true" for one that Spark freed 
meanwhile. Releasing
+//                   the stream creates the file <slow_read>, which says 
whether Spark released it
+//                   "during get_next", which it must not, or "after get_next" 
otherwise.
 //   native_sink     Writes the rows as CSV files under the directory in the 
option "path". With
 //                   the option "commit_messages", a data writer passes its 
commit message that
 //                   many times. Creating and closing a data writer create a 
file named after
@@ -76,6 +83,7 @@
 #include "spark_datasource.h"
 
 #include <algorithm>
+#include <atomic>
 #include <cctype>
 #include <cerrno>
 #include <chrono>
@@ -765,6 +773,12 @@ struct Stream {
   const SparkDsTaskContext* task;
   std::string wait_for_interrupt;
   std::string last_error;
+  std::string slow_read;
+  bool fail_slow_read;
+  // task->is_interrupted, kept to call it without reading the task context, 
which a regression
+  // of Spark could free during get_next.
+  bool (*is_interrupted)(const SparkDsTaskContext*);
+  std::atomic<bool> in_get_next{false};
 };
 
 int streamGetSchema(ArrowArrayStream* stream, ArrowSchema* out) {
@@ -773,6 +787,19 @@ int streamGetSchema(ArrowArrayStream* stream, ArrowSchema* 
out) {
   return 0;
 }
 
+// Exports the next batch of the stream, or its end.
+int nextBatch(Stream* state, ArrowArray* out) {
+  if (state->next >= state->end) {
+    out->release = nullptr;  // The end of the stream.
+    return 0;
+  }
+  int64_t hi = std::min(state->end, state->next + state->batch_size);
+  exportBatch(state->columns, state->next, hi, state->slice, 
state->short_array, out);
+  setBatchNulls(state->null_rows, out);
+  state->next = hi;
+  return 0;
+}
+
 int streamGetNext(ArrowArrayStream* stream, ArrowArray* out) {
   auto* state = static_cast<Stream*>(stream->private_data);
   if (state->fail) {
@@ -790,15 +817,37 @@ int streamGetNext(ArrowArrayStream* stream, ArrowArray* 
out) {
     state->last_error = "the task was killed";
     return ECANCELED;
   }
-  if (state->next >= state->end) {
-    out->release = nullptr;  // The end of the stream.
-    return 0;
+  if (!state->slow_read.empty()) {
+    // The flag covers the slow read, for streamRelease to tell whether it 
comes during get_next.
+    state->in_get_next = true;
+    int result = EIO;
+    try {
+      std::string reading = state->slow_read + ".reading";
+      struct stat info;
+      if (stat(reading.c_str(), &info) != 0) {
+        writeFile(reading, "");
+        std::string go = state->slow_read + ".continue";
+        for (int i = 0; i < 6000 && stat(go.c_str(), &info) != 0; i++) {
+          std::this_thread::sleep_for(std::chrono::milliseconds(10));
+        }
+        // The task context must still be valid, as a library that checks it 
here expects. Spark
+        // only looks its address up, and answers true for a context that it 
freed, unless a new
+        // one got the same address, which no other read in these tests 
creates meanwhile.
+        bool interrupted = state->is_interrupted(state->task);
+        writeFile(state->slow_read + ".interrupted", interrupted ? "true" : 
"false");
+      }
+      if (state->fail_slow_read) {
+        state->last_error = "injected failure after a slow get_next";
+      } else {
+        result = nextBatch(state, out);
+      }
+    } catch (...) {
+      state->last_error = "unexpected failure in get_next";
+    }
+    state->in_get_next = false;
+    return result;
   }
-  int64_t hi = std::min(state->end, state->next + state->batch_size);
-  exportBatch(state->columns, state->next, hi, state->slice, 
state->short_array, out);
-  setBatchNulls(state->null_rows, out);
-  state->next = hi;
-  return 0;
+  return nextBatch(state, out);
 }
 
 const char* streamGetLastError(ArrowArrayStream* stream) {
@@ -806,8 +855,19 @@ const char* streamGetLastError(ArrowArrayStream* stream) {
 }
 
 void streamRelease(ArrowArrayStream* stream) {
-  delete static_cast<Stream*>(stream->private_data);
+  auto* state = static_cast<Stream*>(stream->private_data);
   stream->release = nullptr;
+  if (!state->slow_read.empty()) {
+    bool during = state->in_get_next;
+    try {
+      writeFile(state->slow_read, during ? "during get_next" : "after 
get_next");
+    } catch (...) {
+      // release cannot fail: the test then misses the file.
+    }
+    // Leaks the state that a running get_next still uses, rather than crash 
the test.
+    if (during) return;
+  }
+  delete state;
 }
 
 // The driver-side writer of native_sink.
@@ -963,7 +1023,8 @@ int createReader(void* handle, struct ArrowSchema* schema, 
void** out) {
     reader->state["partitions"] = get(source->options, "partitions", "2");
     reader->state["batch_size"] = get(source->options, "batch_size", "1000");
     for (const char* option :
-         {"fail", "slice", "short", "null_rows", "wait_for_interrupt", 
"task_path"}) {
+         {"fail", "slice", "short", "null_rows", "wait_for_interrupt", 
"task_path",
+          "slow_read"}) {
       reader->state[option] = get(source->options, option, "");
     }
     *out = reader;
@@ -1118,7 +1179,9 @@ int readPartition(const SparkDsTaskContext* task, const 
uint8_t* reader_state,
                                       get(state, "fail", "") == "read",
                                       get(state, "slice", "") == "true", 
get(state, "short", ""),
                                       get(state, "null_rows", ""), task,
-                                      get(state, "wait_for_interrupt", ""), 
""};
+                                      get(state, "wait_for_interrupt", ""), "",
+                                      get(state, "slow_read", ""),
+                                      get(state, "fail", "") == "slow_read", 
task->is_interrupted};
   });
 }
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 0663a2f3213f..86d8c0376df3 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -23,7 +23,7 @@ import java.nio.file.StandardCopyOption.REPLACE_EXISTING
 import java.sql.{Date, Timestamp}
 import java.time.{Instant, LocalDate}
 import java.util.UUID
-import java.util.concurrent.{Callable, CountDownLatch, TimeUnit}
+import java.util.concurrent.{Callable, CountDownLatch, ExecutionException, 
TimeUnit}
 import java.util.concurrent.atomic.AtomicReference
 
 import scala.collection.mutable.ArrayBuffer
@@ -31,7 +31,7 @@ import scala.jdk.CollectionConverters._
 
 import org.scalatest.time.SpanSugar._
 
-import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException}
+import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TestUtils}
 import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
 import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, 
QueryTest, Row}
@@ -1174,6 +1174,88 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     }
   }
 
+  nativeTest("release a stream only once get_next returns") {
+    assume(TestUtils.testCommandAvailable("cat"))
+    val session = newSession(defaultPackage)
+    val marker = new File(Utils.createTempDir(), "released")
+    val reading = marker.getPath + ".reading"
+    val df = range(session, "end" -> "10", "partitions" -> "1", "slow_read" -> 
marker.getPath)
+    try {
+      // PipedRDD reads the scan on a thread of its own, which the task does 
not wait for. The
+      // task completes without reading the output of the command, once that 
thread is in
+      // get_next, which waits for the file .continue, and closes the reader 
then. The job runs
+      // with the artifacts of the session, as in a SQL execution: see 
checkRows.
+      val count = session.artifactManager.withResources {
+        df.rdd.pipe("cat").mapPartitions { _ =>
+          val deadline = System.nanoTime() + TimeUnit.MINUTES.toNanos(1)
+          while (!new File(reading).isFile && System.nanoTime() < deadline) {
+            Thread.sleep(10)
+          }
+          Iterator.empty[String]
+        }.count()
+      }
+      assert(count == 0)
+      assert(new File(reading).isFile, "get_next was not called.")
+      if (marker.isFile) fail(s"The stream was released: 
${Files.readString(marker.toPath)}.")
+    } finally {
+      new File(marker.getPath + ".continue").createNewFile()
+    }
+    // The thread in get_next releases the stream once get_next returns.
+    eventually(timeout(1.minute)) {
+      assert(marker.isFile && Files.readString(marker.toPath) == "after 
get_next")
+    }
+    // The task context was still valid once get_next continued: Spark answers 
that a context
+    // that it freed is interrupted, and this task was not.
+    assert(Files.readString(new File(marker.getPath + ".interrupted").toPath) 
== "false")
+  }
+
+  nativeTest("close a reader while another thread reads it") {
+    val session = newSession(defaultPackage)
+    val pool = ThreadUtils.newDaemonSingleThreadExecutor("native-reader")
+    // Closes a reader while another thread is in get_next. The reader 
releases the stream once
+    // get_next returns, also when it fails, and then next returns false or 
throws.
+    def closeWhileReading(failing: Boolean): Either[Throwable, Boolean] = {
+      val marker = new File(Utils.createTempDir(), "released")
+      val options = Seq("end" -> "10", "partitions" -> "1", "slow_read" -> 
marker.getPath) ++
+        (if (failing) Seq("fail" -> "slow_read") else Nil)
+      val batch = scan(range(session, options: _*)).toBatch
+      val reader =
+        
batch.createReaderFactory().createColumnarReader(batch.planInputPartitions().head)
+      try {
+        val read = pool.submit(new Callable[Boolean] {
+          override def call(): Boolean = reader.next()
+        })
+        eventually(timeout(1.minute)) {
+          assert(new File(marker.getPath + ".reading").isFile)
+        }
+        reader.close()
+        if (marker.isFile) fail(s"The stream was released: 
${Files.readString(marker.toPath)}.")
+        new File(marker.getPath + ".continue").createNewFile()
+        val result = try Right(read.get(1, TimeUnit.MINUTES)) catch {
+          case e: ExecutionException => Left(e.getCause)
+        }
+        assert(Files.readString(marker.toPath) == "after get_next")
+        // The task context was still valid once get_next continued, as in the 
test above.
+        assert(Files.readString(new File(marker.getPath + 
".interrupted").toPath) == "false")
+        // A reader that was closed reads nothing more, and closing it again 
does not fail.
+        assert(!reader.next())
+        reader.close()
+        result
+      } finally {
+        // Lets get_next continue, and does not hide a failure of the test if 
closing fails.
+        new File(marker.getPath + ".continue").createNewFile()
+        Utils.tryLogNonFatalError(reader.close())
+      }
+    }
+    try {
+      assert(closeWhileReading(failing = false) == Right(false))
+      val error = closeWhileReading(failing = true).swap.getOrElse(fail("next 
did not fail"))
+      assert(error.getMessage.contains("injected failure after a slow 
get_next"))
+    } finally {
+      pool.shutdown()
+    }
+  }
+
   nativeTest("load libraries that export the same functions side by side") {
     val session = newSession(
       defaultPackage, createPackage(negatingLibrary, dataSources("b_")))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to