This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 202252606c6a [SPARK-60705][SQL] Do not release the stream of a native
data source while get_next runs
202252606c6a is described below
commit 202252606c6acac655fea17560028c4ad14c271d
Author: Kousuke Saruta <[email protected]>
AuthorDate: Fri Oct 30 20:36:45 2026 +0900
[SPARK-60705][SQL] Do not release the stream of a native data source while
get_next runs
### What changes were proposed in this pull request?
This PR makes `NativePartitionReader`, the reader of a partition of a
native data source (SPARK-60101), release the Arrow stream of the library only
once `get_next` returns, also when another thread closes the reader while
`get_next` runs.
- `close()` no longer releases the stream, the vectors, the task context
and the allocator while a thread is in `next()`. It marks the reader as closed,
and that `next()` releases them once it is done with the batch that `get_next`
returned: it then returns false, or throws the error of the read. A failure to
release them there is logged, since the thread that closed the reader no longer
gets it. `close()` does not wait, so it does not keep the task from completing.
- `next()` returns false once the reader is closed, so the reader never
calls a released stream, and `close()` does nothing when it is called again.
The reading thread may then close the reader again: `DataSourceRDD` closes a
reader whose `next()` returns false, as well as from the completion listener.
Before, a second `close()` failed with `NullPointerException: ArrowArrayStream
is already closed`.
- The constructor runs under the same lock, so that a `close()` on another
thread sees the task and the vectors that it created.
The test library gets an option `slow_read`: the first `get_next` waits
until the test lets it continue, then records what `is_interrupted` returns for
the task context, and releasing the stream records whether Spark released it
during `get_next` or not. With the option `fail` set to `slow_read`, `get_next`
then fails.
### Why are the changes needed?
`spark_datasource.h` promises a library that the callbacks of the stream
never run concurrently, and that the task context passed to `read` stays valid
until the stream is released. When a task is killed, it also says that Spark
only releases the stream once `get_next` returns.
Spark did not keep these promises when another thread reads a partition.
`DataSourceRDD` closes its readers from a task completion listener, and some
operators read their input on a thread of their own that the task does not wait
for. For example, `PipedRDD` reads its parent on its stdin writer thread. On
SPARK-60101 as merged (99a1bf9a837, the same reader as its last PR commit
1f4048404be), with a library whose `get_next` takes 2 seconds,
`spark.read.format(...).load().rdd.pipe("true [...]
`close()` hands the release to the thread in `next()` rather than waiting
for it, since that thread can be in a `get_next` that waits for data:
`task->is_interrupted` stays false for a task that completed, so waiting could
keep the task from completing for as long as `get_next` blocks. Such a
`get_next` still runs until it returns on its own, and keeps its thread and the
resources of the read until then; SPARK-60733 is to tell the library to stop.
Besides when the library gets the release, a closed reader now reads
nothing more and can be closed again. As before, a batch that `next()` returned
is only valid until the reader is closed, so a thread that still uses it after
the task completed reads closed vectors.
### Does this PR introduce _any_ user-facing change?
No. Native data sources are not released yet (4.4.0).
### How was this patch tested?
New tests in `NativeDataSourceSuite`, which hold the library in its first
`get_next` until they let it continue, so that they do not depend on timing:
- `release a stream only once get_next returns` reads a native data source
through `rdd.pipe("cat")`, and completes the task while the stdin writer thread
of `PipedRDD` is in `get_next`. It checks that the stream is not released when
the task completes, that the task context is still valid when `get_next`
continues, and that the library gets the release after `get_next` returns.
- `close a reader while another thread reads it` closes a reader while
another thread is in `next()`, and checks the same, that `next()` then returns
false, or throws when `get_next` fails, and that a closed reader reads nothing
more and can be closed again.
The task context is checked through `is_interrupted`, which Spark answers
with true for a context that it freed. Without the change in
`NativePartitionReader`, both fail with `The stream was released: during
get_next.` Each of the other parts of the change, except running the
constructor under the lock and logging a failure to release the resources in
`next()`, also makes a test fail when it is reverted.
`NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass locally on Java
22 on macOS, except two tests that are canceled without pandas, and the new
tests passed in 5 runs, and in 5 more after the check of the task context was
added.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI
Closes #59914 from sarutak/native-ds-reader-close-race.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
(cherry picked from commit 69743391a88df5220bd8e981236dfe27e8f65779)
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../datasources/v2/ffi/NativeDataSource.scala | 69 ++++++++++++++---
.../native-datasource/test_native_datasource.cc | 87 +++++++++++++++++++---
.../datasources/v2/ffi/NativeDataSourceSuite.scala | 86 ++++++++++++++++++++-
3 files changed, 217 insertions(+), 25 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 9f5c6adec3b9..8c68b24602f7 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -20,6 +20,7 @@ import java.lang.ref.{Cleaner, WeakReference}
import java.util
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicLong
+import javax.annotation.concurrent.GuardedBy
import scala.jdk.CollectionConverters._
import scala.util.control.NonFatal
@@ -468,6 +469,15 @@ object NativeWriterCommitMessage {
/**
* Reads a partition of a native data source: each array of the Arrow stream
exported by the
* library is checked, and imported into the Arrow vectors that back the
returned batch.
+ *
+ * A reader can be closed while another thread is in [[next]]: some operators
read their input on
+ * a thread of their own that the task does not wait for, such as the stdin
writer thread of
+ * `PipedRDD`, while `DataSourceRDD` closes the reader once the task
completes. The library must
+ * not get the release of the stream during get_next, and the task context
must stay valid until
+ * then, so [[close]] then leaves the release to [[next]], once get_next
returns, rather than wait
+ * for it and keep the task from completing. That [[next]] then returns false,
or throws the error
+ * of its read. A [[next]] called once the reader is closed returns false
without calling the
+ * library.
*/
class NativePartitionReader(
location: NativeLibraryLocation,
@@ -487,16 +497,26 @@ class NativePartitionReader(
private var task: NativeLibrary.Task = _
private var root: VectorSchemaRoot = _
private var batch: ColumnarBatch = _
-
- try {
- openStream()
- root = VectorSchemaRoot.create(importSchema(), allocator)
- checkSchema()
- batch = new
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
- } catch {
- case e: Throwable =>
- close()
- throw e
+ @GuardedBy("this")
+ private var closed = false
+ // Whether a thread is in next. While it is, only that thread uses the
stream and the vectors,
+ // and it releases them if the reader was closed meanwhile. Otherwise, they
are only released
+ // under the lock.
+ @GuardedBy("this")
+ private var reading = false
+
+ // Synchronized so that a close on another thread sees the task and the
vectors.
+ synchronized {
+ try {
+ openStream()
+ root = VectorSchemaRoot.create(importSchema(), allocator)
+ checkSchema()
+ batch = new
ColumnarBatch(root.getFieldVectors.asScala.map(columnVector).toArray)
+ } catch {
+ case e: Throwable =>
+ close()
+ throw e
+ }
}
private def openStream(): Unit = {
@@ -667,6 +687,24 @@ class NativePartitionReader(
}
override def next(): Boolean = {
+ // A reader that was closed while it read has released the batch that it
read.
+ startReading() && Utils.tryWithSafeFinally(readNext())(stopReading()) &&
synchronized(!closed)
+ }
+
+ private def startReading(): Boolean = synchronized {
+ reading = !closed
+ reading
+ }
+
+ private def stopReading(): Unit = synchronized {
+ reading = false
+ if (closed) {
+ // The reader was closed by another thread, which no longer gets a
failure from here.
+ Utils.tryLogNonFatalError(closeResources())
+ }
+ }
+
+ private def readNext(): Boolean = {
val array = ArrowArray.allocateNew(allocator)
try {
call(name, "read")(stream.getNext(array))
@@ -817,7 +855,16 @@ class NativePartitionReader(
override def get(): ColumnarBatch = batch
- override def close(): Unit = {
+ override def close(): Unit = synchronized {
+ if (!closed) {
+ closed = true
+ if (!reading) {
+ closeResources()
+ }
+ }
+ }
+
+ private def closeResources(): Unit = {
Utils.tryWithSafeFinally {
Utils.tryWithSafeFinally {
if (root != null) {
diff --git
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index cb9eba8c56e4..80b2e249381d 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -37,7 +37,14 @@
// which has no validity buffer, a null_count of -1 instead.
With the option
// "wait_for_interrupt", the stream of a partition creates
the file
// <wait_for_interrupt>.waiting, waits until its task is
killed, and then creates
-// the file <wait_for_interrupt> and fails.
+// the file <wait_for_interrupt> and fails. With the option
"slow_read", the
+// get_next that creates the file <slow_read>.reading, the
first one, then waits
+// until the file <slow_read>.continue exists, for at most a
minute; with the
+// option "fail" set to "slow_read", each get_next then
fails. After the wait,
+// it creates the file <slow_read>.interrupted with what
is_interrupted returns
+// for the task context, "true" for one that Spark freed
meanwhile. Releasing
+// the stream creates the file <slow_read>, which says
whether Spark released it
+// "during get_next", which it must not, or "after get_next"
otherwise.
// native_sink Writes the rows as CSV files under the directory in the
option "path". With
// the option "commit_messages", a data writer passes its
commit message that
// many times. Creating and closing a data writer create a
file named after
@@ -76,6 +83,7 @@
#include "spark_datasource.h"
#include <algorithm>
+#include <atomic>
#include <cctype>
#include <cerrno>
#include <chrono>
@@ -765,6 +773,12 @@ struct Stream {
const SparkDsTaskContext* task;
std::string wait_for_interrupt;
std::string last_error;
+ std::string slow_read;
+ bool fail_slow_read;
+ // task->is_interrupted, kept to call it without reading the task context,
which a regression
+ // of Spark could free during get_next.
+ bool (*is_interrupted)(const SparkDsTaskContext*);
+ std::atomic<bool> in_get_next{false};
};
int streamGetSchema(ArrowArrayStream* stream, ArrowSchema* out) {
@@ -773,6 +787,19 @@ int streamGetSchema(ArrowArrayStream* stream, ArrowSchema*
out) {
return 0;
}
+// Exports the next batch of the stream, or its end.
+int nextBatch(Stream* state, ArrowArray* out) {
+ if (state->next >= state->end) {
+ out->release = nullptr; // The end of the stream.
+ return 0;
+ }
+ int64_t hi = std::min(state->end, state->next + state->batch_size);
+ exportBatch(state->columns, state->next, hi, state->slice,
state->short_array, out);
+ setBatchNulls(state->null_rows, out);
+ state->next = hi;
+ return 0;
+}
+
int streamGetNext(ArrowArrayStream* stream, ArrowArray* out) {
auto* state = static_cast<Stream*>(stream->private_data);
if (state->fail) {
@@ -790,15 +817,37 @@ int streamGetNext(ArrowArrayStream* stream, ArrowArray*
out) {
state->last_error = "the task was killed";
return ECANCELED;
}
- if (state->next >= state->end) {
- out->release = nullptr; // The end of the stream.
- return 0;
+ if (!state->slow_read.empty()) {
+ // The flag covers the slow read, for streamRelease to tell whether it
comes during get_next.
+ state->in_get_next = true;
+ int result = EIO;
+ try {
+ std::string reading = state->slow_read + ".reading";
+ struct stat info;
+ if (stat(reading.c_str(), &info) != 0) {
+ writeFile(reading, "");
+ std::string go = state->slow_read + ".continue";
+ for (int i = 0; i < 6000 && stat(go.c_str(), &info) != 0; i++) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(10));
+ }
+ // The task context must still be valid, as a library that checks it
here expects. Spark
+ // only looks its address up, and answers true for a context that it
freed, unless a new
+ // one got the same address, which no other read in these tests
creates meanwhile.
+ bool interrupted = state->is_interrupted(state->task);
+ writeFile(state->slow_read + ".interrupted", interrupted ? "true" :
"false");
+ }
+ if (state->fail_slow_read) {
+ state->last_error = "injected failure after a slow get_next";
+ } else {
+ result = nextBatch(state, out);
+ }
+ } catch (...) {
+ state->last_error = "unexpected failure in get_next";
+ }
+ state->in_get_next = false;
+ return result;
}
- int64_t hi = std::min(state->end, state->next + state->batch_size);
- exportBatch(state->columns, state->next, hi, state->slice,
state->short_array, out);
- setBatchNulls(state->null_rows, out);
- state->next = hi;
- return 0;
+ return nextBatch(state, out);
}
const char* streamGetLastError(ArrowArrayStream* stream) {
@@ -806,8 +855,19 @@ const char* streamGetLastError(ArrowArrayStream* stream) {
}
void streamRelease(ArrowArrayStream* stream) {
- delete static_cast<Stream*>(stream->private_data);
+ auto* state = static_cast<Stream*>(stream->private_data);
stream->release = nullptr;
+ if (!state->slow_read.empty()) {
+ bool during = state->in_get_next;
+ try {
+ writeFile(state->slow_read, during ? "during get_next" : "after
get_next");
+ } catch (...) {
+ // release cannot fail: the test then misses the file.
+ }
+ // Leaks the state that a running get_next still uses, rather than crash
the test.
+ if (during) return;
+ }
+ delete state;
}
// The driver-side writer of native_sink.
@@ -963,7 +1023,8 @@ int createReader(void* handle, struct ArrowSchema* schema,
void** out) {
reader->state["partitions"] = get(source->options, "partitions", "2");
reader->state["batch_size"] = get(source->options, "batch_size", "1000");
for (const char* option :
- {"fail", "slice", "short", "null_rows", "wait_for_interrupt",
"task_path"}) {
+ {"fail", "slice", "short", "null_rows", "wait_for_interrupt",
"task_path",
+ "slow_read"}) {
reader->state[option] = get(source->options, option, "");
}
*out = reader;
@@ -1118,7 +1179,9 @@ int readPartition(const SparkDsTaskContext* task, const
uint8_t* reader_state,
get(state, "fail", "") == "read",
get(state, "slice", "") == "true",
get(state, "short", ""),
get(state, "null_rows", ""), task,
- get(state, "wait_for_interrupt", ""),
""};
+ get(state, "wait_for_interrupt", ""), "",
+ get(state, "slow_read", ""),
+ get(state, "fail", "") == "slow_read",
task->is_interrupted};
});
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 0663a2f3213f..86d8c0376df3 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -23,7 +23,7 @@ import java.nio.file.StandardCopyOption.REPLACE_EXISTING
import java.sql.{Date, Timestamp}
import java.time.{Instant, LocalDate}
import java.util.UUID
-import java.util.concurrent.{Callable, CountDownLatch, TimeUnit}
+import java.util.concurrent.{Callable, CountDownLatch, ExecutionException,
TimeUnit}
import java.util.concurrent.atomic.AtomicReference
import scala.collection.mutable.ArrayBuffer
@@ -31,7 +31,7 @@ import scala.jdk.CollectionConverters._
import org.scalatest.time.SpanSugar._
-import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException,
SparkFiles, SparkRuntimeException, SparkThrowable,
SparkUnsupportedOperationException}
+import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException,
SparkFiles, SparkRuntimeException, SparkThrowable,
SparkUnsupportedOperationException, TestUtils}
import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils,
QueryTest, Row}
@@ -1174,6 +1174,88 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
}
}
+ nativeTest("release a stream only once get_next returns") {
+ assume(TestUtils.testCommandAvailable("cat"))
+ val session = newSession(defaultPackage)
+ val marker = new File(Utils.createTempDir(), "released")
+ val reading = marker.getPath + ".reading"
+ val df = range(session, "end" -> "10", "partitions" -> "1", "slow_read" ->
marker.getPath)
+ try {
+ // PipedRDD reads the scan on a thread of its own, which the task does
not wait for. The
+ // task completes without reading the output of the command, once that
thread is in
+ // get_next, which waits for the file .continue, and closes the reader
then. The job runs
+ // with the artifacts of the session, as in a SQL execution: see
checkRows.
+ val count = session.artifactManager.withResources {
+ df.rdd.pipe("cat").mapPartitions { _ =>
+ val deadline = System.nanoTime() + TimeUnit.MINUTES.toNanos(1)
+ while (!new File(reading).isFile && System.nanoTime() < deadline) {
+ Thread.sleep(10)
+ }
+ Iterator.empty[String]
+ }.count()
+ }
+ assert(count == 0)
+ assert(new File(reading).isFile, "get_next was not called.")
+ if (marker.isFile) fail(s"The stream was released:
${Files.readString(marker.toPath)}.")
+ } finally {
+ new File(marker.getPath + ".continue").createNewFile()
+ }
+ // The thread in get_next releases the stream once get_next returns.
+ eventually(timeout(1.minute)) {
+ assert(marker.isFile && Files.readString(marker.toPath) == "after
get_next")
+ }
+ // The task context was still valid once get_next continued: Spark answers
that a context
+ // that it freed is interrupted, and this task was not.
+ assert(Files.readString(new File(marker.getPath + ".interrupted").toPath)
== "false")
+ }
+
+ nativeTest("close a reader while another thread reads it") {
+ val session = newSession(defaultPackage)
+ val pool = ThreadUtils.newDaemonSingleThreadExecutor("native-reader")
+ // Closes a reader while another thread is in get_next. The reader
releases the stream once
+ // get_next returns, also when it fails, and then next returns false or
throws.
+ def closeWhileReading(failing: Boolean): Either[Throwable, Boolean] = {
+ val marker = new File(Utils.createTempDir(), "released")
+ val options = Seq("end" -> "10", "partitions" -> "1", "slow_read" ->
marker.getPath) ++
+ (if (failing) Seq("fail" -> "slow_read") else Nil)
+ val batch = scan(range(session, options: _*)).toBatch
+ val reader =
+
batch.createReaderFactory().createColumnarReader(batch.planInputPartitions().head)
+ try {
+ val read = pool.submit(new Callable[Boolean] {
+ override def call(): Boolean = reader.next()
+ })
+ eventually(timeout(1.minute)) {
+ assert(new File(marker.getPath + ".reading").isFile)
+ }
+ reader.close()
+ if (marker.isFile) fail(s"The stream was released:
${Files.readString(marker.toPath)}.")
+ new File(marker.getPath + ".continue").createNewFile()
+ val result = try Right(read.get(1, TimeUnit.MINUTES)) catch {
+ case e: ExecutionException => Left(e.getCause)
+ }
+ assert(Files.readString(marker.toPath) == "after get_next")
+ // The task context was still valid once get_next continued, as in the
test above.
+ assert(Files.readString(new File(marker.getPath +
".interrupted").toPath) == "false")
+ // A reader that was closed reads nothing more, and closing it again
does not fail.
+ assert(!reader.next())
+ reader.close()
+ result
+ } finally {
+ // Lets get_next continue, and does not hide a failure of the test if
closing fails.
+ new File(marker.getPath + ".continue").createNewFile()
+ Utils.tryLogNonFatalError(reader.close())
+ }
+ }
+ try {
+ assert(closeWhileReading(failing = false) == Right(false))
+ val error = closeWhileReading(failing = true).swap.getOrElse(fail("next
did not fail"))
+ assert(error.getMessage.contains("injected failure after a slow
get_next"))
+ } finally {
+ pool.shutdown()
+ }
+ }
+
nativeTest("load libraries that export the same functions side by side") {
val session = newSession(
defaultPackage, createPackage(negatingLibrary, dataSources("b_")))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]