This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8410fcffa919 [SPARK-60724][SQL] Document the order of the calls of 
native data sources and add is_log_enabled
8410fcffa919 is described below

commit 8410fcffa919b682c66bf4886858ceb9134a0c5a
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:27:06 2026 +0900

    [SPARK-60724][SQL] Document the order of the calls of native data sources 
and add is_log_enabled
    
    ### What changes were proposed in this pull request?
    
    Two follow-ups to SPARK-60101 (#59424) for `spark_datasource.h`, from 
gengliangwang's review in 
https://github.com/apache/spark/pull/59424#issuecomment-6480975990. This PR is 
based on #59931 (version 1 of the interface), whose commit it includes until 
that PR is merged.
    
    1. **The order of the calls on a handle**, as a convention of the header:
       - After a function on a handle fails, Spark calls no other function on 
the handle than its close function. The exceptions: it aborts a writer whose 
commit failed, and a data writer whose `write` or commit failed.
       - Spark does not abort a writer or a data writer whose commit succeeded.
       - Spark calls each function of a batch reader at most once.
       - Spark calls commit and abort at most once each for a writer, and 
`commit_data_writer` and `abort_data_writer` at most once each for a data 
writer, after all the calls to `write`.
       - The data writers of the task attempts that Spark kills, such as 
speculative ones, may still be written to, aborted and closed after Spark 
committed or aborted their writer and closed it. The guide also says that a 
data writer whose task failed after `commit_data_writer` succeeded is neither 
aborted nor passed to `commit`, so `commit` or `abort` cleans up the output of 
the data writers whose messages it does not get.
    
       This documents what Spark does, so that SDKs do not need to guard 
against other orders. The adapter keeps these rules whatever its callers do:
       - `ColumnarBatchWrite` and `ColumnarStreamingWrite` do not abort a write 
or a micro-batch whose commit succeeded.
       - `ColumnarScanBuilder` prunes the columns once. Pruning again with 
columns that the scan does not read is an internal error. A scan keeps the 
failure of its pruning too, so that the planner is not asked again.
       - The data writer of a native data source fails a `write` or a commit 
after a failed one, instead of calling the library.
    2. **`SparkDsHost.is_log_enabled`**, appended to the struct, tells whether 
Spark logs a level. A library can skip formatting the messages that Spark would 
not write, and follow the log level of Spark, for example to log at debug only 
when Spark does. `NativeLibrary` implements it with an upcall to the logger of 
the native data sources. The docs mention it.
    
    The third item of that review, column pruning for streaming reads, is in 
#59931, which defines the schema of `create_stream_reader` as the columns to 
read, and SPARK-60734 (#59943), which prunes the columns of streaming scans.
    
    ### Why are the changes needed?
    
    SDKs such as the Rust prototype in #59520 have to know which call orders 
they can rely on. Without `is_log_enabled`, a library formats every message, 
even those that Spark drops, and cannot follow the log level of Spark.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for released versions: native data sources are new in the unreleased 
4.4.0. `is_log_enabled` is appended to `SparkDsHost`, so it does not change 
version 1 for libraries that do not use it.
    
    ### How was this patch tested?
    
    "pass the version of Spark, and log with the log of Spark" in 
`NativeDataSourceSuite` now also checks `is_log_enabled`. The test library 
writes whether Spark logs each level, which the test checks with the log of 
`NativeLibrary` at each level from TRACE to ERROR. The test library also skips 
the levels that Spark does not log.
    
    The test library records each call that the header rules out. 
`checkingOrderOfCalls` in `NativeDataSourceTestUtils` checks the record at the 
end of every test of `NativeDataSourceSuite`, `NativeDataSourceClusterSuite` 
and `NativeDataSourceConnectSuite`, once no task runs, so a violation fails the 
test that made it.
    
    New tests:
    - `NativeDataSourceSuite`:
      - "keep the order of the calls whatever the callers of the adapter do": a 
committed write or micro-batch is not aborted, and a data writer whose write 
failed only aborts and closes the data writer of the library.
      - "call no other function on a reader whose call failed than its close 
function": failures injected into `push_limit`, `prune_columns`, 
`initial_offset`, `latest_offset`, `stream_partitions` and `commit_offset`, 
after which Spark closes the stream reader.
    - `ColumnarDataSourceSuite`, which runs on any Java version, while the 
native tests need Java 22 or later and a C++ compiler:
      - "prune the columns of a scan once", also when the pruning fails.
      - "do not abort a write whose commit succeeded", for batch and streaming 
writes.
      - "abort a data writer whose commit the coordinator denies, without 
committing it".
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59932 from HyukjinKwon/native-ds-header-rules.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 docs/sql-native-data-sources.md                    |  27 ++-
 .../sql/connect/NativeDataSourceConnectSuite.scala |   2 +-
 .../datasources/v2/ffi/NativeLibrary.java          |  63 +++++-
 .../apache/spark/sql/datasource/spark_datasource.h |  57 ++++--
 .../v2/columnar/ColumnarDataSource.scala           |  10 +-
 .../datasources/v2/columnar/ColumnarTable.scala    |  15 +-
 .../datasources/v2/columnar/ColumnarWrite.scala    |  20 +-
 .../datasources/v2/ffi/NativeDataSource.scala      |  26 ++-
 .../native-datasource/test_native_datasource.cc    | 214 ++++++++++++++++-----
 .../v2/columnar/ColumnarDataSourceSuite.scala      |  95 ++++++++-
 .../v2/ffi/NativeDataSourceClusterSuite.scala      |   3 +-
 .../datasources/v2/ffi/NativeDataSourceSuite.scala | 157 ++++++++++++++-
 .../v2/ffi/NativeDataSourceTestUtils.scala         |  55 +++++-
 13 files changed, 634 insertions(+), 110 deletions(-)

diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index ace2caaa7782..126882e81435 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -70,7 +70,9 @@ A native data source library exports a single C function, 
`spark_ds_init`, which
 of function pointers, a `SparkDsLibrary`, that implements the data source: it 
needs no JVM headers
 or bindings. Each library has its own table, so any number of native data 
sources can be loaded
 side by side. When Spark loads a library, it calls `spark_ds_init` with a 
`SparkDsHost`, which
-gives the library the version of Spark and a function to log messages to the 
log of Spark.
+gives the library the version of Spark, a function to log messages to the log 
of Spark, and a
+function that tells whether Spark logs the messages of a level, so that a 
library follows the log
+level of Spark.
 
 A Spark distribution contains the header in its `include` directory, as
 `$SPARK_HOME/include/spark_datasource.h`, and the `spark-sql` JAR contains it 
as
@@ -130,17 +132,21 @@ Spark calls the functions of the table as follows:
   irreversible work, such as removing the existing data for an overwrite, in 
`commit`. On the
   executors, each task creates a data writer with `create_data_writer`, which 
gets the schema of the
   data and the state of the writer in a `SparkDsDataWriterInfo`, writes the 
batches of its data with
-  `write`, and calls `commit_data_writer`, or `abort_data_writer` if the task 
or the commit fails.
-  Then the driver calls `commit` with the messages of the tasks, or `abort` if 
the write failed.
+  `write`, and calls `commit_data_writer`, or `abort_data_writer` if the task 
or the commit fails
+  before `commit_data_writer` succeeded. Then the driver calls `commit` with 
the messages of the
+  tasks, or `abort` if the write failed. A data writer whose task failed after
+  `commit_data_writer` succeeded is neither aborted nor passed to `commit`, so 
`commit` or `abort`
+  cleans up the output of the data writers whose messages it does not get.
   When Spark rejects what `create_writer` returned, such as a state set more 
than once, it calls
   `abort` with the field `rejected` of `SparkDsFinishWriteInfo` set to 1, and 
an epoch ID of -1 also
   for a streaming write, before `close_writer`. A streaming write commits each 
micro-batch with its
   epoch ID, and the ID of the query stays the same when the query restarts 
from its checkpoint:
   together, they identify the output of a micro-batch, so that a library can 
skip an epoch that it
-  already committed. A micro-batch streaming query creates a new stream writer 
for each micro-batch,
-  and Spark closes the previous one when it creates the next one. Because 
Spark is not told when a
-  streaming query stops, the last stream writer of a query is closed once it 
is no longer
-  referenced, or when the query is restarted.
+  already committed. A restarted query may also abort such an epoch, so an 
abort only removes what
+  the data writers of its own writer wrote. A micro-batch streaming query 
creates a new stream
+  writer for each micro-batch, and Spark closes the previous one when it 
creates the next one.
+  Because Spark is not told when a streaming query stops, the last stream 
writer of a query is
+  closed once it is no longer referenced, or when the query is restarted.
 
 The library defines the format of the partitions, of the state of its readers 
and writers, and of
 the commit messages, which Spark sends to the executors and back as byte 
arrays. The library
@@ -151,7 +157,12 @@ function reports an error by returning a non-zero value, 
with the message return
 `last_error`, and Spark reports it as a `NATIVE_DATA_SOURCE_ERROR`. A C++ 
exception or a Rust panic
 must not cross the interface. Each handle that a `create` function returns is 
released exactly
 once, by the `close` function of its kind. The close functions cannot fail, 
and they can run after
-the query, so do the work that can fail, such as flushing data, in `commit`. 
Strings are UTF-8.
+the query, so do the work that can fail, such as flushing data, in `commit`. 
After a function on
+a handle fails, Spark only closes the handle, except that it aborts a writer 
whose commit failed
+and a data writer whose write or commit failed, and it never aborts one whose 
commit succeeded.
+Spark may read a partition more than once, also at the same time, in several 
attempts of its task,
+such as after a failure or with speculative execution. The header lists the 
order of the calls
+that a library can rely on. Strings are UTF-8.
 
 A library only sets the functions of the operations it supports, and leaves 
the others `NULL`. A
 data source must not declare a capability whose functions are `NULL`:
diff --git 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
index 2b7407340537..e5dc781025e7 100644
--- 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
+++ 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
@@ -40,7 +40,7 @@ class NativeDataSourceConnectSuite extends 
SparkConnectServerTest {
 
   private def nativeTest(name: String)(f: => Unit): Unit = test(name) {
     assume(toolchain.isDefined, "A C++ compiler and Java 22 or later are 
required.")
-    f
+    checkingOrderOfCalls(f)
   }
 
   private def withNewSession(f: SparkSession => Unit): Unit = {
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index 033a0119406e..ad9b59674d4f 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -138,10 +138,11 @@ final class NativeLibrary {
   // struct starts with its size.
   private static final long TABLE_ABI_VERSION = 8;
   private static final long TABLE_FUNCTIONS = 16;
-  private static final long HOST_SIZE = 32;
+  private static final long HOST_SIZE = 40;
   private static final long HOST_ABI_VERSION = 8;
   private static final long HOST_SPARK_VERSION = 16;
   private static final long HOST_LOG = 24;
+  private static final long HOST_IS_LOG_ENABLED = 32;
   private static final long CREATE_INFO_SIZE = 40;
   private static final long CREATE_INFO_NAME = 8;
   private static final long CREATE_INFO_OPTION_KEYS = 16;
@@ -219,6 +220,7 @@ final class NativeLibrary {
     layout.put("SparkDsHost.abi_version", HOST_ABI_VERSION);
     layout.put("SparkDsHost.spark_version", HOST_SPARK_VERSION);
     layout.put("SparkDsHost.log", HOST_LOG);
+    layout.put("SparkDsHost.is_log_enabled", HOST_IS_LOG_ENABLED);
     layout.put("SparkDsCreateInfo", CREATE_INFO_SIZE);
     layout.put("SparkDsCreateInfo.size", 0L);
     layout.put("SparkDsCreateInfo.name", CREATE_INFO_NAME);
@@ -384,11 +386,13 @@ final class NativeLibrary {
     static final MethodHandle LOG;
     // The addresses of the C functions that call Java, which live as long as 
the JVM:
     // SparkDsSink, which calls NativeLibrary.sink, the SparkDsSink set_state 
of the functions
-    // that also return a state, which calls NativeLibrary.stateSink, and
-    // SparkDsTaskContext.is_interrupted, which calls 
NativeLibrary.isInterrupted.
+    // that also return a state, which calls NativeLibrary.stateSink,
+    // SparkDsTaskContext.is_interrupted, which calls 
NativeLibrary.isInterrupted, and
+    // SparkDsHost.is_log_enabled, which calls NativeLibrary.isLogEnabled.
     static final long SINK;
     static final long STATE_SINK;
     static final long IS_INTERRUPTED;
+    static final long IS_LOG_ENABLED;
 
     static {
       try {
@@ -447,6 +451,10 @@ final class NativeLibrary {
           lookup.findStatic(NativeLibrary.class, "isInterrupted",
             MethodType.methodType(boolean.class, long.class)),
           C.BOOL, C.PTR);
+        IS_LOG_ENABLED = upcall(
+          lookup.findStatic(NativeLibrary.class, "isLogEnabled",
+            MethodType.methodType(boolean.class, int.class)),
+          C.BOOL, C.I32);
       } catch (Throwable e) {
         // The message names the cause, because the JVM only keeps this 
message for the later
         // uses of the class.
@@ -528,6 +536,7 @@ final class NativeLibrary {
       Platform.putInt(null, host + HOST_ABI_VERSION, ABI_VERSION);
       Platform.putLong(null, host + HOST_SPARK_VERSION, versionAddress);
       Platform.putLong(null, host + HOST_LOG, log);
+      Platform.putLong(null, host + HOST_IS_LOG_ENABLED, IS_LOG_ENABLED);
       return host;
     }
   }
@@ -565,6 +574,24 @@ final class NativeLibrary {
     }
   }
 
+  /**
+   * The SPARK_DS_LOG_* levels, which {@link #log} and {@link #isLogEnabled} 
share: a level that
+   * is not one of them is treated as SPARK_DS_LOG_INFO.
+   */
+  private enum LogLevel {
+    ERROR, WARN, INFO, DEBUG, TRACE;
+
+    static LogLevel of(int level) {
+      return switch (level) {
+        case 1 -> ERROR;
+        case 2 -> WARN;
+        case 4 -> DEBUG;
+        case 5 -> TRACE;
+        default -> INFO;
+      };
+    }
+  }
+
   /**
    * Called by the library, through {@code SparkDsHost.log}, with the path of 
the library. It must
    * not throw: an exception thrown by an upcall crashes the JVM.
@@ -572,14 +599,14 @@ final class NativeLibrary {
   private static void log(String path, int level, long message) {
     try {
       String text = message == 0 ? "" : new String(readCString(message), 
StandardCharsets.UTF_8);
-      switch (level) {
-        case 1 -> LOGGER.error("Native data source library {}: {}",
+      switch (LogLevel.of(level)) {
+        case ERROR -> LOGGER.error("Native data source library {}: {}",
           MDC.of(LogKeys.PATH, path), MDC.of(LogKeys.MESSAGE, text));
-        case 2 -> LOGGER.warn("Native data source library {}: {}",
+        case WARN -> LOGGER.warn("Native data source library {}: {}",
           MDC.of(LogKeys.PATH, path), MDC.of(LogKeys.MESSAGE, text));
-        case 4 -> LOGGER.debug("Native data source library {}: {}", path, 
text);
-        case 5 -> LOGGER.trace("Native data source library {}: {}", path, 
text);
-        default -> LOGGER.info("Native data source library {}: {}",
+        case DEBUG -> LOGGER.debug("Native data source library {}: {}", path, 
text);
+        case TRACE -> LOGGER.trace("Native data source library {}: {}", path, 
text);
+        case INFO -> LOGGER.info("Native data source library {}: {}",
           MDC.of(LogKeys.PATH, path), MDC.of(LogKeys.MESSAGE, text));
       }
     } catch (Throwable e) {
@@ -587,6 +614,24 @@ final class NativeLibrary {
     }
   }
 
+  /**
+   * Called by the library, through {@code SparkDsHost.is_log_enabled}, with 
the level of the
+   * messages of {@link #log}. It must not throw: an exception thrown by an 
upcall crashes the JVM.
+   */
+  private static boolean isLogEnabled(int level) {
+    try {
+      return switch (LogLevel.of(level)) {
+        case ERROR -> LOGGER.isErrorEnabled();
+        case WARN -> LOGGER.isWarnEnabled();
+        case INFO -> LOGGER.isInfoEnabled();
+        case DEBUG -> LOGGER.isDebugEnabled();
+        case TRACE -> LOGGER.isTraceEnabled();
+      };
+    } catch (Throwable e) {
+      return false;
+    }
+  }
+
   // 
---------------------------------------------------------------------------------------------
   // Calls
   // 
---------------------------------------------------------------------------------------------
diff --git 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index 3d842b5bf725..a4d98446c066 100644
--- 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++ 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -84,18 +84,23 @@
  *   and the state of the writer, calls write for each batch of at most
  *   spark.sql.execution.arrow.maxRecordsPerBatch rows, which also ends once 
it holds
  *   spark.sql.execution.arrow.maxBytesPerBatch bytes, and calls 
commit_data_writer, or
- *   abort_data_writer if the task or the commit fails, before it closes the 
data writer. Then the
- *   driver calls commit with the messages of the tasks if they all succeeded, 
or abort otherwise,
- *   including when commit fails, and closes the writer. Several attempts of a 
task may write the
- *   same partition, after a failure or with speculative execution, each with 
a data writer of its
- *   own. Spark lets the attempts of a partition commit one at a time, and 
passes to commit the
- *   message of one committed attempt for each partition, so the write 
consists of the data of these
- *   attempts only. It aborts the data writers of the other attempts, except 
on an executor that is
- *   lost, where a data writer is neither committed nor aborted, and may even 
have committed before
- *   its message reached the driver. A streaming write commits or aborts each 
micro-batch (epoch).
- *   Spark may commit the same epoch more than once, for example when a 
streaming query restarts
- *   after a failure, so committing an epoch that is already committed must 
have no effect, for
- *   exactly-once output. The ID of the query, which create_writer gets for a 
streaming write, stays
+ *   abort_data_writer if the task or the commit fails before 
commit_data_writer succeeded, before
+ *   it closes the data writer. Then the driver calls commit with the messages 
of the tasks if they
+ *   all succeeded, or abort otherwise, including when commit fails, and 
closes the writer. Several
+ *   attempts of a task may write the same partition, after a failure or with 
speculative execution,
+ *   each with a data writer of its own. Spark lets the attempts of a 
partition commit one at a
+ *   time, and passes to commit the message of one committed attempt for each 
partition, so the
+ *   write consists of the data of these attempts only. It aborts the data 
writers of the other
+ *   attempts, except two: one whose commit_data_writer succeeded before its 
task failed, which is
+ *   not aborted, and one on an executor that is lost, which is neither 
committed nor aborted, and
+ *   may even have committed before its message reached the driver. Neither is 
passed to commit, so
+ *   the commit or the abort of the writer cleans up the output of the data 
writers whose messages
+ *   it does not get. A streaming write commits or aborts each micro-batch 
(epoch). Spark may commit
+ *   the same epoch more than once, for example when a streaming query 
restarts after a failure, so
+ *   committing an epoch that is already committed must have no effect, for 
exactly-once output. A
+ *   restarted query may also abort an epoch that an earlier run of the query 
committed, so an abort
+ *   must only remove what the data writers of its own writer wrote, not the 
output that an earlier
+ *   writer committed. The ID of the query, which create_writer gets for a 
streaming write, stays
  *   the same when the query restarts from its checkpoint, so the ID of the 
query and the epoch ID
  *   identify the output of a micro-batch, also when other queries write to 
the same destination. A
  *   micro-batch streaming query creates a new stream writer for each 
micro-batch, and Spark closes
@@ -121,6 +126,18 @@
  *   released once they are no longer referenced (see Lifecycle above). A 
reader or writer handle
  *   must not depend on the data source handle it was created from, because 
Spark closes the data
  *   source handle right after creating it.
+ * - Order of the calls. After a function on a handle fails, Spark calls no 
other function on the
+ *   handle than its close function, except that it aborts a writer whose 
commit failed and a data
+ *   writer whose write or commit failed. It does not abort a writer or a data 
writer whose commit
+ *   succeeded, though the writer of a restarted streaming query may abort an 
epoch that an earlier
+ *   writer committed (see Writes above). It calls each function of a batch 
reader at most once,
+ *   but it may call read more than once for the same partition, also at the 
same time, in several
+ *   attempts of its task: after a failure, with speculative execution, or 
when it recomputes lost
+ *   data. It calls commit and abort at most once each for a writer, and 
commit_data_writer and
+ *   abort_data_writer at most once each for a data writer, with all the calls 
to write before
+ *   them. The data writers of the task attempts that Spark kills, such as 
speculative ones, may
+ *   still be written to, aborted and closed after Spark committed or aborted 
their writer and
+ *   closed it.
  * - Memory. What Spark passes to a function, such as strings, byte arrays and 
the structs whose
  *   names end with Info, is only valid during the call, except for 
SparkDsHost and
  *   SparkDsTaskContext, which document how long they live. Strings are UTF-8 
and NUL-terminated:
@@ -269,7 +286,7 @@ typedef void (*SparkDsSink)(void* sink, const uint8_t* 
data, int64_t length);
  * What Spark passes to the library
  * 
---------------------------------------------------------------------------------------------
 */
 
-/* The levels of the messages that a library logs with SparkDsHost.log. */
+/* The levels of the messages of SparkDsHost.log and 
SparkDsHost.is_log_enabled. */
 #define SPARK_DS_LOG_ERROR 1
 #define SPARK_DS_LOG_WARN 2
 #define SPARK_DS_LOG_INFO 3
@@ -289,9 +306,21 @@ typedef struct SparkDsHost {
   const char* spark_version;
   /*
    * Logs a message, a NUL-terminated UTF-8 string, at one of the 
SPARK_DS_LOG_* levels: in the log
-   * of the driver or of the executor, with the path of the library.
+   * of the driver or of the executor, with the path of the library. Another 
level is treated as
+   * SPARK_DS_LOG_INFO.
    */
   void (*log)(int32_t level, const char* message);
+  /*
+   * Returns whether the logger of Spark that log writes to is enabled for one 
of the
+   * SPARK_DS_LOG_* levels, another level being treated as SPARK_DS_LOG_INFO, 
so that the library
+   * only formats the messages that Spark may write, and can log at the levels 
that it would not log
+   * by default, such as debug, when Spark does. The configuration of the log 
can still drop a
+   * message of an enabled level, for example with a filter. It follows the 
level of the
+   * logger, which can change while Spark runs, but each call is a call into 
the JVM: a library
+   * may keep the answer for a while, and ask again from time to time. It is 
never NULL in a struct
+   * whose size holds it.
+   */
+  bool (*is_log_enabled)(int32_t level);
 } SparkDsHost;
 
 /* A data source, as create gets it. */
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index d72c95b0596f..47f19d42aab2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -146,8 +146,8 @@ trait ColumnarStreamReader extends Serializable {
 /**
  * Writes data to a [[ColumnarDataSource]] for a batch write. Spark serializes 
the writer to the
  * executors, where `createWriter` is called for each task, and then commits 
or aborts the write
- * on the driver. The batches passed to the data writers are backed by Arrow, 
and are only valid
- * during the call.
+ * on the driver. Spark does not abort a write whose commit succeeded. The 
batches passed to the
+ * data writers are backed by Arrow, and are only valid during the call.
  */
 trait ColumnarWriter extends Serializable {
   def createWriter(partitionId: Int, taskId: Long): DataWriter[ColumnarBatch]
@@ -157,7 +157,11 @@ trait ColumnarWriter extends Serializable {
   def abort(messages: Array[WriterCommitMessage]): Unit
 }
 
-/** Writes data to a [[ColumnarDataSource]] for a streaming write, one 
micro-batch at a time. */
+/**
+ * Writes data to a [[ColumnarDataSource]] for a streaming write, one 
micro-batch at a time. Spark
+ * does not abort a micro-batch once the writer committed it, but a restarted 
query may abort a
+ * micro-batch that a writer of an earlier run of the query committed.
+ */
 trait ColumnarStreamWriter extends Serializable {
   def createWriter(partitionId: Int, taskId: Long, epochId: Long): 
DataWriter[ColumnarBatch]
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
index 2037d0b0cf40..21bbbeb2e5b1 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarTable.scala
@@ -105,6 +105,8 @@ class ColumnarScanBuilder(
   private var limitResult: Option[(Int, Boolean)] = None
   // The top-level columns that the query needs, if fewer than those of the 
schema.
   private var requiredSchema: Option[StructType] = None
+  // Whether the columns were pruned.
+  private var pruned = false
 
   private def batchPlanner: ColumnarBatchPlanner = {
     if (planner == null) {
@@ -137,8 +139,17 @@ class ColumnarScanBuilder(
     // reader.
     val required =
       StructType(requiredSchema.fieldNames.flatMap(name => schema.find(_.name 
== name)))
-    if (required.length < schema.length) {
-      this.requiredSchema = Some(required)
+    if (!pruned) {
+      // The columns are pruned at most once, since the scan asks the reader 
once, as the header
+      // promises, though Spark prunes the columns of a scan builder once 
anyway.
+      pruned = true
+      if (required.length < schema.length) {
+        this.requiredSchema = Some(required)
+      }
+    } else if (!required.fieldNames.toSet.subsetOf(
+        this.requiredSchema.getOrElse(schema).fieldNames.toSet)) {
+      throw SparkException.internalError(
+        "The columns of a scan were pruned again, with columns that it does 
not read.")
     }
   }
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
index 09ff622e7dcd..60588c470843 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
@@ -75,13 +75,16 @@ class ColumnarWrite(
  * A batch write. The writer of the data source is created with the writer 
factory, right before
  * the tasks of the write start: after the shuffle stages that adaptive query 
execution runs
  * before the write, so no writer is created if they fail. The rest of the 
data, such as its last
- * stage, is computed in the tasks of the write, and a failure there, as any 
later failure, aborts
- * the write. So irreversible work belongs in the commit of the writer.
+ * stage, is computed in the tasks of the write, and a failure there, as any 
later failure before
+ * the commit succeeds, aborts the write. So irreversible work belongs in the 
commit of the writer.
  */
 class ColumnarBatchWrite(createWriter: () => ColumnarWriter, schema: 
StructType)
   extends BatchWrite {
 
   private var writer: ColumnarWriter = _
+  // Whether the writer committed the write, after which it is not aborted, 
even if Spark fails
+  // afterwards: a native data source relies on it.
+  private var committed = false
 
   override def createBatchWriterFactory(info: PhysicalWriteInfo): 
DataWriterFactory = {
     if (writer == null) {
@@ -90,10 +93,13 @@ class ColumnarBatchWrite(createWriter: () => 
ColumnarWriter, schema: StructType)
     new ColumnarDataWriterFactory(writer, ArrowBatchingOptions(schema))
   }
 
-  override def commit(messages: Array[WriterCommitMessage]): Unit = 
writer.commit(messages)
+  override def commit(messages: Array[WriterCommitMessage]): Unit = {
+    writer.commit(messages)
+    committed = true
+  }
 
   override def abort(messages: Array[WriterCommitMessage]): Unit = {
-    if (writer != null) {
+    if (writer != null && !committed) {
       writer.abort(messages)
     }
   }
@@ -115,6 +121,9 @@ class ColumnarStreamingWrite(
   extends StreamingWrite {
 
   private var writer: ColumnarStreamWriter = _
+  // Whether the writer committed the micro-batch, which is not aborted then, 
as for a batch write.
+  // Spark creates a streaming write for each micro-batch, so it commits at 
most one epoch.
+  private var committed = false
 
   override def createStreamingWriterFactory(
       info: PhysicalWriteInfo): StreamingDataWriterFactory = {
@@ -132,10 +141,11 @@ class ColumnarStreamingWrite(
 
   override def commit(epochId: Long, messages: Array[WriterCommitMessage]): 
Unit = {
     writer.commit(epochId, messages)
+    committed = true
   }
 
   override def abort(epochId: Long, messages: Array[WriterCommitMessage]): 
Unit = {
-    if (writer != null) {
+    if (writer != null && !committed) {
       writer.abort(epochId, messages)
     }
   }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
index 75dadec67441..0e359ff7f72f 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSource.scala
@@ -415,7 +415,8 @@ class NativeDataSourceWriter(
     call(name, "commit a write to") {
       library.commit(handle.get, -1L, 
NativeWriterCommitMessage.toBytes(messages))
     }
-    // Not closed if the commit fails: Spark then aborts the write, which 
closes it.
+    // Not closed if the commit fails: Spark then aborts the write, which 
closes it. Spark does
+    // not abort a committed write, see ColumnarBatchWrite.
     handle.close()
   }
 
@@ -453,6 +454,7 @@ class NativeDataSourceStreamWriter(
       location, name, schema, timeZoneId, writerState, partitionId, taskId, 
epochId)
   }
 
+  // Spark does not abort a committed micro-batch, see ColumnarStreamingWrite.
   override def commit(epochId: Long, messages: Array[WriterCommitMessage]): 
Unit = {
     call(name, "commit a write to") {
       library.commit(handle.get, epochId, 
NativeWriterCommitMessage.toBytes(messages))
@@ -970,11 +972,26 @@ class NativeDataWriter(
   // even if Spark then fails the commit, for example because of an invalid 
message.
   private var finished = false
   private var closed = false
+  // Whether a write or a commit failed, after which the library only gets 
abort and close, as the
+  // header says.
+  private var failed = false
 
-  override def write(batch: ColumnarBatch): Unit = {
+  // Fails a write or a commit that the header rules out, and records a 
failure of `f`.
+  private def writeOrCommit[T](f: => T): T = {
     if (finished || closed) {
       throw SparkException.internalError("The data writer was already 
committed or aborted.")
     }
+    if (failed) {
+      throw SparkException.internalError("A write or a commit of the data 
writer failed.")
+    }
+    try f catch {
+      case e: Throwable =>
+        failed = true
+        throw e
+    }
+  }
+
+  override def write(batch: ColumnarBatch): Unit = writeOrCommit {
     val vectors = (0 until batch.numCols()).map { i =>
       batch.column(i) match {
         case vector: ArrowColumnVector => 
vector.getValueVector.asInstanceOf[FieldVector]
@@ -996,10 +1013,7 @@ class NativeDataWriter(
     }
   }
 
-  override def commit(): WriterCommitMessage = {
-    if (finished || closed) {
-      throw SparkException.internalError("The data writer was already 
committed or aborted.")
-    }
+  override def commit(): WriterCommitMessage = writeOrCommit {
     val message = call(name, "write to")(library.commitDataWriter(handle, () 
=> finished = true))
     NativeWriterCommitMessage(bytesOrEmpty(Some(message)))
   }
diff --git 
a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc 
b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
index aebd15806f80..4fcb530179cd 100644
--- a/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
+++ b/sql/core/src/test/resources/native-datasource/test_native_datasource.cc
@@ -74,17 +74,21 @@
 // "capabilities", a data source declares these capabilities instead, a 
number. With the option
 // "task_path", the reads and the data writers of native_range and native_sink 
record their task
 // in a file named after its partition, in that directory. With the option 
"log", creating a data
-// source logs its value as a warning, and with the option "spark_version", it 
fails unless Spark
-// has that version.
+// source logs its value as a warning, and with "log_calls_path" also appends 
the level to that
+// file when it calls log. With the option "log_levels_path", it writes to 
that file whether Spark
+// logs each level, from error to trace, such as "1,1,1,0,0", and with the 
option "spark_version",
+// it fails unless Spark has that version.
 //
-// With the option "fail", a step fails with an error. With the option "zero", 
a function that
-// creates a handle succeeds without setting it: "create", "create_reader",
-// "create_stream_reader", "create_writer" or "create_data_writer". With the 
option "states", the
-// functions that return a state pass it that many times. With the option 
"stateless", readers and
-// writers have no state: partitions, stream_partitions and create_writer do 
not call set_state.
-// read then fails unless it gets a state of length 0, and takes the columns 
from the schema to
-// read. A data writer that gets a state of length 0 passes its rows to commit 
in its message,
-// and commit fails unless the messages have rows.
+// With the option "fail", a step fails with an error, such as a function of a 
reader:
+// "push_limit", "prune_columns", "plan", "initial_offset", "latest_offset", 
"stream_partitions"
+// or "commit_offset". With the option "zero", a function that creates a 
handle succeeds without
+// setting it: "create", "create_reader", "create_stream_reader", 
"create_writer" or
+// "create_data_writer". With the option "states", the functions that return a 
state pass it that
+// many times. With the option "stateless", readers and writers have no state: 
partitions,
+// stream_partitions and create_writer do not call set_state. read then fails 
unless it gets a
+// state of length 0, and takes the columns from the schema to read. A data 
writer that gets a
+// state of length 0 passes its rows to commit in its message, and commit 
fails unless the
+// messages have rows.
 //
 // read fails unless it gets the kind of read of its reader and a schema to 
read whose columns
 // are columns of the reader, with the names and the formats that the reader 
got them with, and
@@ -104,6 +108,10 @@
 //                   the functions after it.
 //   INIT_FAILURE    1 to make spark_ds_init fail, after logging why, 2 to 
make it succeed without
 //                   a table.
+//   ORDER_VIOLATIONS_PATH
+//                   A file to which the calls that the order of the calls of 
the header rules out
+//                   are appended, as lines, which the suites check at the end 
of each test, with
+//                   checkingOrderOfCalls.
 //   NO_INIT         Leaves out spark_ds_init, which makes the library invalid.
 
 #include "spark_datasource.h"
@@ -706,14 +714,43 @@ thread_local std::string last_error;
 // Spark, as spark_ds_init got it.
 const SparkDsHost* host = nullptr;
 
-// Logs with the log of Spark. A library must only use the fields that fit in 
the size of the
-// struct that Spark passes, which may be of another version of the header.
-void hostLog(int32_t level, const std::string& message) {
+// Whether the struct that Spark passes has is_log_enabled, which an earlier 
version of the header
+// does not have.
+bool hasIsLogEnabled() {
+  return host->size >= static_cast<int64_t>(offsetof(SparkDsHost, 
is_log_enabled) +
+                                            sizeof(host->is_log_enabled));
+}
+
+// Appends a line to a file, or fails.
+void appendLine(const std::string& path, const std::string& line) {
+  FILE* file = std::fopen(path.c_str(), "a");
+  if (file == nullptr) throw std::runtime_error("cannot write " + path);
+  std::fputs((line + "\n").c_str(), file);
+  std::fclose(file);
+}
+
+// Logs with the log of Spark, unless it does not write the level. A library 
must only use the
+// fields that fit in the size of the struct that Spark passes, which may be 
of another version of
+// the header. With `calls_path`, appends the level to that file right before 
it calls log.
+void hostLog(int32_t level, const std::string& message, const std::string& 
calls_path = "") {
+  if (hasIsLogEnabled() && !host->is_log_enabled(level)) return;
   if (host->size >= static_cast<int64_t>(offsetof(SparkDsHost, log) + 
sizeof(host->log))) {
+    if (!calls_path.empty()) appendLine(calls_path, std::to_string(level));
     host->log(level, message.c_str());
   }
 }
 
+// Whether Spark logs each level, from error to trace, such as "1,1,1,0,0".
+std::string logLevels() {
+  if (!hasIsLogEnabled()) throw std::runtime_error("the host has no 
is_log_enabled");
+  std::string levels;
+  for (int32_t level = SPARK_DS_LOG_ERROR; level <= SPARK_DS_LOG_TRACE; 
level++) {
+    if (level > SPARK_DS_LOG_ERROR) levels += ",";
+    levels += host->is_log_enabled(level) ? "1" : "0";
+  }
+  return levels;
+}
+
 // Runs `f`, and turns a C++ exception into an error: none may cross the 
boundary.
 template <typename F>
 int guarded(F&& f) {
@@ -792,6 +829,30 @@ void writeFile(const std::string& path, const std::string& 
content) {
   std::fclose(file);
 }
 
+// Records a call that the order of the calls of the header rules out. Unlike 
the files of the
+// options, a failure to record it is ignored: it is called outside of 
`guarded`, by the checks at
+// the start of the functions on a handle, before `tracked`.
+std::mutex order_violations_lock;
+void orderViolation(const std::string& call) {
+#ifdef ORDER_VIOLATIONS_PATH
+  std::lock_guard<std::mutex> guard(order_violations_lock);
+  try {
+    appendLine(ORDER_VIOLATIONS_PATH, call);
+  } catch (...) {
+  }
+#else
+  (void)call;
+#endif
+}
+
+// Runs a function on a handle with `guarded`, and records in `failed` whether 
it failed.
+template <typename F>
+int tracked(bool* failed, F&& f) {
+  int status = guarded(f);
+  if (status != 0) *failed = true;
+  return status;
+}
+
 void makeDirs(const std::string& path) {
   for (size_t pos = path.find('/', 1); pos != std::string::npos; pos = 
path.find('/', pos + 1)) {
     mkdir(path.substr(0, pos).c_str(), 0755);
@@ -816,6 +877,12 @@ void recordTask(const Options& options, const 
SparkDsTaskContext* task) {
 struct Reader {
   Options state;  // Serialized to the executors, unless the reader is 
stateless.
   int64_t stream_partitions_calls = 0;
+  // Whether a function failed, and which functions of a batch reader were 
called, for the order of
+  // the calls.
+  bool failed = false;
+  bool limited = false;
+  bool pruned = false;
+  bool planned = false;
 };
 
 // Whether a reader or a writer has no state, with the option "stateless".
@@ -955,6 +1022,10 @@ struct Writer {
   // The file that closeWriter creates, so that the tests can check that the 
writer of a committed
   // or aborted write is closed: _CLOSED for a batch write, _CLOSED_<epoch> 
for a streaming one.
   std::string closed_marker;
+  // The calls of commit and abort, and whether commit succeeded, for the 
order of the calls.
+  bool commit_called = false;
+  bool abort_called = false;
+  bool committed = false;
 };
 
 [[maybe_unused]] std::string closedMarker(int64_t epoch_id) {
@@ -972,8 +1043,20 @@ struct DataWriter {
   // rather than writing them to a file.
   bool stateless;
   std::string rows;
+  // Whether write or commit_data_writer failed, and the calls of 
commit_data_writer and
+  // abort_data_writer, for the order of the calls.
+  bool failed = false;
+  bool commit_called = false;
+  bool abort_called = false;
 };
 
+// Records a write or a commit of a data writer that the order of the calls 
rules out.
+[[maybe_unused]] void checkWriteOrCommit(const DataWriter* writer, const 
std::string& function) {
+  if (writer->failed) orderViolation(function + " after a failed call");
+  if (writer->commit_called) orderViolation(function + " after 
commit_data_writer");
+  if (writer->abort_called) orderViolation(function + " after 
abort_data_writer");
+}
+
 // The data writers that Spark closed, with the path of their write, so that 
using one of them
 // afterwards, which the header forbids, is recorded in the file 
_DATA_WRITER_USED_AFTER_CLOSE
 // rather than crashing.
@@ -1067,7 +1150,11 @@ int createSource(const SparkDsCreateInfo* info, 
uint64_t* capabilities, void** h
       throw std::runtime_error("unknown data source " + source->name);
     }
     std::string message = get(source->options, "log", "");
-    if (!message.empty()) hostLog(SPARK_DS_LOG_WARN, message);
+    if (!message.empty()) {
+      hostLog(SPARK_DS_LOG_WARN, message, get(source->options, 
"log_calls_path", ""));
+    }
+    std::string levels_path = get(source->options, "log_levels_path", "");
+    if (!levels_path.empty()) writeFile(levels_path, logLevels());
     std::string version = get(source->options, "spark_version", "");
     if (!version.empty() && version != host->spark_version) {
       throw std::runtime_error("the version of Spark is " + 
std::string(host->spark_version) +
@@ -1167,20 +1254,29 @@ int createStreamReader(void* handle, struct 
ArrowSchema* schema,
 }
 
 int pushLimit(void* handle, int32_t limit, bool* applied) {
-  return guarded([&] {
-    auto* reader = fromHandle<Reader>(handle);
-    // Spark calls push_limit at most once.
-    if (reader->state.count("limit") > 0) throw std::runtime_error("pushLimit 
was called twice");
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("push_limit after a failed call");
+  if (reader->limited) orderViolation("push_limit called twice");
+  // Spark pushes the limit before it prunes the columns.
+  if (reader->pruned) orderViolation("push_limit after prune_columns");
+  reader->limited = true;
+  return tracked(&reader->failed, [&] {
+    maybeFail(reader->state, "push_limit");
     reader->state["limit"] = std::to_string(limit);
     *applied = true;
   });
 }
 
 int pruneColumns(void* handle, const char* const* columns, int64_t count) {
-  return guarded([&] {
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("prune_columns after a failed call");
+  if (reader->pruned) orderViolation("prune_columns called twice");
+  reader->pruned = true;
+  return tracked(&reader->failed, [&] {
+    maybeFail(reader->state, "prune_columns");
     auto names = toStrings(columns, count);
     for (const auto& column : names) findColumn(column);
-    fromHandle<Reader>(handle)->state["columns"] = join(names);
+    reader->state["columns"] = join(names);
   });
 }
 
@@ -1188,9 +1284,12 @@ int pruneColumns(void* handle, const char* const* 
columns, int64_t count) {
 
 int planPartitions(void* handle, const SparkDsPartitionsInfo* info, 
SparkDsSink add_partition,
                    SparkDsSink set_state, void* sink) {
-  return guarded([&] {
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("partitions after a failed call");
+  if (reader->planned) orderViolation("partitions called twice");
+  reader->planned = true;
+  return tracked(&reader->failed, [&] {
     checkSize("SparkDsPartitionsInfo", info->size, 
sizeof(SparkDsPartitionsInfo));
-    auto* reader = fromHandle<Reader>(handle);
     maybeFail(reader->state, "plan");
     int64_t lo = getLong(reader->state, "lo", 0);
     int64_t hi = getLong(reader->state, "hi", 0);
@@ -1235,9 +1334,12 @@ void sendOffset(const Options& state, const std::string& 
bad_option, SparkDsSink
 
 int initialOffset(void* handle, const SparkDsInitialOffsetInfo* info, 
SparkDsSink set_offset,
                   void* sink) {
-  return guarded([&] {
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("initial_offset after a failed call");
+  return tracked(&reader->failed, [&] {
     checkSize("SparkDsInitialOffsetInfo", info->size, 
sizeof(SparkDsInitialOffsetInfo));
-    const Options& state = fromHandle<Reader>(handle)->state;
+    const Options& state = reader->state;
+    maybeFail(state, "initial_offset");
     int64_t offset = 0;
     std::string path = get(state, "initial_offsets_path", "");
     if (!path.empty()) {
@@ -1256,17 +1358,15 @@ int initialOffset(void* handle, const 
SparkDsInitialOffsetInfo* info, SparkDsSin
 // Makes `step` more rows available after the start of the micro-batch, up to 
`max_offset`.
 int latestOffset(void* handle, const SparkDsLatestOffsetInfo* info, 
SparkDsSink set_offset,
                  void* sink) {
-  return guarded([&] {
-    const Options& state = fromHandle<Reader>(handle)->state;
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("latest_offset after a failed call");
+  return tracked(&reader->failed, [&] {
+    const Options& state = reader->state;
     checkSize("SparkDsLatestOffsetInfo", info->size, 
sizeof(SparkDsLatestOffsetInfo));
+    maybeFail(state, "latest_offset");
     if (info->start == nullptr) throw std::runtime_error("no start offset");
     std::string starts_path = get(state, "starts_path", "");
-    if (!starts_path.empty()) {
-      FILE* file = std::fopen(starts_path.c_str(), "a");
-      if (file == nullptr) throw std::runtime_error("cannot write " + 
starts_path);
-      std::fputs((std::string(info->start) + "\n").c_str(), file);
-      std::fclose(file);
-    }
+    if (!starts_path.empty()) appendLine(starts_path, info->start);
     auto start = 
static_cast<int64_t>(JsonParser(info->start).parse()["offset"].number);
     int64_t latest =
         std::min(getLong(state, "max_offset", 10), start + getLong(state, 
"step", 10));
@@ -1276,9 +1376,10 @@ int latestOffset(void* handle, const 
SparkDsLatestOffsetInfo* info, SparkDsSink
 
 int streamPartitions(void* handle, const SparkDsStreamPartitionsInfo* info,
                      SparkDsSink add_partition, SparkDsSink set_state, void* 
sink) {
-  return guarded([&] {
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("stream_partitions after a failed call");
+  return tracked(&reader->failed, [&] {
     checkSize("SparkDsStreamPartitionsInfo", info->size, 
sizeof(SparkDsStreamPartitionsInfo));
-    auto* reader = fromHandle<Reader>(handle);
     maybeFail(reader->state, "stream_partitions");
     std::string ranges_path = get(reader->state, "ranges_path", "");
     if (!ranges_path.empty()) {
@@ -1307,9 +1408,12 @@ int streamPartitions(void* handle, const 
SparkDsStreamPartitionsInfo* info,
 }
 
 int commitOffset(void* handle, const SparkDsCommitOffsetInfo* info) {
-  return guarded([&] {
+  auto* reader = fromHandle<Reader>(handle);
+  if (reader->failed) orderViolation("commit_offset after a failed call");
+  return tracked(&reader->failed, [&] {
     checkSize("SparkDsCommitOffsetInfo", info->size, 
sizeof(SparkDsCommitOffsetInfo));
-    std::string path = get(fromHandle<Reader>(handle)->state, "commit_path", 
"");
+    maybeFail(reader->state, "commit_offset");
+    std::string path = get(reader->state, "commit_path", "");
     if (!path.empty()) writeFile(path, info->end);
   });
 }
@@ -1443,11 +1547,14 @@ int createWriter(void* handle, struct ArrowSchema* 
schema, const SparkDsWriteInf
 }
 
 int commitWrite(void* handle, const SparkDsFinishWriteInfo* info) {
+  auto* writer = fromHandle<Writer>(handle);
+  if (writer->commit_called) orderViolation("commit called twice");
+  if (writer->abort_called) orderViolation("commit after abort");
+  writer->commit_called = true;
   return guarded([&] {
     checkSize("SparkDsFinishWriteInfo", info->size, 
sizeof(SparkDsFinishWriteInfo));
     int64_t epoch_id = info->epoch_id;
     if (info->rejected) throw std::runtime_error("commit of a rejected 
writer");
-    auto* writer = fromHandle<Writer>(handle);
     std::string path = writer->state["path"];
     maybeFail(writer->state, "commit");
     bool has_null = false;
@@ -1474,6 +1581,7 @@ int commitWrite(void* handle, const 
SparkDsFinishWriteInfo* info) {
     writeFile(path + (epoch_id < 0 ? "/_SUCCESS" : "/_EPOCH_" + 
std::to_string(epoch_id)),
               std::to_string(messages.size()));
     writer->closed_marker = closedMarker(epoch_id);
+    writer->committed = true;
   });
 }
 
@@ -1482,10 +1590,13 @@ int commitWrite(void* handle, const 
SparkDsFinishWriteInfo* info) {
 // tasks that did not commit, the number of messages, and 1 for a writer whose 
create_writer
 // Spark rejected.
 int abortWrite(void* handle, const SparkDsFinishWriteInfo* info) {
+  auto* writer = fromHandle<Writer>(handle);
+  if (writer->abort_called) orderViolation("abort called twice");
+  if (writer->committed) orderViolation("abort after a successful commit");
+  writer->abort_called = true;
   return guarded([&] {
     checkSize("SparkDsFinishWriteInfo", info->size, 
sizeof(SparkDsFinishWriteInfo));
     int64_t epoch_id = info->epoch_id;
-    auto* writer = fromHandle<Writer>(handle);
     writer->closed_marker = closedMarker(epoch_id);
     bool has_null = false;
     for (const auto& message :
@@ -1572,8 +1683,9 @@ int createDataWriter(const SparkDsTaskContext* task, 
const SparkDsDataWriterInfo
 }
 
 int writeBatch(void* handle, struct ArrowArray* array) {
-  return guarded([&] {
-    auto* writer = fromHandle<DataWriter>(handle);
+  auto* writer = fromHandle<DataWriter>(handle);
+  checkWriteOrCommit(writer, "write");
+  return tracked(&writer->failed, [&] {
     // The batch is released here, so Spark does not release it after the call.
     std::unique_ptr<ArrowArray, void (*)(ArrowArray*)> array_owner(
         array, [](ArrowArray* a) { a->release(a); });
@@ -1603,9 +1715,11 @@ int writeBatch(void* handle, struct ArrowArray* array) {
 int commitDataWriter(void* handle, const SparkDsFinishDataWriterInfo* info,
                      SparkDsSink set_message, void* sink) {
   if (usedAfterClose(handle)) return 0;
-  return guarded([&] {
+  auto* writer = fromHandle<DataWriter>(handle);
+  checkWriteOrCommit(writer, "commit_data_writer");
+  writer->commit_called = true;
+  return tracked(&writer->failed, [&] {
     checkSize("SparkDsFinishDataWriterInfo", info->size, 
sizeof(SparkDsFinishDataWriterInfo));
-    auto* writer = fromHandle<DataWriter>(handle);
     // If the commit fails, Spark aborts the data writer.
     maybeFail(writer->state, "commitDataWriter");
     if (writer->stateless) {
@@ -1624,19 +1738,21 @@ int commitDataWriter(void* handle, const 
SparkDsFinishDataWriterInfo* info,
 
 int abortDataWriter(void* handle, const SparkDsFinishDataWriterInfo* info) {
   if (usedAfterClose(handle)) return 0;
+  auto* writer = fromHandle<DataWriter>(handle);
+  if (writer->abort_called) orderViolation("abort_data_writer called twice");
+  writer->abort_called = true;
+  // Spark does not abort a data writer that committed: records it if it does, 
and keeps its file.
+  if (writer->committed) {
+    orderViolation("abort_data_writer after a successful commit");
+    return 0;
+  }
   return guarded([&] {
     checkSize("SparkDsFinishDataWriterInfo", info->size, 
sizeof(SparkDsFinishDataWriterInfo));
-    auto* writer = fromHandle<DataWriter>(handle);
-    // A data writer without a state has no file, nor a path to record 
anything under.
+    // A data writer without a state has no file.
     if (writer->stateless) {
       writer->rows.clear();
       return;
     }
-    // Spark does not abort a data writer that committed: records it if it 
does.
-    if (writer->committed) {
-      writeFile(writer->state["path"] + "/_ABORTED_COMMITTED_DATA_WRITER", "");
-      return;
-    }
     if (writer->file != nullptr) {
       std::fclose(writer->file);
       writer->file = nullptr;
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 8ba35951fd01..2433ea3cb20f 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -24,14 +24,15 @@ import java.util.concurrent.atomic.AtomicInteger
 import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
 
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkException, TaskContextImpl}
 import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.connector.catalog.{Table, TableCapability, 
TableProvider}
 import org.apache.spark.sql.connector.expressions.Transform
 import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
 import org.apache.spark.sql.connector.read.streaming.{ReadLimit, 
SupportsAdmissionControl}
-import org.apache.spark.sql.connector.write.{DataWriter, WriterCommitMessage}
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.connector.write.{DataWriter, 
PhysicalWriteInfoImpl, WriterCommitMessage}
+import 
org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, 
DataWritingSparkTask}
 import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
 import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector
 import org.apache.spark.sql.internal.SQLConf
@@ -206,6 +207,94 @@ class ColumnarDataSourceSuite extends QueryTest with 
SharedSparkSession {
     assert(failingPruning.prunes.get() == 1 && 
failingPruning.batchPlanners.get() == 1)
   }
 
+  test("prune the columns of a scan once") {
+    val dataSource = new TestRangeDataSource(options("end" -> "4"))
+    val builder = new ColumnarScanBuilder(format, dataSource.schema(), () => 
dataSource)
+    builder.pruneColumns(StructType.fromDDL("id BIGINT"))
+    // Pruning again with the same columns, or fewer, keeps the first ones.
+    builder.pruneColumns(StructType.fromDDL("id BIGINT"))
+    builder.pruneColumns(new StructType())
+    val scan = builder.build()
+    scan.toBatch.planInputPartitions()
+    assert(scan.readSchema() == StructType.fromDDL("id BIGINT"))
+    assert(dataSource.prunes.get() == 1)
+    // Columns that the scan does not read cannot be added afterwards.
+    val e = 
intercept[SparkException](builder.pruneColumns(StructType.fromDDL("name 
STRING")))
+    assert(e.getMessage.contains("pruned again"))
+    assert(dataSource.prunes.get() == 1)
+
+    // A failure is kept too, so the planner is not asked again, and Spark 
gets the same error.
+    val failing = new TestRangeDataSource(options("end" -> "4", "failPruning" 
-> "true"))
+    val failingBuilder = new ColumnarScanBuilder(format, failing.schema(), () 
=> failing)
+    failingBuilder.pruneColumns(StructType.fromDDL("id BIGINT"))
+    val failingScan = failingBuilder.build()
+    // The read schema does not need the planner.
+    assert(failingScan.readSchema() == StructType.fromDDL("id BIGINT"))
+    val failure = intercept[IllegalStateException](failingScan.toBatch)
+    assert(failure.getMessage == "injected pruning failure")
+    assert(intercept[IllegalStateException](failingScan.toBatch) eq failure)
+    assert(failing.prunes.get() == 1)
+  }
+
+  test("do not abort a write whose commit succeeded") {
+    val dataSource = new TestRangeDataSource(options("table" -> 
"abort_after_commit"))
+    val schema = dataSource.schema()
+    val batchWrite = new ColumnarBatchWrite(() => dataSource.writer(schema, 
false), schema)
+    batchWrite.createBatchWriterFactory(PhysicalWriteInfoImpl(1))
+    batchWrite.commit(Array.empty)
+    batchWrite.abort(Array.empty)
+    assert(TestColumnarSink.committed.containsKey("abort_after_commit"))
+    assert(!TestColumnarSink.aborted.contains("abort_after_commit"))
+
+    val streamingWrite = new ColumnarStreamingWrite(
+      format, () => dataSource.streamWriter(schema, false, "query"), schema)
+    streamingWrite.createStreamingWriterFactory(PhysicalWriteInfoImpl(1))
+    streamingWrite.commit(0, Array.empty)
+    streamingWrite.abort(0, Array.empty)
+    assert(TestColumnarSink.epochs.get("abort_after_commit") == Seq(0L))
+    assert(!TestColumnarSink.aborted.contains("abort_after_commit"))
+
+    // Without a commit, both writes are aborted.
+    val failedWrite = new ColumnarBatchWrite(() => dataSource.writer(schema, 
false), schema)
+    failedWrite.createBatchWriterFactory(PhysicalWriteInfoImpl(1))
+    failedWrite.abort(Array.empty)
+    assert(TestColumnarSink.aborted.remove("abort_after_commit"))
+    val failedStreamingWrite = new ColumnarStreamingWrite(
+      format, () => dataSource.streamWriter(schema, false, "query"), schema)
+    failedStreamingWrite.createStreamingWriterFactory(PhysicalWriteInfoImpl(1))
+    failedStreamingWrite.abort(1, Array.empty)
+    assert(TestColumnarSink.aborted.contains("abort_after_commit"))
+  }
+
+  test("abort a data writer whose commit the coordinator denies, without 
committing it") {
+    val calls = ArrayBuffer.empty[String]
+    val writer = new ColumnarWriter {
+      override def createWriter(partitionId: Int, taskId: Long): 
DataWriter[ColumnarBatch] =
+        new DataWriter[ColumnarBatch] {
+          override def write(batch: ColumnarBatch): Unit = calls += "write"
+          override def commit(): WriterCommitMessage = {
+            calls += "commit"
+            null
+          }
+          override def abort(): Unit = calls += "abort"
+          override def close(): Unit = calls += "close"
+        }
+      override def commit(messages: Array[WriterCommitMessage]): Unit = {}
+      override def abort(messages: Array[WriterCommitMessage]): Unit = {}
+    }
+    val batchWrite = new ColumnarBatchWrite(() => writer, 
StructType.fromDDL("id BIGINT"))
+    val factory = batchWrite.createBatchWriterFactory(PhysicalWriteInfoImpl(1))
+    // The coordinator denies the commits of a stage that it does not know, as 
it denies those of
+    // the other attempts of a partition whose commit it authorized.
+    val context = new TaskContextImpl(Int.MaxValue, 0, 0, 0L, 0, 1, null, new 
util.Properties, null)
+    val e = intercept[SparkException] {
+      DataWritingSparkTask.run(factory, context, Iterator(InternalRow(1L)),
+        batchWrite.useCommitCoordinator(), Map.empty)
+    }
+    assert(e.getCondition == "_LEGACY_ERROR_TEMP_2071")
+    assert(calls == Seq("abort", "close"))
+  }
+
   test("a streaming scan reads only the columns that the query needs") {
     val dataSource = new TestRangeDataSource(options("end" -> "3"))
     val builder = new ColumnarScanBuilder(format, dataSource.schema(), () => 
dataSource)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceClusterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceClusterSuite.scala
index bc7f369ba930..c50997d6bba9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceClusterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceClusterSuite.scala
@@ -69,9 +69,10 @@ class NativeDataSourceClusterSuite extends QueryTest {
     }
   }
 
+  // The executors append the calls that the order of the calls rules out to 
the same file.
   private def nativeTest(name: String)(f: => Unit): Unit = test(name) {
     assume(toolchain.isDefined, "A C++ compiler and Java 22 or later are 
required.")
-    try f finally SparkSession.setActiveSession(spark)
+    try checkingOrderOfCalls(f) finally SparkSession.setActiveSession(spark)
   }
 
   // Like Python data sources, native data sources are found in the active 
session.
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 84ed08d58a5c..9dc836ce12e6 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -31,9 +31,10 @@ import java.util.zip.{ZipException, ZipFile}
 import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
 
+import org.apache.arrow.vector.BigIntVector
 import org.scalatest.time.SpanSugar._
 
-import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TestUtils}
+import org.apache.spark.{SPARK_VERSION, SparkClassNotFoundException, 
SparkException, SparkFiles, SparkRuntimeException, SparkThrowable, 
SparkUnsupportedOperationException, TestUtils}
 import org.apache.spark.api.python.{PythonBroadcast, SimplePythonFunction}
 import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, 
QueryTest, Row}
@@ -42,10 +43,11 @@ import org.apache.spark.sql.classic.{DataFrame, 
SparkSession}
 import org.apache.spark.sql.connector.catalog.SupportsRead
 import org.apache.spark.sql.connector.read.SupportsPushDownRequiredColumns
 import org.apache.spark.sql.connector.read.streaming.{Offset, ReadLimit, 
SupportsAdmissionControl}
+import org.apache.spark.sql.connector.write.PhysicalWriteInfoImpl
 import org.apache.spark.sql.execution.WholeStageCodegenExec
 import org.apache.spark.sql.execution.datasources.DataSource
 import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
-import 
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchScan, 
ColumnarMicroBatchStream, ColumnarScan, ColumnarStreamOffset}
+import 
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchScan, 
ColumnarBatchWrite, ColumnarMicroBatchStream, ColumnarScan, 
ColumnarScanBuilder, ColumnarStreamingWrite, ColumnarStreamOffset}
 import 
org.apache.spark.sql.execution.datasources.v2.python.{PythonDataSourceV2, 
UserDefinedPythonDataSource}
 import org.apache.spark.sql.execution.vectorized.ReusedArrowColumnVector
 import org.apache.spark.sql.functions.lit
@@ -53,7 +55,8 @@ import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
 import org.apache.spark.sql.streaming.{StreamingQuery, Trigger}
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types._
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.util.{ArrowUtils, CaseInsensitiveStringMap}
+import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch}
 import org.apache.spark.util.{ThreadUtils, Utils}
 
 /**
@@ -106,9 +109,11 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     session
   }
 
+  // The test libraries record the calls that the order of the calls of the 
header rules out,
+  // which fail the test that made them.
   private def nativeTest(name: String)(f: => Unit): Unit = test(name) {
     assume(toolchain.isDefined, "A C++ compiler and Java 22 or later are 
required.")
-    try f finally SparkSession.setActiveSession(spark)
+    try checkingOrderOfCalls(f) finally SparkSession.setActiveSession(spark)
   }
 
   // Checks the answer without converting the DataFrame to an RDD: a job 
outside of a SQL
@@ -775,14 +780,87 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
         "action" -> "write to",
         "name" -> "native_sink",
         "msg" -> "commit_data_writer returned 2 values instead of one."))
-    // The task failed without aborting the committed data writer, which it 
closed, and the driver
-    // aborted the write.
-    assert(!new File(dir, "_ABORTED_COMMITTED_DATA_WRITER").exists())
+    // The task failed without aborting the committed data writer, which the 
test library would
+    // record as a violation of the order of the calls, and closed it. The 
driver aborted the write.
     checkDataWritersClosed(dir)
     assert(Files.readString(new File(dir, "_ABORTED").toPath) == "1,1,0")
     assert(!dir.listFiles().exists(_.getName.startsWith("part-")))
   }
 
+  nativeTest("keep the order of the calls whatever the callers of the adapter 
do") {
+    val session = newSession(defaultPackage)
+    def dataSource(name: String, options: (String, String)*): NativeDataSource 
= {
+      val (resolved, location) =
+        NativeDataSourceRegistry.lookup(name, session.sessionState.conf).get
+      new NativeDataSource(location, resolved, new 
CaseInsensitiveStringMap(options.toMap.asJava))
+    }
+    val schema = StructType.fromDDL("id BIGINT")
+    // The test library records each call that the header rules out, which 
fails the test.
+
+    // A write or a micro-batch that committed is not aborted.
+    val batchDir = Utils.createTempDir()
+    val batchWrite = new ColumnarBatchWrite(
+      () => dataSource("native_sink", "path" -> 
batchDir.getPath).writer(schema, false), schema)
+    batchWrite.createBatchWriterFactory(PhysicalWriteInfoImpl(1))
+    batchWrite.commit(Array.empty)
+    batchWrite.abort(Array.empty)
+    assert(new File(batchDir, "_SUCCESS").isFile && !new File(batchDir, 
"_ABORTED").exists())
+    val streamDir = Utils.createTempDir()
+    val streamWrite = new ColumnarStreamingWrite("native_sink",
+      () => dataSource("native_sink", "path" -> streamDir.getPath)
+        .streamWriter(schema, false, UUID.randomUUID().toString),
+      schema)
+    streamWrite.createStreamingWriterFactory(PhysicalWriteInfoImpl(1))
+    streamWrite.commit(0, Array.empty)
+    streamWrite.abort(0, Array.empty)
+    assert(new File(streamDir, "_EPOCH_0").isFile && !new File(streamDir, 
"_ABORTED_0").exists())
+
+    // After a write fails, a data writer only aborts and closes the data 
writer of the library.
+    val failingDir = Utils.createTempDir()
+    val writer = dataSource("native_sink", "path" -> failingDir.getPath, 
"fail" -> "write")
+      .writer(schema, false)
+    val dataWriter = writer.createWriter(0, 0)
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("test", 0, 
Long.MaxValue)
+    val vector = new BigIntVector("id", allocator)
+    try {
+      vector.allocateNew(1)
+      vector.set(0, 1L)
+      vector.setValueCount(1)
+      val batch = new ColumnarBatch(Array(new ArrowColumnVector(vector)), 1)
+      val writeError = intercept[Exception](dataWriter.write(batch))
+      assert(findError(writeError, 
"NATIVE_DATA_SOURCE_ERROR").getMessageParameters
+        .get("msg") == "injected failure in write")
+      Seq(() => dataWriter.write(batch), () => dataWriter.commit()).foreach { 
call =>
+        checkError(
+          exception = intercept[SparkException](call()),
+          condition = "INTERNAL_ERROR",
+          parameters = Map("message" -> "A write or a commit of the data 
writer failed."))
+      }
+      dataWriter.abort()
+    } finally {
+      dataWriter.close()
+      vector.close()
+      allocator.close()
+      writer.abort(Array.empty)
+    }
+
+    // The reader of a scan prunes its columns once.
+    val rangeSource = dataSource("native_range")
+    val builder = new ColumnarScanBuilder("native_range", 
rangeSource.schema(), () => rangeSource)
+    builder.pruneColumns(StructType.fromDDL("id BIGINT"))
+    builder.pruneColumns(StructType.fromDDL("id BIGINT"))
+    checkError(
+      exception = intercept[SparkException] {
+        builder.pruneColumns(StructType.fromDDL("name STRING"))
+      },
+      condition = "INTERNAL_ERROR",
+      parameters = Map(
+        "message" ->
+          "The columns of a scan were pruned again, with columns that it does 
not read."))
+    // Plans the scan, which releases the reader.
+    builder.build().toBatch.planInputPartitions()
+  }
+
   nativeTest("abort a micro-batch whose write failed") {
     val session = newSession(defaultPackage)
     val dir = Utils.createTempDir()
@@ -1235,6 +1313,38 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     assert(new File(streamDir, "_CLOSED").isFile)
   }
 
+  nativeTest("call no other function on a reader whose call failed than its 
close function") {
+    def checkFailure(step: String)(f: => Unit): Unit = {
+      val e = intercept[Exception](f)
+      assert(findError(e, 
"NATIVE_DATA_SOURCE_ERROR").getMessageParameters.get("msg") ==
+        s"injected failure in $step")
+    }
+    // The test library records the calls after a failed one, which fail the 
test.
+    Seq("push_limit" -> ((df: DataFrame) => df.limit(1)),
+        "prune_columns" -> ((df: DataFrame) => df.select("id"))).foreach { 
case (step, query) =>
+      checkFailure(step) {
+        query(range(newSession(defaultPackage), "fail" -> step)).collect()
+      }
+    }
+    // A stream reader is closed after one of its functions failed, when the 
query stops. The
+    // offsets of the first micro-batch are committed when the second one runs.
+    Seq("initial_offset", "latest_offset", "stream_partitions", 
"commit_offset").foreach { step =>
+      val closedPath = new File(Utils.createTempDir(), "closed")
+      checkFailure(step) {
+        val query = 
newSession(defaultPackage).readStream.format("native_counter")
+          .option("max_offset", "12").option("step", "5").option("fail", step)
+          .option("closed_path", closedPath.getPath)
+          .load()
+          .writeStream
+          .format("noop")
+          .option("checkpointLocation", Utils.createTempDir().getPath)
+          .start()
+        try query.processAllAvailable() finally query.stop()
+      }
+      assert(closedPath.isFile, s"the stream reader is not closed after $step 
failed")
+    }
+  }
+
   nativeTest("read and write without a state") {
     // A reader or a writer without a state does not call set_state. read then 
fails unless it
     // gets a state of length 0, and takes the columns from the schema to 
read. A data writer that
@@ -1480,15 +1590,46 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
           "interface")))
 
     val logAppender = new LogAppender("native data source logs")
+    // The library records each call of log, with its level, in a file.
+    val logCalls = new File(Utils.createTempDir(), "log_calls")
     withLogAppender(logAppender, loggerNames = 
Seq(classOf[NativeLibrary].getName)) {
-      range(session, "log" -> "a message from native code")
+      range(session, "log" -> "a message from native code", "log_calls_path" 
-> logCalls.getPath)
     }
+    // Each data source that the read creates logs, at SPARK_DS_LOG_WARN.
+    assert(Files.readAllLines(logCalls.toPath).asScala.toSet == Set("2"))
     assert(logAppender.loggingEvents.exists { event =>
       val message = event.getMessage.getFormattedMessage
       event.getLevel == org.apache.logging.log4j.Level.WARN &&
         message.contains("a message from native code") &&
         message.contains(System.mapLibraryName("test_native_datasource"))
     })
+    // The library does not even call log for a warning when Spark logs errors 
only.
+    val errorAppender = new LogAppender("native data source error logs")
+    val errorLogCalls = new File(Utils.createTempDir(), "log_calls")
+    withLogAppender(errorAppender, loggerNames = 
Seq(classOf[NativeLibrary].getName),
+        level = Some(org.apache.logging.log4j.Level.ERROR)) {
+      range(session, "log" -> "a message that Spark does not log",
+        "log_calls_path" -> errorLogCalls.getPath)
+    }
+    assert(!errorLogCalls.exists())
+    assert(!errorAppender.loggingEvents.exists(
+      _.getMessage.getFormattedMessage.contains("a message that Spark does not 
log")))
+
+    // is_log_enabled follows the level of the log of Spark: the library gets 
whether Spark logs
+    // each level, from error to trace.
+    val levels = new File(Utils.createTempDir(), "levels")
+    Seq(
+      org.apache.logging.log4j.Level.TRACE -> "1,1,1,1,1",
+      org.apache.logging.log4j.Level.DEBUG -> "1,1,1,1,0",
+      org.apache.logging.log4j.Level.INFO -> "1,1,1,0,0",
+      org.apache.logging.log4j.Level.WARN -> "1,1,0,0,0",
+      org.apache.logging.log4j.Level.ERROR -> "1,0,0,0,0").foreach { case 
(level, expected) =>
+      withLogAppender(new LogAppender("native data source log levels"),
+          loggerNames = Seq(classOf[NativeLibrary].getName), level = 
Some(level)) {
+        range(session, "log_levels_path" -> levels.getPath)
+      }
+      assert(Files.readString(levels.toPath) == expected, level)
+    }
   }
 
   nativeTest("pass the task to reads and data writers") {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceTestUtils.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceTestUtils.scala
index 938f838c6f6b..71538a29a92a 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceTestUtils.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceTestUtils.scala
@@ -23,6 +23,12 @@ import java.util.concurrent.ConcurrentHashMap
 import java.util.concurrent.atomic.AtomicInteger
 import java.util.zip.{ZipEntry, ZipOutputStream}
 
+import scala.util.control.NonFatal
+
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.SparkContext
 import org.apache.spark.util.Utils
 
 /**
@@ -37,6 +43,51 @@ object NativeDataSourceTestUtils {
 
   private val libraries = new ConcurrentHashMap[String, File]()
 
+  /**
+   * The file to which the test libraries append the calls that the order of 
the calls of the
+   * header rules out, one per line. It does not exist as long as there are 
none.
+   */
+  lazy val orderViolations: File = new File(buildDir, "order_violations")
+
+  /**
+   * Runs a test that uses the test libraries, and fails it if the libraries 
got calls that the
+   * order of the calls of the header rules out. It waits until no task runs 
before it checks, so
+   * that the late calls of the tasks of the test, such as of a task of a 
failed job that is still
+   * being killed, fail the test that made them. If the test fails, the calls 
are added to its
+   * error as suppressed.
+   */
+  def checkingOrderOfCalls[T](f: => T): T = {
+    orderViolations.delete()
+    val result = try f catch {
+      case e: Throwable =>
+        try {
+          violations().foreach(v => e.addSuppressed(new AssertionError(v)))
+        } catch {
+          case NonFatal(check) => e.addSuppressed(check)
+        }
+        throw e
+    }
+    violations().foreach(org.scalatest.Assertions.fail(_))
+    result
+  }
+
+  // The calls that the test libraries recorded as violations of the order of 
the calls, once no
+  // task runs.
+  private def violations(): Option[String] = {
+    SparkContext.getActive.foreach { sc =>
+      eventually(timeout(1.minute), interval(10.milliseconds)) {
+        assert(sc.statusTracker.getExecutorInfos.forall(_.numRunningTasks() == 
0))
+      }
+    }
+    if (orderViolations.exists()) {
+      val violations = Files.readString(orderViolations.toPath)
+      orderViolations.delete()
+      Some(s"The test library got calls that the header rules 
out:\n$violations")
+    } else {
+      None
+    }
+  }
+
   // The executors store the files of the sessions that are not isolated in 
the same directory, so
   // each package has its own name.
   private val packageCount = new AtomicInteger()
@@ -75,7 +126,9 @@ object NativeDataSourceTestUtils {
     val library = new File(buildDir, System.mapLibraryName(name))
     // The symbols are hidden, as libraries are often built, so only 
spark_ds_init is exported.
     run(Seq(compiler, "-std=c++17", "-O1", "-shared", "-fPIC", 
"-fvisibility=hidden") ++
-      includes.map("-I" + _) ++ defines ++ Seq("-o", library.getPath, 
source.getPath))
+      includes.map("-I" + _) ++ defines ++
+      Seq(s"-DORDER_VIOLATIONS_PATH=\"${orderViolations.getPath}\"", "-o", 
library.getPath,
+        source.getPath))
     library
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to