This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new d96cafba3b16 [SPARK-60725][SQL] Write the columnar batches of a query 
to native data sources without converting them to rows
d96cafba3b16 is described below

commit d96cafba3b1620ca0c7f0891fc089b60070f1b2e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:38:24 2026 +0900

    [SPARK-60725][SQL] Write the columnar batches of a query to native data 
sources without converting them to rows
    
    ### What changes were proposed in this pull request?
    
    A follow-up to SPARK-60101 (#59424), discussed there with zhouyuan and 
zhztheplayer. When the query of a batch or micro-batch write to a native data 
source produces columnar batches, the data writers now get the batches. Before, 
a `ColumnarToRowExec` converted them to rows first. A scan of a native data 
source is one such query. This PR is based on #59931 (version 1 of the 
interface), whose commit it includes until that PR is merged.
    
    - **Planning and execution.** An internal mixin, `SupportsColumnarInput` in 
`org.apache.spark.sql.internal.connector`, marks:
      - a `Write`, for appends and overwrites;
      - the `StreamingWrite` of a micro-batch.
    
      Its data writers are `ColumnarInputDataWriter`s, which also take columnar 
batches.
      - `V2TableWriteExec.writesColumnar` tells whether a write node writes the 
batches of its query: when its data writers take them 
(`supportsColumnarInput`), and the query only supports columnar output. 
Row-level operations do not: their tasks project the rows.
      - `ApplyColumnarRulesAndInsertTransitions` then inserts no transition to 
rows above a columnar query, as it already does for planned V1 writes.
      - `writeWithV2` runs `runColumnar` of the writing task, a method of 
`WritingSparkTask` that only `DataWritingSparkTask` implements, and that shares 
the commit protocol of `WritingSparkTask` (moved into `writeAndCommit`). It 
releases each batch with `closeIfFreeable` once written, as `ColumnarToRowExec` 
does.
    - **Copying the batches.** The data writer of the columnar adapter 
(`ArrowBatchingDataWriter`) decides for each column. It copies an Arrow vector 
that is physically congruent with the Arrow schema of the write 
(`ArrowUtils.isCompatibleWithDeclaredField`), as the vectors of a native scan 
are, and that Arrow Java can slice (`BaseSliceArrowOutputProcessor.canSlice`, 
recursively), buffer by buffer: each buffer of a slice of rows is copied once, 
at its size, through `VectorUnloader` and `V [...]
      - The native data writer may move the arrays that it gets, to keep them 
after the call.
      - The producer of a `ColumnarBatch` may overwrite its vectors once the 
batch is consumed, as `ArrowWriter.reset()` does.
    
      Arrow vectors are sliced before they are copied, because Arrow Java reads 
the offsets of a vector as if they started at 0, which they need not in an 
array imported through the C data interface, and do in a slice.
    
    The batches only reach the write when the query only supports columnar 
output, as for a plain copy from one native data source to another. Spark 
inserts a conversion to rows before a filter, a projection or a shuffle, which 
are not columnar, with or without adaptive query execution. With adaptive query 
execution, the query of a write with a subquery is also an 
`AdaptiveSparkPlanExec`, which is not columnar. A query that supports rows too, 
such as the scan of a cached DataFrame, is wri [...]
    
    ### Why are the changes needed?
    
    The batches of a native scan were converted to rows by `ColumnarToRowExec`, 
and then back to Arrow by the data writer. With this change, a copy from one 
native data source to another stays in Arrow, at the cost of a copy of the 
buffers. A native engine can also write its batches to a native sink without 
converting them to rows.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, for the data writers of native data sources, when Spark copies the 
Arrow batches of a scan:
    - The copied values are written as the scan returned them, without the 
checks of a conversion. For example, a decimal that does not fit the precision 
of the schema is now written, while the conversion to rows failed with 
`NUMERIC_VALUE_OUT_OF_RANGE` before.
    - The batches end with the batches of the scan, so a library may get 
batches with fewer rows than `spark.sql.execution.arrow.maxRecordsPerBatch`, as 
the header and the guide now say.
    
    Otherwise the written data is the same. The plan of such a write no longer 
has a `ColumnarToRow` node.
    
    ### How was this patch tested?
    
    New tests in `ColumnarDataSourceSuite`:
    - "write the columnar batches of a query" checks the plan and the batch 
sizes for on-heap and Arrow batches. It also checks a data writer that keeps 
the data of its batches, through the C data interface, while the scan 
overwrites its vectors with each batch, and that a cached DataFrame, whose scan 
supports rows too, is written as rows.
    - "copy the Arrow columns of every type, or convert the columns of a batch" 
covers every type that Spark converts to Arrow, nulls, slices at offsets that 
are not aligned to bytes, the limits in rows and in bytes, Arrow types other 
than those of the schema, and on-heap vectors. Each case of Arrow vectors 
checks the data that the data writer kept after the producer overwrote the 
batch; the on-heap vectors are not overwritten.
    - "write the pending rows of converted batches before a copied batch".
    - "a copied batch ends at the byte limit as a batch of rows does" compares 
the batches and the rows with those of the row path, at limits around the size 
of 2 rows, for copied columns and for converted ones.
    - "convert the columns of a batch that are not copied up to the end of each 
batch" writes 20 rows of 1 MiB strings that are converted, with copied ids, in 
batches of 2 MiB: it counts the conversions, and checks that the peak memory of 
the writer stays below that of 5 rows, while converting the 10 rows of 
`maxRecordsPerBatch` at once peaks at 24 MiB.
    - "copy only the Arrow columns that Arrow can slice" checks that a 
fixed-width vector of 2 GiB, and a list with such a child, are converted, with 
vectors that report more values than they hold.
    - "an empty columnar batch does not end the pending batch of rows".
    - "release the converted columns when a later column fails to convert".
    
    These tests also check that the writer holds no memory once it is closed, 
so that a leak, which the writer only logs, fails them.
    
    The plans of the batch and streaming writes of `NativeDataSourceSuite`, and 
of "streaming read and write" in `ColumnarDataSourceSuite`, are now also 
checked.
    
    Replacing the copy with the slices of the batch makes the tests of kept 
data fail, and converting `maxRecordsPerBatch` rows at once, or copying vectors 
that Arrow cannot slice, makes the tests of the peak memory and of `canCopy` 
fail. `PythonArrowOutputSuite`, `DataSourceV2Suite`, 
`SparkSessionExtensionSuite`, `ColumnarRulesSuite`, 
`DeltaBasedUpdateTableSuite`, `DeltaBasedMergeIntoTableSuite`, 
`StreamingDataSourceV2Suite` and `V1WriteFallbackSuite` pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59933 from HyukjinKwon/native-ds-columnar-writes.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 docs/sql-native-data-sources.md                    |  27 +-
 .../internal/connector/SupportsColumnarInput.scala |  43 ++
 .../apache/spark/sql/datasource/spark_datasource.h |  47 +-
 .../org/apache/spark/sql/execution/Columnar.scala  |   6 +
 .../datasources/v2/WriteToDataSourceV2Exec.scala   | 115 ++-
 .../v2/columnar/ColumnarDataSource.scala           |  11 +-
 .../datasources/v2/columnar/ColumnarWrite.scala    | 337 ++++++++-
 .../sql/execution/python/PythonArrowOutput.scala   |   5 +-
 .../v2/columnar/ColumnarDataSourceSuite.scala      | 783 ++++++++++++++++++++-
 .../datasources/v2/ffi/NativeDataSourceSuite.scala |  18 +-
 10 files changed, 1312 insertions(+), 80 deletions(-)

diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index 126882e81435..03b5eae4ab0c 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -37,7 +37,19 @@ package, and loads it when a query uses the data source. 
Spark adapts it to
 
 Data is exchanged only as columnar batches, through the
 [Arrow C data 
interface](https://arrow.apache.org/docs/format/CDataInterface.html), so Spark 
reads
-the data that a library returns without copying it.
+the data that a library returns without copying it. When a write gets the 
batches of a scan as
+they are, as in a plain copy from one native data source to another, Spark 
writes them without
+converting them to rows: it copies the buffers of their columns, since a 
library may keep the data
+that it writes until it closes its data writer, and converts value by value 
the columns of other
+Arrow types than those of the schema, such as `utf8_view` strings, and those 
that Arrow Java cannot
+slice, such as a column of a fixed width with 2 GiB of values or more. The 
copied values are written
+as the scan returned them, without the checks of a conversion, such as the 
precision of a decimal.
+The batches that Spark copies end with the batches of the scan, so a library 
may get smaller
+batches than `spark.sql.execution.arrow.maxRecordsPerBatch` rows then. The 
data of a query in which
+Spark also filters, projects or shuffles the rows is still converted to rows, 
and with adaptive
+query execution, so is the data of a query with a subquery. So is the data of 
every query when an
+extension of the session (`spark.sql.extensions`) has columnar rules, which 
may replace the
+conversions to rows with their own, such as those of an engine that runs 
queries in native code.
 
 Spark calls native data sources with the
 [Foreign Function and Memory API](https://openjdk.org/jeps/454) of Java, which 
is final from Java
@@ -249,12 +261,13 @@ Spark does not manage the memory of native data sources. 
The memory that a libra
 the Arrow buffers that it exports, which Spark reads without copying them, are 
not accounted by the
 memory manager of Spark, nor limited by `spark.memory.offHeap.size`. Neither 
are the Arrow batches
 that Spark writes, which end at `spark.sql.execution.arrow.maxRecordsPerBatch` 
rows or once they
-hold `spark.sql.execution.arrow.maxBytesPerBatch` bytes, while the size of the 
batches that a
-library reads is up to the library. This memory comes out of the memory 
overhead of the driver and
-the executors, `spark.driver.memoryOverhead` and 
`spark.executor.memoryOverhead`, which are 10% of
-their memory, and at least 384 MiB, by default. Increase it for libraries that 
use much memory, and
-bound the memory that a library uses, for example with the size of its 
batches: on YARN or
-Kubernetes, an executor that uses more memory than it requested is killed.
+hold `spark.sql.execution.arrow.maxBytesPerBatch` bytes, or earlier when Spark 
copies the batches of
+a scan, while the size of the batches that a library reads is up to the 
library. This memory comes
+out of the memory overhead of the driver and the executors, 
`spark.driver.memoryOverhead` and
+`spark.executor.memoryOverhead`, which are 10% of their memory, and at least 
384 MiB, by default.
+Increase it for libraries that use much memory, and bound the memory that a 
library uses, for
+example with the size of its batches: on YARN or Kubernetes, an executor that 
uses more memory than
+it requested is killed.
 
 ## Finding Native Data Sources
 
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/SupportsColumnarInput.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/SupportsColumnarInput.scala
new file mode 100644
index 000000000000..2da0e9d5eef0
--- /dev/null
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/SupportsColumnarInput.scala
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.internal.connector
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.connector.write.DataWriter
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+/**
+ * An internal mixin for a `Write`, or for the `StreamingWrite` of a 
micro-batch write, whose data
+ * writers are [[ColumnarInputDataWriter]]s. When the query of an append or an 
overwrite, or of a
+ * micro-batch, only produces columnar batches, and the session has no 
columnar rules of
+ * extensions, Spark passes the batches to the data writers instead of 
converting them to rows
+ * first. Otherwise, the data writers get rows: for example for the scan of a 
cached DataFrame,
+ * which supports rows too, or for a query that adaptive query execution wraps 
(see
+ * `V2TableWriteExec.writesColumnar`). It is internal: only the columnar 
adapter of Spark, which the
+ * native data sources use, implements it.
+ */
+trait SupportsColumnarInput
+
+/**
+ * A data writer that also takes the data to write as columnar batches, see
+ * [[SupportsColumnarInput]]. A task writes either rows or batches. Like a 
row, a batch is only
+ * valid during the call: its producer may reuse it for the next batch.
+ */
+trait ColumnarInputDataWriter extends DataWriter[InternalRow] {
+  def writeBatch(batch: ColumnarBatch): Unit
+}
diff --git 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
index a4d98446c066..1aa4eb346885 100644
--- 
a/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
+++ 
b/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
@@ -81,26 +81,27 @@
  *   the tasks of the write, so a writer may still be aborted because 
computing the data failed: do
  *   irreversible work, such as removing the existing data for an overwrite, 
in commit. On the
  *   executors, each task creates a data writer with create_data_writer, with 
the schema of the data
- *   and the state of the writer, calls write for each batch of at most
- *   spark.sql.execution.arrow.maxRecordsPerBatch rows, which also ends once 
it holds
- *   spark.sql.execution.arrow.maxBytesPerBatch bytes, and calls 
commit_data_writer, or
- *   abort_data_writer if the task or the commit fails before 
commit_data_writer succeeded, before
- *   it closes the data writer. Then the driver calls commit with the messages 
of the tasks if they
- *   all succeeded, or abort otherwise, including when commit fails, and 
closes the writer. Several
- *   attempts of a task may write the same partition, after a failure or with 
speculative execution,
- *   each with a data writer of its own. Spark lets the attempts of a 
partition commit one at a
- *   time, and passes to commit the message of one committed attempt for each 
partition, so the
- *   write consists of the data of these attempts only. It aborts the data 
writers of the other
- *   attempts, except two: one whose commit_data_writer succeeded before its 
task failed, which is
- *   not aborted, and one on an executor that is lost, which is neither 
committed nor aborted, and
- *   may even have committed before its message reached the driver. Neither is 
passed to commit, so
- *   the commit or the abort of the writer cleans up the output of the data 
writers whose messages
- *   it does not get. A streaming write commits or aborts each micro-batch 
(epoch). Spark may commit
- *   the same epoch more than once, for example when a streaming query 
restarts after a failure, so
- *   committing an epoch that is already committed must have no effect, for 
exactly-once output. A
- *   restarted query may also abort an epoch that an earlier run of the query 
committed, so an abort
- *   must only remove what the data writers of its own writer wrote, not the 
output that an earlier
- *   writer committed. The ID of the query, which create_writer gets for a 
streaming write, stays
+ *   and the state of the writer, and calls write for each batch. A batch has 
at most
+ *   spark.sql.execution.arrow.maxRecordsPerBatch rows, and also ends once it 
holds
+ *   spark.sql.execution.arrow.maxBytesPerBatch bytes, or earlier when Spark 
copies the batches of a
+ *   scan, which end with them. The task then calls commit_data_writer, or 
abort_data_writer if the
+ *   task or the commit fails before commit_data_writer succeeded, before it 
closes the data writer.
+ *   Then the driver calls commit with the messages of the tasks if they all 
succeeded, or abort
+ *   otherwise, including when commit fails, and closes the writer. Several 
attempts of a task may
+ *   write the same partition, after a failure or with speculative execution, 
each with a data
+ *   writer of its own. Spark lets the attempts of a partition commit one at a 
time, and passes to
+ *   commit the message of one committed attempt for each partition, so the 
write consists of the
+ *   data of these attempts only. It aborts the data writers of the other 
attempts, except two: one
+ *   whose commit_data_writer succeeded before its task failed, which is not 
aborted, and one on an
+ *   executor that is lost, which is neither committed nor aborted, and may 
even have committed
+ *   before its message reached the driver. Neither is passed to commit, so 
the commit or the abort
+ *   of the writer cleans up the output of the data writers whose messages it 
does not get. A
+ *   streaming write commits or aborts each micro-batch (epoch). Spark may 
commit the same epoch
+ *   more than once, for example when a streaming query restarts after a 
failure, so committing an
+ *   epoch that is already committed must have no effect, for exactly-once 
output. A restarted query
+ *   may also abort an epoch that an earlier run of the query committed, so an 
abort must only
+ *   remove what the data writers of its own writer wrote, not the output that 
an earlier writer
+ *   committed. The ID of the query, which create_writer gets for a streaming 
write, stays
  *   the same when the query restarts from its checkpoint, so the ID of the 
query and the epoch ID
  *   identify the output of a micro-batch, also when other queries write to 
the same destination. A
  *   micro-batch streaming query creates a new stream writer for each 
micro-batch, and Spark closes
@@ -762,7 +763,11 @@ typedef struct SparkDsLibrary {
    * Writes a batch of data.
    *
    * array: a struct array whose fields are the columns, with the schema that 
create_data_writer
-   *        got
+   *        got. The library may move it to keep its data, until 
close_data_writer: its buffers
+   *        belong to the data writer, so the library must release it before 
close_data_writer
+   *        returns. When Spark writes the batches of a native scan without 
converting them to
+   *        rows, it copies the values of their columns as the scan returned 
them, without the
+   *        checks of a conversion, such as the precision of a decimal.
    */
   int (*write)(void* data_writer, struct ArrowArray* array);
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
index 7055fcc631e5..121011e3becd 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.errors.ExecutionErrors
 import org.apache.spark.sql.execution.command.DataWritingCommandExec
 import org.apache.spark.sql.execution.datasources.V1WriteCommand
+import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
 import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
 import org.apache.spark.sql.execution.vectorized.WritableColumnVector
 import org.apache.spark.sql.types._
@@ -589,6 +590,11 @@ case class ApplyColumnarRulesAndInsertTransitions(
         case write: DataWritingCommandExec
             if write.cmd.isInstanceOf[V1WriteCommand] && 
conf.plannedWriteEnabled =>
           write.child.supportsColumnar
+        // A V2 write whose data writers take columnar batches writes those of 
a query that only
+        // supports columnar output, unless an extension has columnar rules: 
they may replace the
+        // transitions to rows with their own, which may be the only way to 
read their batches, so
+        // the write then gets rows through such a transition, as other writes 
do.
+        case write: V2TableWriteExec if write.writesColumnar && 
columnarRules.isEmpty => true
         // If it is not required to output columnar (`outputsColumnar` is 
false), and the plan
         // supports row-based and columnar, we don't need to output row-based 
data on its children
         // nodes. So we set `outputsColumnar` to true.
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
index 4b1dbfa775f2..fb686ca8701e 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
@@ -39,9 +39,12 @@ import org.apache.spark.sql.errors.{QueryCompilationErrors, 
QueryExecutionErrors
 import org.apache.spark.sql.execution.{EmptyRDDWithPartitions, QueryExecution, 
SparkPlan, SQLExecution, UnaryExecNode}
 import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
 import org.apache.spark.sql.execution.metric.{CustomMetrics, 
SQLLastAttemptMetric, SQLLastAttemptMetrics, SQLMetric, SQLMetrics}
+import org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite
+import org.apache.spark.sql.internal.connector.{ColumnarInputDataWriter, 
SupportsColumnarInput}
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.util.CaseInsensitiveStringMap
 import org.apache.spark.sql.util.SchemaValidationMode.PROHIBIT_CHANGES
+import org.apache.spark.sql.vectorized.ColumnarBatch
 import org.apache.spark.util.ArrayImplicits._
 import org.apache.spark.util.Utils
 
@@ -476,6 +479,12 @@ case class WriteToDataSourceV2Exec(
   override lazy val customMetrics: Map[String, SQLMetric] =
     createCustomMetrics(writeMetrics.toArray)
 
+  override def supportsColumnarInput: Boolean = batchWrite match {
+    case microBatchWrite: MicroBatchWrite =>
+      microBatchWrite.writeSupport.isInstanceOf[SupportsColumnarInput]
+    case _ => false
+  }
+
   override protected def run(): Seq[InternalRow] = {
     val writtenRows = writeWithV2(batchWrite)
     transaction.foreach(TransactionUtils.commit)
@@ -508,6 +517,8 @@ trait V2ExistingTableWriteExec extends V2TableWriteExec 
with TransactionalExec {
   override lazy val customMetrics: Map[String, SQLMetric] =
     createCustomMetrics(write.supportedCustomMetrics())
 
+  override def supportsColumnarInput: Boolean = 
write.isInstanceOf[SupportsColumnarInput]
+
   override protected def run(): Seq[InternalRow] = {
     val writtenRows = try {
       writeWithV2(write.toBatch)
@@ -526,6 +537,9 @@ trait V2ExistingTableWriteExec extends V2TableWriteExec 
with TransactionalExec {
 trait RowLevelWriteExec extends V2ExistingTableWriteExec {
   def rowLevelCommand: RowLevelOperation.Command
 
+  // Its task writes projections of the rows of the query.
+  override def supportsColumnarInput: Boolean = false
+
   override protected lazy val sparkMetrics: Map[String, SQLMetric] = 
super.sparkMetrics ++ (
     rowLevelCommand match {
       case UPDATE =>
@@ -605,6 +619,24 @@ trait V2TableWriteExec
   def query: SparkPlan
   def writingTask: WritingSparkTask[_] = DataWritingSparkTask
 
+  /**
+   * Whether the data writers take the columnar batches of the query, see
+   * [[SupportsColumnarInput]].
+   */
+  def supportsColumnarInput: Boolean = false
+
+  /**
+   * Whether the write takes the columnar batches of the query: when the data 
writers take them,
+   * and the query only supports columnar output. A query that supports rows 
too, such as a scan
+   * of a cached DataFrame, is written as rows. When the write takes the 
batches, the query gets no
+   * transition to rows (see `ApplyColumnarRulesAndInsertTransitions`), and 
its batches are written
+   * with `WritingSparkTask.runColumnar` of `writingTask`. With columnar rules 
of extensions, which
+   * may replace the transitions to rows with their own, the query gets a 
transition, so the write
+   * then takes rows.
+   */
+  def writesColumnar: Boolean =
+    supportsColumnarInput && query.supportsColumnar && !query.supportsRowBased
+
   var commitProgress: Option[StreamWriterCommitProgress] = None
 
   override def child: SparkPlan = query
@@ -619,8 +651,9 @@ trait V2TableWriteExec
     "numOutputRows" -> numOutputRowsMetric)
 
   protected def writeWithV2(batchWrite: BatchWrite): Seq[InternalRow] = {
-    val rdd: RDD[InternalRow] = {
-      val tempRdd = query.execute()
+    val columnar = writesColumnar
+    val rdd: RDD[_] = {
+      val tempRdd = if (columnar) query.executeColumnar() else query.execute()
       // SPARK-23271 If we are attempting to write a zero partition rdd, 
create a dummy single
       // partition rdd to make sure we at least set up one write task to write 
the metadata.
       if (tempRdd.partitions.length == 0) {
@@ -646,8 +679,13 @@ trait V2TableWriteExec
     try {
       sparkContext.runJob(
         rdd,
-        (context: TaskContext, iter: Iterator[InternalRow]) =>
-          task.run(writerFactory, context, iter, useCommitCoordinator, 
writeMetrics),
+        (context: TaskContext, iter: Iterator[Any]) => if (columnar) {
+          task.runColumnar(writerFactory, context,
+            iter.asInstanceOf[Iterator[ColumnarBatch]], useCommitCoordinator, 
writeMetrics)
+        } else {
+          task.run(writerFactory, context, 
iter.asInstanceOf[Iterator[InternalRow]],
+            useCommitCoordinator, writeMetrics)
+        },
         rdd.partitions.indices,
         (index, result: DataWritingSparkTaskResult) => {
           val commitMessage = result.writerCommitMessage
@@ -701,18 +739,49 @@ trait WritingSparkTask[W <: DataWriter[InternalRow]] 
extends Logging with Serial
       iter: Iterator[InternalRow],
       useCommitCoordinator: Boolean,
       customMetrics: Map[String, SQLMetric]): DataWritingSparkTaskResult = {
+    val dataWriter =
+      writerFactory.createWriter(context.partitionId(), 
context.taskAttemptId()).asInstanceOf[W]
+
+    val iterWithMetrics = IteratorWithMetrics(iter, dataWriter, customMetrics)
+
+    writeAndCommit(dataWriter, context, useCommitCoordinator, customMetrics) {
+      write(dataWriter, iterWithMetrics)
+      iterWithMetrics.count
+    }
+  }
+
+  /**
+   * Writes the columnar batches of a query, for a write whose data writers 
take them (see
+   * [[SupportsColumnarInput]]). Only `DataWritingSparkTask` implements it.
+   */
+  def runColumnar(
+      writerFactory: DataWriterFactory,
+      context: TaskContext,
+      iter: Iterator[ColumnarBatch],
+      useCommitCoordinator: Boolean,
+      customMetrics: Map[String, SQLMetric]): DataWritingSparkTaskResult = {
+    throw SparkException.internalError(s"${getClass.getName} does not write 
columnar batches.")
+  }
+
+  /**
+   * Writes the data of the task with `writeData`, which returns the number of 
rows that it wrote,
+   * and commits the data writer, once the output commit coordinator 
authorizes it if
+   * `useCommitCoordinator` is true. Aborts the data writer if either fails, 
and closes it.
+   */
+  protected def writeAndCommit(
+      dataWriter: DataWriter[_],
+      context: TaskContext,
+      useCommitCoordinator: Boolean,
+      customMetrics: Map[String, SQLMetric])(writeData: => Long): 
DataWritingSparkTaskResult = {
     val stageId = context.stageId()
     val stageAttempt = context.stageAttemptNumber()
     val partId = context.partitionId()
     val taskId = context.taskAttemptId()
     val attemptId = context.attemptNumber()
-    val dataWriter = writerFactory.createWriter(partId, taskId).asInstanceOf[W]
-
-    val iterWithMetrics = IteratorWithMetrics(iter, dataWriter, customMetrics)
 
     // write the data and commit this writer.
     Utils.tryWithSafeFinallyAndFailureCallbacks(block = {
-      write(dataWriter, iterWithMetrics)
+      val numRows = writeData
 
       CustomMetrics.updateMetrics(
         dataWriter.currentMetricsValues.toImmutableArraySeq, customMetrics)
@@ -748,7 +817,7 @@ trait WritingSparkTask[W <: DataWriter[InternalRow]] 
extends Logging with Serial
         log"stage ${MDC(LogKeys.STAGE_ID, stageId)}." +
         log"${MDC(LogKeys.STAGE_ATTEMPT_ID, stageAttempt)})")
 
-      DataWritingSparkTaskResult(iterWithMetrics.count, msg)
+      DataWritingSparkTaskResult(numRows, msg)
 
     })(catchBlock = {
       // If there is an error, abort this writer
@@ -873,6 +942,34 @@ object DataWritingSparkTask extends 
WritingSparkTask[DataWriter[InternalRow]] {
       writer: DataWriter[InternalRow], iter: java.util.Iterator[InternalRow]): 
Unit = {
     writer.writeAll(iter)
   }
+
+  override def runColumnar(
+      writerFactory: DataWriterFactory,
+      context: TaskContext,
+      iter: Iterator[ColumnarBatch],
+      useCommitCoordinator: Boolean,
+      customMetrics: Map[String, SQLMetric]): DataWritingSparkTaskResult = {
+    val dataWriter = writerFactory.createWriter(context.partitionId(), 
context.taskAttemptId())
+    writeAndCommit(dataWriter, context, useCommitCoordinator, customMetrics) {
+      val columnarWriter = dataWriter match {
+        case writer: ColumnarInputDataWriter => writer
+        case writer => throw SparkException.internalError(
+          s"Expected a data writer of columnar batches, but got 
${writer.getClass.getName}.")
+      }
+      var numRows = 0L
+      iter.foreach { batch =>
+        // Releases the batch once it is written, as ColumnarToRowExec does 
once it returned its
+        // rows: a producer of new column vectors for each batch leaves them 
to the consumer.
+        Utils.tryWithSafeFinally {
+          columnarWriter.writeBatch(batch)
+          numRows += batch.numRows()
+        }(batch.closeIfFreeable())
+        CustomMetrics.updateMetrics(
+          columnarWriter.currentMetricsValues.toImmutableArraySeq, 
customMetrics)
+      }
+      numRows
+    }
+  }
 }
 
 case class DeltaWritingSparkTask(
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
index 47f19d42aab2..d0e1fc430635 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSource.scala
@@ -147,7 +147,16 @@ trait ColumnarStreamReader extends Serializable {
  * Writes data to a [[ColumnarDataSource]] for a batch write. Spark serializes 
the writer to the
  * executors, where `createWriter` is called for each task, and then commits 
or aborts the write
  * on the driver. Spark does not abort a write whose commit succeeded. The 
batches passed to the
- * data writers are backed by Arrow, and are only valid during the call.
+ * data writers are backed by new Arrow vectors, with the Arrow types of the 
schema, except for the
+ * time zones of timestamps and the `keysSorted` flag of maps, though not 
necessarily with its
+ * names, nullability and metadata, nor the Spark types that 
`ArrowColumnVector` infers from them. A
+ * batch is only valid during the call, but nothing else writes to the buffers 
of its vectors, so a
+ * data writer may take ownership of them to keep its data, until it is 
closed: the buffers belong
+ * to the data writer, which must release them by the time its `close` 
returns. The columns that
+ * Spark copies from the Arrow batches of a scan have the values that the scan 
returned: they are
+ * not checked as converted values are, for example against the precision of a 
decimal. The batches
+ * have at most `spark.sql.execution.arrow.maxRecordsPerBatch` rows, and may 
have fewer, since the
+ * batches that Spark copies end with the batches of the scan.
  */
 trait ColumnarWriter extends Serializable {
   def createWriter(partitionId: Int, taskId: Long): DataWriter[ColumnarBatch]
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
index 60588c470843..eaf0c81216a8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarWrite.scala
@@ -16,18 +16,26 @@
  */
 package org.apache.spark.sql.execution.datasources.v2.columnar
 
+import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
 
-import org.apache.arrow.vector.VectorSchemaRoot
+import org.apache.arrow.memory.ArrowBuf
+import org.apache.arrow.util.AutoCloseables
+import org.apache.arrow.vector.{BaseFixedWidthVector, BaseVariableWidthVector, 
BitVector, BitVectorHelper, FieldVector, NullVector, ValueVector, VectorLoader, 
VectorSchemaRoot, VectorUnloader}
+import org.apache.arrow.vector.complex.{BaseRepeatedValueVector, ListVector, 
NonNullableStructVector, StructVector}
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch
+import org.apache.arrow.vector.types.pojo.Field
 
 import org.apache.spark.{SparkContext, SparkUnsupportedOperationException}
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.connector.metric.CustomTaskMetric
 import org.apache.spark.sql.connector.write._
 import 
org.apache.spark.sql.connector.write.streaming.{StreamingDataWriterFactory, 
StreamingWrite}
-import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.arrow.{ArrowFieldWriter, ArrowWriter}
+import org.apache.spark.sql.execution.python.BaseSliceArrowOutputProcessor
 import org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution
 import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.connector.{ColumnarInputDataWriter, 
SupportsColumnarInput}
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.util.ArrowUtils
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch, 
ColumnVector}
@@ -49,12 +57,17 @@ class ColumnarWriteBuilder(
   override def build(): Write = new ColumnarWrite(shortName, info, overwrite, 
createDataSource)
 }
 
+/**
+ * A write to a [[ColumnarDataSource]]. Its data writers also take the batches 
of a query that only
+ * produces columnar batches, in a session without columnar rules of 
extensions, which Spark then
+ * does not convert to rows (see [[SupportsColumnarInput]] and 
[[ArrowBatchingDataWriter]]).
+ */
 class ColumnarWrite(
     shortName: String,
     info: LogicalWriteInfo,
     overwrite: Boolean,
     createDataSource: () => ColumnarDataSource)
-  extends Write {
+  extends Write with SupportsColumnarInput {
 
   override def description(): String = shortName
 
@@ -118,7 +131,7 @@ class ColumnarStreamingWrite(
     description: String,
     createWriter: () => ColumnarStreamWriter,
     schema: StructType)
-  extends StreamingWrite {
+  extends StreamingWrite with SupportsColumnarInput {
 
   private var writer: ColumnarStreamWriter = _
   // Whether the writer committed the micro-batch, which is not aborted then, 
as for a batch write.
@@ -172,7 +185,9 @@ class ColumnarStreamingDataWriterFactory(
 
 /**
  * How [[ArrowBatchingDataWriter]] converts rows to Arrow, captured on the 
driver from the
- * session configuration. Like the other Arrow batches of Spark, a batch ends 
when it reaches
+ * session configuration, such as the session time zone of the timestamps that 
it converts. The
+ * timestamps that it copies from the Arrow batches of a scan keep the time 
zone of the scan. Like
+ * the other Arrow batches of Spark, a batch ends when it reaches
  * `spark.sql.execution.arrow.maxRecordsPerBatch` rows or, after a row, holds 
at least
  * `spark.sql.execution.arrow.maxBytesPerBatch` bytes.
  */
@@ -195,17 +210,29 @@ object ArrowBatchingOptions {
 }
 
 /**
- * Converts the rows of a write task to Arrow-backed [[ColumnarBatch]]es for 
the [[DataWriter]]
- * of a [[ColumnarDataSource]]. A new Arrow vector schema root is created for 
each batch, so a
- * native writer that took ownership of the data of a batch can keep it after 
the call.
+ * Converts the data of a write task to Arrow-backed [[ColumnarBatch]]es for 
the [[DataWriter]]
+ * of a [[ColumnarDataSource]]: its rows, or the columnar batches of a 
columnar query. Each batch
+ * gets new Arrow vectors, so a native writer that took ownership of the data 
of a batch can keep
+ * it after the call, until the data writer is closed, which closes the 
allocator of the vectors,
+ * while the producer of a columnar batch may reuse its vectors once the call 
returns.
+ *
+ * Each column of a columnar batch that is an Arrow vector in the layout of 
the Arrow schema of the
+ * write, and that Arrow can slice, is copied buffer by buffer, with its 
values as they are, which
+ * are not checked as the converted ones are, and the other columns are 
converted value by value,
+ * row by row up to the end of each batch, so that a conversion does not grow 
past the size of a
+ * batch. The batches end as those of rows do, though a batch with copied 
columns also ends with
+ * the columnar batch, so small columnar batches give small batches, which are 
not merged. A
+ * columnar batch without columns to copy is converted as rows are.
  */
 class ArrowBatchingDataWriter(delegate: DataWriter[ColumnarBatch], options: 
ArrowBatchingOptions)
-  extends DataWriter[InternalRow] {
+  extends ColumnarInputDataWriter {
 
   private val arrowSchema = ArrowUtils.toArrowSchema(
     options.schema, options.timeZoneId, errorOnDuplicatedFieldNames = true, 
largeVarTypes = false)
-  private val allocator =
+  // The allocator of the vectors of the batches, which holds nothing once the 
writer is closed.
+  private[columnar] val allocator =
     ArrowUtils.rootAllocator.newChildAllocator("columnar data source writer", 
0, Long.MaxValue)
+  // The batch of rows being written, created on its first row.
   private var root: VectorSchemaRoot = _
   private var arrowWriter: ArrowWriter = _
   private var numRows = 0
@@ -224,15 +251,240 @@ class ArrowBatchingDataWriter(delegate: 
DataWriter[ColumnarBatch], options: Arro
     }
   }
 
-  private def flush(): Unit = {
-    if (root != null) {
-      arrowWriter.finish()
-      val columns = root.getFieldVectors.asScala.map(new ArrowColumnVector(_): 
ColumnVector)
+  override def writeBatch(batch: ColumnarBatch): Unit = {
+    val numBatchRows = batch.numRows()
+    val copied = copiedVectors(batch)
+    if (copied.forall(_.isEmpty)) {
+      // Without columns to copy, the values are converted as rows are.
+      var i = 0
+      while (i < numBatchRows) {
+        write(batch.getRow(i))
+        i += 1
+      }
+    } else if (numBatchRows > 0) {
+      // A batch without rows leaves the pending batch of rows as it is.
+      flush()
+      // Each batch starts where the last one ended, so that the batches end 
as those of rows do.
+      var start = 0
+      while (start < numBatchRows) {
+        start += writeRows(
+          batch, copied, start, start + math.min(numBatchRows - start, 
options.maxRecordsPerBatch))
+      }
+    }
+  }
+
+  /**
+   * Returns, for each column of a batch, its Arrow vector if it can be copied 
buffer by buffer, see
+   * `ArrowBatchingDataWriter.canCopy`. The other columns are converted value 
by value.
+   */
+  private def copiedVectors(batch: ColumnarBatch): Seq[Option[FieldVector]] = {
+    val fields = arrowSchema.getFields.asScala
+    if (batch.numCols() != fields.length) {
+      Seq.fill(fields.length)(None)
+    } else {
+      fields.indices.map { i =>
+        batch.column(i) match {
+          case column: ArrowColumnVector => column.getValueVector match {
+            case vector: FieldVector
+                if ArrowBatchingDataWriter.canCopy(vector, fields(i), 
batch.numRows()) =>
+              Some(vector)
+            case _ => None
+          }
+          case _ => None
+        }
+      }
+    }
+  }
+
+  /**
+   * Writes the batch of the rows of a columnar batch that starts at the row 
`start`, and returns
+   * its number of rows: up to `end`, or as a batch of rows ends, up to the 
first row with which it
+   * holds `maxBytesPerBatch` bytes, by the measure of `getBufferSizeFor`, 
which `rangeSize`
+   * computes for the copied columns without slicing them.
+   *
+   * @param copied the vector of each column of the batch that is copied, see 
`copiedVectors`
+   */
+  private def writeRows(
+      batch: ColumnarBatch,
+      copied: Seq[Option[FieldVector]],
+      start: Int,
+      end: Int): Int = {
+    // The columns that are not copied, and the vectors that they are 
converted into.
+    val ordinals = copied.indices.filter(copied(_).isEmpty)
+    val converted = ArrayBuffer.empty[FieldVector]
+    Utils.tryWithSafeFinally {
+      val writers = ordinals.map { ordinal =>
+        val vector = arrowSchema.getFields.get(ordinal).createVector(allocator)
+        converted += vector
+        vector.setInitialCapacity(end - start)
+        vector.allocateNew()
+        ArrowWriter.createFieldWriter(vector)
+      }
+      val length = if (writers.isEmpty) {
+        batchLength(copied.flatten, start, end)
+      } else {
+        convertRows(batch, copied.flatten, ordinals.zip(writers), start, end)
+      }
+      writers.foreach(_.finish())
+      writeSlices(copied, converted.toSeq, start, length)
+      length
+    } {
+      AutoCloseables.close(converted.asJava)
+    }
+  }
+
+  /**
+   * Returns the number of rows of the batch of copied columns that starts at 
the row `start`, see
+   * `writeRows`.
+   */
+  private def batchLength(copied: Seq[FieldVector], start: Int, end: Int): Int 
= {
+    def sizeInBytes(rows: Int): Long =
+      copied.map(ArrowBatchingDataWriter.rangeSize(_, start, start + rows)).sum
+    // The size is not computed when the limit is the largest possible one, as 
for rows.
+    if (options.maxBytesPerBatch >= Int.MaxValue ||
+        sizeInBytes(end - start) < options.maxBytesPerBatch) {
+      end - start
+    } else {
+      // The size grows with the number of rows.
+      var lo = 1
+      var hi = end - start
+      while (lo < hi) {
+        val mid = (lo + hi) >>> 1
+        if (sizeInBytes(mid) >= options.maxBytesPerBatch) hi = mid else lo = 
mid + 1
+      }
+      lo
+    }
+  }
+
+  /**
+   * Converts the columns of a columnar batch that are not copied with their 
writers, by ordinal,
+   * row by row from the row `start` up to the end of the batch, and returns 
its number of rows, see
+   * `writeRows`. As for rows, the size of the batch is computed after each 
row, so that the
+   * conversion stops at the row with which the batch holds `maxBytesPerBatch` 
bytes.
+   */
+  private def convertRows(
+      batch: ColumnarBatch,
+      copied: Seq[FieldVector],
+      writers: Seq[(Int, ArrowFieldWriter)],
+      start: Int,
+      end: Int): Int = {
+    val copiedVectors = copied.toArray
+    val ordinals = writers.map(_._1).toArray
+    val fieldWriters = writers.map(_._2).toArray
+    // The size of the batch once it holds the rows [start, rowEnd).
+    def sizeInBytes(rowEnd: Int): Long = {
+      var size = 0L
+      var i = 0
+      while (i < copiedVectors.length) {
+        size += ArrowBatchingDataWriter.rangeSize(copiedVectors(i), start, 
rowEnd)
+        i += 1
+      }
+      i = 0
+      while (i < fieldWriters.length) {
+        size += fieldWriters(i).getSizeInBytes()
+        i += 1
+      }
+      size
+    }
+    // The size is not computed when the limit is the largest possible one, as 
for rows.
+    val sized = options.maxBytesPerBatch < Int.MaxValue
+    var next = start
+    var full = false
+    while (next < end && !full) {
+      val row = batch.getRow(next)
+      var i = 0
+      while (i < fieldWriters.length) {
+        fieldWriters(i).write(row, ordinals(i))
+        i += 1
+      }
+      next += 1
+      full = sized && sizeInBytes(next) >= options.maxBytesPerBatch
+    }
+    next - start
+  }
+
+  /**
+   * Writes the rows [start, start + length) of a columnar batch, with the 
vectors that its columns
+   * that are not copied were converted into from these rows, in their order. 
The copied columns are
+   * sliced, which shares most buffers of the batch, though it allocates new 
offsets for lists, and
+   * for strings and binaries unless the offset of the row `start` is 0, and 
new validity buffers,
+   * and value buffers for booleans, unless `start` is a multiple of 8. Then 
each buffer of the
+   * slice is copied at its size. Arrow Java reads the offsets of a vector of 
variable width, or of
+   * a list, as if they started at 0, which those of a slice do. The converted 
vectors, which
+   * nothing else writes to, are written as they are.
+   */
+  private def writeSlices(
+      copied: Seq[Option[FieldVector]],
+      converted: Seq[FieldVector],
+      start: Int,
+      length: Int): Unit = {
+    // The slices of the copied columns share the buffers of the batch, which 
their producer
+    // releases, so they are closed once they are copied.
+    val slices = ArrayBuffer.empty[FieldVector]
+    val copy = Utils.tryWithSafeFinally {
+      copied.flatten.foreach { vector =>
+        // The slice is closed also when the buffers that it allocates fail to 
allocate.
+        val pair = vector.getTransferPair(vector.getAllocator)
+        slices += pair.getTo.asInstanceOf[FieldVector]
+        pair.splitAndTransfer(start, length)
+      }
+      copyRoot(new VectorSchemaRoot(slices.map(_.getField).asJava, 
slices.asJava, length))
+    } {
+      AutoCloseables.close(slices.asJava)
+    }
+    Utils.tryWithSafeFinally {
+      val copies = copy.getFieldVectors.asScala.iterator
+      val conversions = converted.iterator
+      val vectors = copied.map { column =>
+        if (column.isDefined) copies.next() else conversions.next()
+      }
+      delegate.write(new ColumnarBatch(
+        vectors.map(new ArrowColumnVector(_): ColumnVector).toArray, length))
+    } {
+      copy.close()
+    }
+  }
+
+  /** Copies the buffers of the vectors of a root into new vectors, once each, 
at their size. */
+  private def copyRoot(source: VectorSchemaRoot): VectorSchemaRoot = {
+    val copy = VectorSchemaRoot.create(source.getSchema, allocator)
+    try {
+      val unloaded = new VectorUnloader(source).getRecordBatch
       try {
-        delegate.write(new ColumnarBatch(columns.toArray, root.getRowCount))
+        val buffers = ArrayBuffer.empty[ArrowBuf]
+        try {
+          unloaded.getBuffers.asScala.foreach { buffer =>
+            val size = buffer.readableBytes()
+            val copied = allocator.buffer(size)
+            buffers += copied
+            copied.setBytes(0, buffer, buffer.readerIndex(), size)
+            copied.writerIndex(size)
+          }
+          val batch = new ArrowRecordBatch(source.getRowCount, 
unloaded.getNodes, buffers.asJava)
+          try new VectorLoader(copy).load(batch) finally batch.close()
+        } finally {
+          buffers.foreach(_.close())
+        }
       } finally {
-        releaseBatch()
+        unloaded.close()
       }
+      copy
+    } catch {
+      case e: Throwable =>
+        copy.close()
+        throw e
+    }
+  }
+
+  private def writeArrow(batchRoot: VectorSchemaRoot): Unit = {
+    val columns = batchRoot.getFieldVectors.asScala.map(new 
ArrowColumnVector(_): ColumnVector)
+    delegate.write(new ColumnarBatch(columns.toArray, batchRoot.getRowCount))
+  }
+
+  private def flush(): Unit = {
+    if (root != null) {
+      arrowWriter.finish()
+      Utils.tryWithSafeFinally(writeArrow(root))(releaseBatch())
     }
   }
 
@@ -266,3 +518,56 @@ class ArrowBatchingDataWriter(delegate: 
DataWriter[ColumnarBatch], options: Arro
 
   override def currentMetricsValues(): Array[CustomTaskMetric] = 
delegate.currentMetricsValues()
 }
+
+object ArrowBatchingDataWriter {
+  /**
+   * Returns whether the Arrow vector of a column of a batch of `numRows` rows 
can be copied buffer
+   * by buffer as a column of the field of a write: it has a value for each 
row, it is physically
+   * congruent with the field, though it may differ in the names, nullability 
and metadata of its
+   * fields, and in the time zones of timestamps, which the copy keeps, and 
Arrow can slice it.
+   * Arrow computes some byte positions of a slice in int, so it cannot slice 
a fixed-width vector
+   * of 2 GiB or more, for example (see 
`BaseSliceArrowOutputProcessor.canSlice`), whose values are
+   * then converted.
+   */
+  def canCopy(vector: FieldVector, field: Field, numRows: Int): Boolean = {
+    vector.getValueCount == numRows &&
+      ArrowUtils.isCompatibleWithDeclaredField(vector.getField, field) &&
+      BaseSliceArrowOutputProcessor.canSlice(vector)
+  }
+
+  /**
+   * Returns the size in bytes of the rows [start, end) of a vector, as 
`getBufferSizeFor` measures
+   * a slice of these rows, without slicing them: from the offsets of the 
vectors of variable width
+   * and of the lists, at every level. The vectors of other types than those 
of `toArrowSchema` are
+   * sliced.
+   */
+  def rangeSize(vector: ValueVector, start: Int, end: Int): Long = {
+    val n = end - start
+    def validity: Long = BitVectorHelper.getValidityBufferSizeFromCount(n)
+    if (n == 0) {
+      0L
+    } else {
+      vector match {
+        case _: NullVector => 0L
+        case _: BitVector => 2 * validity
+        case v: BaseFixedWidthVector => n.toLong * v.getTypeWidth + validity
+        case v: BaseVariableWidthVector =>
+          validity + (n + 1L) * BaseVariableWidthVector.OFFSET_WIDTH +
+            (v.getStartOffset(end) - v.getStartOffset(start))
+        case v: ListVector =>
+          val offsets = v.getOffsetBuffer
+          val width = BaseRepeatedValueVector.OFFSET_WIDTH.toLong
+          validity + (n + 1L) * width + rangeSize(v.getDataVector,
+            offsets.getInt(start * width), offsets.getInt(end * width))
+        case v: StructVector =>
+          validity + v.getChildrenFromFields.asScala.map(rangeSize(_, start, 
end)).sum
+        case v: NonNullableStructVector =>
+          v.getChildrenFromFields.asScala.map(rangeSize(_, start, end)).sum
+        case v =>
+          val pair = v.getTransferPair(v.getAllocator)
+          pair.splitAndTransfer(start, n)
+          try pair.getTo.getBufferSizeFor(n).toLong finally pair.getTo.close()
+      }
+    }
+  }
+}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
index aa6c0f2a9bf9..6e05edcf4e36 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
@@ -278,13 +278,14 @@ object BaseSliceArrowOutputProcessor {
    * Returns whether `VectorSchemaRoot.slice` can slice `vector` and its 
children. Arrow computes
    * some byte positions of a slice in int, so it throws, and leaks the 
vectors sliced so far, for
    * a fixed-width vector of 2 GiB or more, a view vector with 2^27 rows or 
more, or a list vector
-   * with 2^29 - 1 rows or more. A batch that holds one is output whole 
instead.
+   * with 2^29 - 1 rows or more. A batch that holds one is output whole 
instead, and the writers of
+   * columnar data sources convert such a column instead of slicing it.
    *
    * The offset positions of a list vector overflow only from 2^29 rows, but a 
slice of `length`
    * rows also allocates `(length + 1) * 4` bytes of offsets in int, and a 
list vector nested in
    * another one is sliced over all its rows when the slice of its parent 
covers them.
    */
-  private[python] def canSlice(vector: FieldVector): Boolean = {
+  private[execution] def canSlice(vector: FieldVector): Boolean = {
     val fits = vector match {
       case v: BaseFixedWidthVector => v.getValueCount.toLong * v.getTypeWidth 
<= Int.MaxValue
       case v: BaseVariableWidthViewVector =>
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
index 2433ea3cb20f..94e2b5a78b88 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnarDataSourceSuite.scala
@@ -18,36 +18,50 @@ package 
org.apache.spark.sql.execution.datasources.v2.columnar
 
 import java.nio.charset.StandardCharsets.UTF_8
 import java.util
-import java.util.concurrent.ConcurrentHashMap
-import java.util.concurrent.atomic.AtomicInteger
+import java.util.Locale
+import java.util.concurrent.{ConcurrentHashMap, ConcurrentLinkedQueue}
+import java.util.concurrent.atomic.{AtomicInteger, AtomicLong}
 
 import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
+import scala.util.Try
 
-import org.apache.spark.{SparkException, TaskContextImpl}
+import org.apache.arrow.c.{ArrowArray, ArrowSchema, Data}
+import org.apache.arrow.vector.{BigIntVector, DecimalVector, FieldVector, 
FixedSizeBinaryVector, LargeVarCharVector, ValueVector, VectorSchemaRoot}
+import org.apache.arrow.vector.complex.ListVector
+import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType, Schema}
+
+import org.apache.spark.{SparkArithmeticException, SparkException, 
TaskContextImpl}
 import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
 import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
 import org.apache.spark.sql.connector.catalog.{Table, TableCapability, 
TableProvider}
 import org.apache.spark.sql.connector.expressions.Transform
 import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader}
 import org.apache.spark.sql.connector.read.streaming.{ReadLimit, 
SupportsAdmissionControl}
 import org.apache.spark.sql.connector.write.{DataWriter, 
PhysicalWriteInfoImpl, WriterCommitMessage}
-import 
org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation, 
DataWritingSparkTask}
-import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
-import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector
+import org.apache.spark.sql.execution.{ApplyColumnarRulesAndInsertTransitions, 
ColumnarRule, ColumnarToRowExec, RowToColumnConverter, SparkPlan}
+import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, 
AdaptiveSparkPlanHelper}
+import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, 
DataSourceV2ScanRelation, DataWritingSparkTask, MicroBatchScanExec, 
V2TableWriteExec}
+import org.apache.spark.sql.execution.streaming.runtime.{MemoryStream, 
StreamingQueryWrapper}
+import org.apache.spark.sql.execution.vectorized.{OnHeapColumnVector, 
ReusedArrowColumnVector, WritableColumnVector}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.sources.DataSourceRegister
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.util.{ArrowUtils, CaseInsensitiveStringMap}
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch, 
ColumnVector}
+import org.apache.spark.unsafe.types.UTF8String
 import org.apache.spark.util.Utils
 
 /**
  * Tests the Data Source V2 adapter of [[ColumnarDataSource]] with a data 
source implemented on the
  * JVM. `NativeDataSourceSuite` tests the native data sources, which implement 
it.
  */
-class ColumnarDataSourceSuite extends QueryTest with SharedSparkSession {
+class ColumnarDataSourceSuite extends QueryTest with SharedSparkSession
+  with AdaptiveSparkPlanHelper {
   import testImplicits._
 
   private val format = "test_columnar_range"
@@ -136,6 +150,541 @@ class ColumnarDataSourceSuite extends QueryTest with 
SharedSparkSession {
     }
   }
 
+  test("write the columnar batches of a query") {
+    // Runs a write, and returns the plan of the query that it writes.
+    def writtenPlan(write: => Unit): SparkPlan = {
+      QueryTest.withQueryExecutionsCaptured(spark)(write).flatMap { qe =>
+        qe.executedPlan.collectFirst { case writeExec: V2TableWriteExec => 
writeExec.query }
+      }.head
+    }
+    // Writes the ids [0, 10) and their names, read in Arrow batches of 
`arrowBatchSize` rows, or in
+    // a batch of on-heap vectors for each of the two partitions of the scan.
+    def copy(table: String, arrowBatchSize: Int = 0, keep: Boolean = false): 
SparkPlan = {
+      writtenPlan {
+        spark.read.format(format).option("end", "10").option("arrowBatchSize", 
arrowBatchSize)
+          .load()
+          .write.format(format).option("table", table).option("keep", 
keep).mode("append").save()
+      }
+    }
+    val expected = (0L until 10L).map(id => (id, s"name$id"))
+
+    withSQLConf(
+        SQLConf.ARROW_EXECUTION_MAX_RECORDS_PER_BATCH.key -> "3",
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+        SQLConf.ADAPTIVE_EXECUTION_FORCE_APPLY.key -> "false") {
+      // The data writers get the batches of the scan, without a transition to 
rows. On-heap
+      // vectors are converted to Arrow value by value, into batches that end 
as those of rows do.
+      assert(copy("columnar_write").isInstanceOf[BatchScanExec])
+      assert(TestColumnarSink.rows("columnar_write") == expected)
+      assert(TestColumnarSink.batchSizes("columnar_write") == Seq(3, 2, 3, 2))
+
+      // Arrow batches are copied buffer by buffer, each in batches of at most 
3 rows.
+      assert(copy("arrow_write", arrowBatchSize = 
2).isInstanceOf[BatchScanExec])
+      assert(TestColumnarSink.rows("arrow_write") == expected)
+      assert(TestColumnarSink.batchSizes("arrow_write") == Seq(2, 2, 1, 2, 2, 
1))
+      copy("large_arrow_write", arrowBatchSize = 5)
+      assert(TestColumnarSink.rows("large_arrow_write") == expected)
+      assert(TestColumnarSink.batchSizes("large_arrow_write") == Seq(3, 2, 3, 
2))
+
+      // A data writer may keep the data of the batches that it gets, though 
the scan overwrites
+      // its Arrow vectors with each batch.
+      copy("kept_arrow_write", arrowBatchSize = 2, keep = true)
+      assert(TestColumnarSink.rows("kept_arrow_write") == expected)
+
+      // A query that does not produce columnar batches is written as rows.
+      val plan = writtenPlan {
+        spark.read.format(format).option("end", "10").load()
+          .selectExpr("id", "upper(name) AS name")
+          .write.format(format).option("table", 
"row_write").mode("append").save()
+      }
+      assert(plan.exists(_.isInstanceOf[ColumnarToRowExec]), plan)
+
+      // With adaptive query execution, a query with a shuffle is an adaptive 
plan, which is not
+      // columnar, so its batches are converted to rows.
+      val shuffled = writtenPlan {
+        spark.read.format(format).option("end", "10").option("arrowBatchSize", 
2).load()
+          .repartition(2)
+          .write.format(format).option("table", 
"shuffled_write").mode("append").save()
+      }
+      assert(shuffled.isInstanceOf[AdaptiveSparkPlanExec], shuffled)
+      assert(find(shuffled)(_.isInstanceOf[ColumnarToRowExec]).isDefined, 
shuffled)
+      assert(TestColumnarSink.rows("shuffled_write").sorted == expected)
+
+      // A query that supports rows too, such as the scan of a cached 
DataFrame, is written as rows.
+      val cached = spark.read.format(format).option("end", 
"10").load().select("id").cache()
+      try {
+        cached.count()
+        val writes = QueryTest.withQueryExecutionsCaptured(spark) {
+          cached.write.format(format).option("table", 
"cached_write").mode("append").save()
+        }.flatMap(_.executedPlan.collectFirst { case writeExec: 
V2TableWriteExec => writeExec })
+        val write = writes.head
+        assert(write.supportsColumnarInput, write)
+        assert(write.query.isInstanceOf[InMemoryTableScanExec], write.query)
+        assert(write.query.supportsColumnar && write.query.supportsRowBased, 
write.query)
+        assert(!write.writesColumnar, write)
+        assert(TestColumnarSink.rows("cached_write").sorted == expected.map { 
case (id, _) =>
+          (id, "")
+        })
+      } finally {
+        cached.unpersist()
+      }
+      assert(TestColumnarSink.rows("row_write") == expected.map { case (id, 
name) =>
+        (id, name.toUpperCase(Locale.ROOT))
+      })
+    }
+  }
+
+  test("count the rows and release each batch of a columnar write") {
+    // The write node counts the rows of the batches that its data writers 
wrote.
+    val writes = QueryTest.withQueryExecutionsCaptured(spark) {
+      spark.read.format(format).option("end", "10").option("arrowBatchSize", 
"3").load()
+        .write.format(format).option("table", 
"counted_write").mode("append").save()
+    }.flatMap(_.executedPlan.collectFirst { case writeExec: V2TableWriteExec 
=> writeExec })
+    assert(writes.head.writesColumnar, writes.head)
+    assert(writes.head.metrics("numOutputRows").value == 10)
+
+    // A scan whose batches have new Arrow vectors leaves them to the 
consumer, which releases each
+    // batch once its data writer took its data, also when the data writer 
fails on it.
+    Seq(false, true).foreach { fail =>
+      TestOwnedBatches.reset()
+      val table = s"owned_batches_write_$fail"
+      val write = Try {
+        spark.read.format(format).option("end", "10").option("arrowBatchSize", 
"3")
+          .option("ownedBatches", "true").load()
+          .write.format(format).option("table", table).option("fail", 
fail).mode("append").save()
+      }
+      assert(write.isFailure == fail, write)
+      if (!fail) {
+        assert(TestColumnarSink.rows(table) == (0L until 10L).map(id => (id, 
s"name$id")))
+      }
+      assert(!TestOwnedBatches.releases.isEmpty)
+      assert(TestOwnedBatches.releases.asScala.map(_.get()).toSet == Set(1))
+      assert(TestOwnedBatches.leakedBytes.get() == 0)
+    }
+  }
+
+  test("write rows when an extension has columnar rules") {
+    // The physical plan of a copy, before the transitions between rows and 
columns.
+    val qe = QueryTest.withQueryExecutionsCaptured(spark) {
+      spark.read.format(format).option("end", "10").load()
+        .write.format(format).option("table", 
"extension_write").mode("append").save()
+    }.find(_.executedPlan.exists(_.isInstanceOf[V2TableWriteExec])).get
+    def write(columnarRules: Seq[ColumnarRule]): V2TableWriteExec = {
+      ApplyColumnarRulesAndInsertTransitions(columnarRules, outputsColumnar = 
false)
+        .apply(qe.sparkPlan)
+        .collectFirst { case write: V2TableWriteExec => write }.get
+    }
+    // Without columnar rules, the write takes the batches of the scan.
+    assert(write(Nil).writesColumnar && 
write(Nil).query.isInstanceOf[BatchScanExec])
+    // The columnar rules of an extension, such as of an engine that runs 
queries in native code,
+    // may replace the transitions to rows with their own, which may be the 
only way to read its
+    // batches, so the write then takes rows through a transition.
+    val withRules = write(Seq(new ColumnarRule))
+    assert(!withRules.writesColumnar, withRules)
+    assert(withRules.query.isInstanceOf[ColumnarToRowExec], withRules.query)
+  }
+
+  test("copy the Arrow columns of every type, or convert the columns of a 
batch") {
+    val nullable = Seq(
+      "id % 2 = 0", "CAST(id AS TINYINT)", "CAST(id AS SMALLINT)", "CAST(id AS 
INT)", "id",
+      "CAST(id AS FLOAT)", "CAST(id AS DOUBLE)", "CAST(id AS DECIMAL(10, 2))",
+      "CAST(id AS DECIMAL(38, 10))", "CAST(id AS STRING)", "CAST(CAST(id AS 
STRING) AS BINARY)",
+      "DATE_ADD(DATE'2020-01-01', CAST(id AS INT))", "TIMESTAMP_SECONDS(id)",
+      "CAST(TIMESTAMP_SECONDS(id) AS TIMESTAMP_NTZ)", 
"MAKE_YM_INTERVAL(CAST(id AS INT), 1)",
+      "MAKE_DT_INTERVAL(CAST(id AS INT))", "MAKE_INTERVAL(0, 1, 0, CAST(id AS 
INT))",
+      "PARSE_JSON(CAST(id AS STRING))", "ARRAY(id, NULL)", "ARRAY(CAST(id AS 
STRING))",
+      "MAP(CAST(id AS STRING), id)", "MAP(id, ARRAY(CAST(id AS STRING)))",
+      "NAMED_STRUCT('a', id, 'b', CAST(id AS STRING))", 
"ARRAY(NAMED_STRUCT('x', id))")
+    // c0 is not nullable, c1 has the null type, and the other columns have 
nulls.
+    val df = spark.range(7).selectExpr(
+      Seq("id AS c0", "NULL AS c1") ++ nullable.zipWithIndex.map { case (expr, 
i) =>
+        s"IF(id % 3 = 1, NULL, $expr) AS c${i + 2}"
+      }: _*)
+    val schema = df.schema
+    val rows = df.queryExecution.toRdd.map(_.copy()).collect().toSeq
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("test", 0, 
Long.MaxValue)
+    try {
+      // Writes a batch with the session time zone America/Los_Angeles, and 
returns the sizes of
+      // the batches that the data writer got, the time zone of their 
timestamp column c14, which
+      // a copy keeps, the Arrow types of their string column c11, and their 
rows. The data writer
+      // keeps the data of the batches, and reads it once `overwrite` 
overwrote the batch.
+      def writeAndRead(
+          batch: ColumnarBatch,
+          overwrite: () => Unit,
+          schema: StructType = schema,
+          maxRecordsPerBatch: Int = 10,
+          maxBytesPerBatch: Long = Int.MaxValue)
+          : (Seq[Int], Seq[String], Seq[ArrowType], Seq[InternalRow]) = {
+        val sizes = ArrayBuffer.empty[Int]
+        val timeZones = ArrayBuffer.empty[String]
+        val stringTypes = ArrayBuffer.empty[ArrowType]
+        val kept = ArrayBuffer.empty[TestArrowExport]
+        def field(batch: ColumnarBatch, name: String) = 
batch.column(schema.fieldIndex(name))
+          .asInstanceOf[ArrowColumnVector].getValueVector.getField
+        val dataWriter = new DataWriter[ColumnarBatch] {
+          override def write(batch: ColumnarBatch): Unit = {
+            sizes += batch.numRows()
+            timeZones += field(batch, 
"c14").getType.asInstanceOf[ArrowType.Timestamp].getTimezone
+            stringTypes += field(batch, "c11").getType
+            kept += TestArrowExport(batch)
+          }
+          override def commit(): WriterCommitMessage = null
+          override def abort(): Unit = {}
+          override def close(): Unit = kept.foreach(_.close())
+        }
+        val writer = new ArrowBatchingDataWriter(dataWriter, 
ArrowBatchingOptions(
+          schema, "America/Los_Angeles", maxRecordsPerBatch, maxBytesPerBatch))
+        val result = try {
+          writer.writeBatch(batch)
+          writer.commit()
+          overwrite()
+          val toUnsafe = UnsafeProjection.create(schema)
+          val written = kept.toSeq.flatMap { exported =>
+            
exported.read(_.rowIterator().asScala.map(toUnsafe(_).copy()).toSeq)
+          }
+          kept.clear()
+          (sizes.toSeq, timeZones.toSeq, stringTypes.toSeq, written)
+        } finally {
+          writer.close()
+        }
+        // Once closed, the writer holds nothing, which a leak would still 
hold.
+        assert(writer.allocator.getAllocatedMemory == 0)
+        result
+      }
+
+      // Runs `f` with a batch of Arrow vectors with the time zone UTC, and a 
function that
+      // overwrites the vectors with the rows in reverse order.
+      def withArrowBatch(largeVarTypes: Boolean)(f: (ColumnarBatch, () => 
Unit) => Unit): Unit = {
+        val root = VectorSchemaRoot.create(
+          ArrowUtils.toArrowSchema(
+            schema, "UTC", errorOnDuplicatedFieldNames = true, largeVarTypes = 
largeVarTypes),
+          allocator)
+        try {
+          val arrowWriter = ArrowWriter.create(root)
+          rows.foreach(arrowWriter.write)
+          arrowWriter.finish()
+          val batch = new ColumnarBatch(
+            root.getFieldVectors.asScala.map(new ArrowColumnVector(_): 
ColumnVector).toArray,
+            rows.length)
+          f(batch, () => {
+            arrowWriter.reset()
+            rows.reverse.foreach(arrowWriter.write)
+            arrowWriter.finish()
+          })
+        } finally {
+          root.close()
+        }
+      }
+
+      val expected = rows.map(UnsafeProjection.create(schema)(_).copy())
+      val utf8 = ArrowType.Utf8.INSTANCE
+      Seq(false, true).foreach { largeVarTypes =>
+        withArrowBatch(largeVarTypes) { (batch, _) =>
+          // The size of a range of rows, which the batches of a copy end at, 
is the size of a slice
+          // of these rows.
+          (0 until batch.numCols()).foreach { i =>
+            val vector = 
batch.column(i).asInstanceOf[ArrowColumnVector].getValueVector
+            Seq((0, 7), (1, 4), (3, 7), (6, 7), (2, 2)).foreach { case (start, 
end) =>
+              val pair = vector.getTransferPair(vector.getAllocator)
+              pair.splitAndTransfer(start, end - start)
+              val sliceSize =
+                try pair.getTo.getBufferSizeFor(end - start) finally 
pair.getTo.close()
+              assert(ArrowBatchingDataWriter.rangeSize(vector, start, end) == 
sliceSize,
+                s"${vector.getField} [$start, $end)")
+            }
+          }
+        }
+      }
+      withArrowBatch(largeVarTypes = false) { (batch, overwrite) =>
+        // The columns are Arrow vectors with the types of the schema, so they 
are copied.
+        assert(writeAndRead(batch, overwrite) == ((Seq(7), Seq("UTC"), 
Seq(utf8), expected)))
+      }
+      withArrowBatch(largeVarTypes = false) { (batch, overwrite) =>
+        // In slices of at most 3 rows, which start in the middle of the bytes 
of validity.
+        assert(writeAndRead(batch, overwrite, maxRecordsPerBatch = 3) ==
+          ((Seq(3, 3, 1), Seq.fill(3)("UTC"), Seq.fill(3)(utf8), expected)))
+      }
+      withArrowBatch(largeVarTypes = false) { (batch, overwrite) =>
+        // A copied batch also ends after the first row with which it holds 
the maximum size.
+        assert(writeAndRead(batch, overwrite, maxBytesPerBatch = 1) ==
+          ((Seq.fill(7)(1), Seq.fill(7)("UTC"), Seq.fill(7)(utf8), expected)))
+      }
+      withArrowBatch(largeVarTypes = true) { (batch, overwrite) =>
+        // Each column of another Arrow type than that of the schema, here a 
string or a binary, is
+        // converted on its own, and the other columns are copied.
+        assert(writeAndRead(batch, overwrite, maxRecordsPerBatch = 3) ==
+          ((Seq(3, 3, 1), Seq.fill(3)("UTC"), Seq.fill(3)(utf8), expected)))
+      }
+      withArrowBatch(largeVarTypes = true) { (batch, overwrite) =>
+        // The size of the converted columns counts too.
+        assert(writeAndRead(batch, overwrite, maxBytesPerBatch = 1) ==
+          ((Seq.fill(7)(1), Seq.fill(7)("UTC"), Seq.fill(7)(utf8), expected)))
+      }
+
+      // The columns of a batch imported with the C data interface, as a 
native read imports them,
+      // are copied the same way, from any row, though their buffers are those 
of their producer:
+      // the batch is still readable afterwards, until its producer releases 
it.
+      Seq(3 -> Int.MaxValue.toLong, 10 -> 1L).foreach { case 
(maxRecordsPerBatch, maxBytes) =>
+        withArrowBatch(largeVarTypes = false) { (exported, _) =>
+          val vectors = (0 until exported.numCols()).map { i =>
+            exported.column(i).asInstanceOf[ArrowColumnVector].getValueVector
+              .asInstanceOf[FieldVector]
+          }
+          val importer = 
ArrowUtils.rootAllocator.newChildAllocator("importer", 0, Long.MaxValue)
+          try {
+            val array = ArrowArray.allocateNew(importer)
+            val arrowSchema = ArrowSchema.allocateNew(importer)
+            val imported = try {
+              Data.exportVectorSchemaRoot(allocator,
+                new VectorSchemaRoot(vectors.map(_.getField).asJava, 
vectors.asJava, rows.length),
+                null, array, arrowSchema)
+              Data.importVectorSchemaRoot(importer, array, arrowSchema, null)
+            } finally {
+              array.close()
+              arrowSchema.close()
+            }
+            try {
+              val batch = new ColumnarBatch(imported.getFieldVectors.asScala
+                .map(new ArrowColumnVector(_): ColumnVector).toArray, 
imported.getRowCount)
+              val sizes = if (maxBytes == 1L) Seq.fill(7)(1) else Seq(3, 3, 1)
+              assert(writeAndRead(batch, () => (), maxRecordsPerBatch = 
maxRecordsPerBatch,
+                maxBytesPerBatch = maxBytes) ==
+                ((sizes, Seq.fill(sizes.length)("UTC"), 
Seq.fill(sizes.length)(utf8), expected)))
+              val toUnsafe = UnsafeProjection.create(schema)
+              assert(batch.rowIterator().asScala.map(toUnsafe(_).copy()).toSeq 
== expected)
+            } finally {
+              imported.close()
+            }
+          } finally {
+            // Fails if the copy left any memory of the imported batch 
allocated.
+            importer.close()
+          }
+        }
+      }
+
+      // On-heap vectors are converted too. They do not support the null type.
+      val onHeapSchema = StructType(schema.filterNot(_.name == "c1"))
+      val onHeapRows = rows.map(row => 
InternalRow.fromSeq(row.toSeq(schema).patch(1, Nil, 1)))
+      val vectors: Seq[WritableColumnVector] =
+        OnHeapColumnVector.allocateColumns(onHeapRows.length, 
onHeapSchema).toSeq
+      try {
+        val converter = new RowToColumnConverter(onHeapSchema)
+        onHeapRows.foreach(converter.convert(_, vectors.toArray))
+        val batch = new ColumnarBatch(vectors.map(v => v: 
ColumnVector).toArray, onHeapRows.length)
+        assert(writeAndRead(batch, () => (), onHeapSchema, maxRecordsPerBatch 
= 3) ==
+          ((Seq(3, 3, 1), Seq.fill(3)("America/Los_Angeles"), 
Seq.fill(3)(utf8),
+            onHeapRows.map(UnsafeProjection.create(onHeapSchema)(_).copy()))))
+      } finally {
+        vectors.foreach(_.close())
+      }
+    } finally {
+      allocator.close()
+    }
+  }
+
+  test("write the pending rows of converted batches before a copied batch") {
+    val schema = StructType.fromDDL("id BIGINT, name STRING")
+    val writer = new ArrowBatchingDataWriter(
+      new TestDataWriter(fail = false), ArrowBatchingOptions(schema, "UTC", 
10, Int.MaxValue))
+    val onHeap = new TestBatchReader(0, 3, schema)
+    val arrow = new TestArrowBatchReader(3, 7, schema, 4)
+    try {
+      // The rows of the on-heap batch stay in the batch of rows until the 
Arrow batch is copied.
+      assert(onHeap.next())
+      writer.writeBatch(onHeap.get())
+      assert(arrow.next())
+      writer.writeBatch(arrow.get())
+      val message = writer.commit().asInstanceOf[TestCommitMessage]
+      assert(message.rows == (0L until 7L).map(id => (id, s"name$id")))
+      assert(message.batchSizes == Seq(3, 4))
+    } finally {
+      writer.close()
+      onHeap.close()
+      arrow.close()
+    }
+    assert(writer.allocator.getAllocatedMemory == 0)
+  }
+
+  test("a copied batch ends at the byte limit as a batch of rows does") {
+    val schema = StructType.fromDDL("id BIGINT, name STRING")
+    def write(batch: ColumnarBatch, maxBytesPerBatch: Long): TestCommitMessage 
= {
+      val writer = new ArrowBatchingDataWriter(new TestDataWriter(fail = 
false),
+        ArrowBatchingOptions(schema, "UTC", 5, maxBytesPerBatch))
+      val message = try {
+        writer.writeBatch(batch)
+        writer.commit().asInstanceOf[TestCommitMessage]
+      } finally {
+        writer.close()
+      }
+      assert(writer.allocator.getAllocatedMemory == 0)
+      message
+    }
+    val onHeap = new TestBatchReader(0, 20, schema)
+    val arrow = new TestArrowBatchReader(0, 20, schema, 20)
+    // The names are large strings, which are converted, while the ids are 
copied.
+    val largeNames = new TestArrowBatchReader(0, 20, schema, 20, largeVarTypes 
= true)
+    try {
+      assert(onHeap.next() && arrow.next() && largeNames.next())
+      val vectors = (0 until 2).map {
+        arrow.get().column(_).asInstanceOf[ArrowColumnVector].getValueVector
+      }
+      // The size of the first 2 rows. The names of the ids from 10 are 1 byte 
longer.
+      val twoRows = vectors.map(ArrowBatchingDataWriter.rangeSize(_, 0, 2)).sum
+      Seq(twoRows - 1, twoRows, twoRows + 1, 3 * twoRows).foreach { limit =>
+        val rows = write(onHeap.get(), limit)
+        assert(rows.batchSizes.sum == 20)
+        // The batches end at the same rows when the names are copied, or 
converted row by row.
+        Seq(arrow, largeNames).foreach { reader =>
+          assert(write(reader.get(), limit) == rows, s"limit $limit")
+        }
+      }
+      // The limit ends the batches every 2 rows, also when 5 rows do not 
divide by them.
+      assert(write(arrow.get(), twoRows).batchSizes == Seq.fill(10)(2))
+    } finally {
+      onHeap.close()
+      arrow.close()
+      largeNames.close()
+    }
+  }
+
+  test("convert the columns of a batch that are not copied up to the end of 
each batch") {
+    val schema = StructType.fromDDL("id BIGINT, name STRING")
+    val size = 1 << 20
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("test", 0, 
Long.MaxValue)
+    // The ids are copied, and the names, large strings of 1 MiB, are 
converted.
+    val root = VectorSchemaRoot.create(
+      ArrowUtils.toArrowSchema(
+        schema, "UTC", errorOnDuplicatedFieldNames = true, largeVarTypes = 
true),
+      allocator)
+    try {
+      root.allocateNew()
+      val ids = root.getVector("id").asInstanceOf[BigIntVector]
+      val names = root.getVector("name").asInstanceOf[LargeVarCharVector]
+      (0 until 20).foreach { i =>
+        ids.setSafe(i, i)
+        names.setSafe(i, Array.fill(size)(('a' + i).toByte))
+      }
+      root.setRowCount(20)
+      var conversions = 0
+      val batch = new ColumnarBatch(Array[ColumnVector](
+        new ArrowColumnVector(ids),
+        new ArrowColumnVector(names) {
+          override def getUTF8String(rowId: Int): UTF8String = {
+            conversions += 1
+            super.getUTF8String(rowId)
+          }
+        }), 20)
+      // The batches hold 2 MiB from their second row, before their maximum of 
10 rows.
+      val writer = new ArrowBatchingDataWriter(new TestDataWriter(fail = 
false),
+        ArrowBatchingOptions(schema, "UTC", 10, 2L * size))
+      val message = try {
+        writer.writeBatch(batch)
+        writer.commit().asInstanceOf[TestCommitMessage]
+      } finally {
+        writer.close()
+      }
+      assert(writer.allocator.getAllocatedMemory == 0)
+      assert(message.batchSizes == Seq.fill(10)(2))
+      assert(message.rows.map { case (id, name) => (id, name.length, 
name.distinct) } ==
+        (0 until 20).map(i => (i.toLong, size, ('a' + i).toChar.toString)))
+      // Each row is converted once, and only up to the row at which its batch 
ends, so the
+      // converted names never hold more than those of a batch, rather than 
those of 10 rows.
+      assert(conversions == 20)
+      assert(writer.allocator.getPeakMemoryAllocation < 5L * size)
+    } finally {
+      root.close()
+      allocator.close()
+    }
+  }
+
+  test("copy only the Arrow columns that Arrow can slice") {
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("test", 0, 
Long.MaxValue)
+    val idField = ArrowUtils.toArrowSchema(StructType.fromDDL("id BIGINT"), 
"UTC",
+      errorOnDuplicatedFieldNames = true, largeVarTypes = 
false).getFields.get(0)
+    val ids = new BigIntVector(idField, allocator)
+    // A vector that reports more values than it holds, which Arrow cannot 
slice, since it computes
+    // the byte positions of a slice of a fixed-width vector of 2 GiB or more 
in int (see
+    // `BaseSliceArrowOutputProcessor.canSlice`), and a list vector with such 
a child.
+    def binaries(): FixedSizeBinaryVector = new 
FixedSizeBinaryVector("binaries", allocator, 8) {
+      override def getValueCount: Int = Int.MaxValue / 8 + 1
+    }
+    val unsliceable = Seq(
+      binaries(),
+      new ListVector("lists", allocator, 
FieldType.nullable(ArrowType.List.INSTANCE), null) {
+        replaceDataVector(binaries())
+      })
+    try {
+      ids.allocateNew(3)
+      ids.setValueCount(3)
+      assert(ArrowBatchingDataWriter.canCopy(ids, idField, 3))
+      // A vector without a value for each row of its batch is converted too.
+      assert(!ArrowBatchingDataWriter.canCopy(ids, idField, 2))
+      unsliceable.foreach { vector =>
+        assert(!ArrowBatchingDataWriter.canCopy(vector, vector.getField, 
vector.getValueCount),
+          vector.getField)
+      }
+    } finally {
+      (ids +: unsliceable).foreach(_.close())
+      allocator.close()
+    }
+  }
+
+  test("an empty columnar batch does not end the pending batch of rows") {
+    val schema = StructType.fromDDL("id BIGINT, name STRING")
+    val writer = new ArrowBatchingDataWriter(
+      new TestDataWriter(fail = false), ArrowBatchingOptions(schema, "UTC", 
10, Int.MaxValue))
+    val first = new TestBatchReader(0, 2, schema)
+    val empty = new TestArrowBatchReader(0, 0, schema, 1)
+    val second = new TestBatchReader(2, 4, schema)
+    try {
+      assert(first.next() && second.next())
+      writer.writeBatch(first.get())
+      writer.writeBatch(empty.get())
+      writer.writeBatch(second.get())
+      assert(writer.commit().asInstanceOf[TestCommitMessage].batchSizes == 
Seq(4))
+    } finally {
+      writer.close()
+      first.close()
+      empty.close()
+      second.close()
+    }
+    assert(writer.allocator.getAllocatedMemory == 0)
+  }
+
+  test("release the converted columns when a later column fails to convert") {
+    val schema = StructType.fromDDL("id BIGINT, a DECIMAL(10, 2), b 
DECIMAL(10, 2)")
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("test", 0, 
Long.MaxValue)
+    val root = VectorSchemaRoot.create(new Schema(Seq(
+      Field.notNullable("id", new ArrowType.Int(64, true)),
+      Field.nullable("a", new ArrowType.Decimal(38, 2, 128)),
+      Field.nullable("b", new ArrowType.Decimal(38, 2, 128))).asJava), 
allocator)
+    try {
+      root.allocateNew()
+      root.getVector("id").asInstanceOf[BigIntVector].setSafe(0, 0)
+      // A decimal of another precision than that of the schema is converted, 
and the value of b
+      // does not fit the precision of the schema.
+      root.getVector("a").asInstanceOf[DecimalVector].setSafe(0, new 
java.math.BigDecimal("1.00"))
+      root.getVector("b").asInstanceOf[DecimalVector]
+        .setSafe(0, new java.math.BigDecimal("1000000000000.00"))
+      root.setRowCount(1)
+      val batch = new ColumnarBatch(
+        root.getFieldVectors.asScala.map(new ArrowColumnVector(_): 
ColumnVector).toArray, 1)
+      val writer = new ArrowBatchingDataWriter(
+        new TestDataWriter(fail = false), ArrowBatchingOptions(schema, "UTC", 
10, Int.MaxValue))
+      try {
+        val e = intercept[SparkArithmeticException](writer.writeBatch(batch))
+        assert(e.getCondition == 
"NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION")
+        // The vector of a, which was converted, is released with the failure.
+        assert(writer.allocator.getAllocatedMemory == 0)
+      } finally {
+        writer.close()
+      }
+    } finally {
+      root.close()
+      allocator.close()
+    }
+  }
+
   test("create the writer once the data to write is computed") {
     val table = "failed_upstream"
     // The error happens in a shuffle stage, which adaptive query execution 
runs before the write,
@@ -347,6 +896,13 @@ class ColumnarDataSourceSuite extends QueryTest with 
SharedSparkSession {
       .start()
     try {
       query.processAllAvailable()
+      // A micro-batch writes the batches of the scan, without a transition to 
rows.
+      val plan =
+        
query.asInstanceOf[StreamingQueryWrapper].streamingQuery.lastExecution.executedPlan
+      assert(plan.collectFirst { case write: V2TableWriteExec => write.query }
+        .exists(_.isInstanceOf[MicroBatchScanExec]), plan)
+      // The progress of the micro-batch counts the rows of the batches that 
it wrote.
+      assert(query.recentProgress.map(_.sink.numOutputRows).sum == 5)
     } finally {
       query.stop()
     }
@@ -448,10 +1004,13 @@ class TestRangeDataSource(options: 
CaseInsensitiveStringMap) extends ColumnarDat
     }.getOrElse(super.capabilities())
   }
 
+  // With the option "arrowBatchSize", the batch scans return Arrow batches of 
that size, with new
+  // vectors for each batch with the option "ownedBatches".
   override def batchPlanner(schema: StructType): ColumnarBatchPlanner = {
     batchPlanners.incrementAndGet()
     new TestRangePlanner(end, schema, options.getBoolean("failPlanning", 
false), plans, prunes,
-      options.getBoolean("failPruning", false))
+      options.getBoolean("failPruning", false), 
options.getInt("arrowBatchSize", 0),
+      options.getBoolean("ownedBatches", false))
   }
 
   override def streamReader(
@@ -461,9 +1020,11 @@ class TestRangeDataSource(options: 
CaseInsensitiveStringMap) extends ColumnarDat
     new TestRangeStreamReader(end, schema, table)
   }
 
+  // With the option "keep", the data writers keep the data of the batches 
until they commit.
   override def writer(schema: StructType, overwrite: Boolean): ColumnarWriter 
= {
     TestColumnarSink.writers.add(table)
-    new TestSinkWriter(table, overwrite, options.getBoolean("fail", false))
+    new TestSinkWriter(
+      table, overwrite, options.getBoolean("fail", false), 
options.getBoolean("keep", false))
   }
 
   override def streamWriter(
@@ -484,7 +1045,9 @@ class TestRangePlanner(
     fail: Boolean,
     plans: AtomicInteger,
     prunes: AtomicInteger,
-    failPruning: Boolean)
+    failPruning: Boolean,
+    arrowBatchSize: Int,
+    ownedBatches: Boolean)
   extends ColumnarBatchPlanner {
   private var limit = Long.MaxValue
   private var readSchema = schema
@@ -511,14 +1074,23 @@ class TestRangePlanner(
     val mid = hi / 2
     ColumnarPlannedBatch(
       Array(TestRangePartition(0, mid), TestRangePartition(mid, hi)),
-      new TestRangeReaderFactory(readSchema))
+      new TestRangeReaderFactory(readSchema, arrowBatchSize, ownedBatches))
   }
 }
 
-class TestRangeReaderFactory(readSchema: StructType) extends 
ColumnarOnlyPartitionReaderFactory {
+/**
+ * @param arrowBatchSize the size of the Arrow batches to return, or 0 for 
on-heap batches
+ * @param ownedBatches whether each Arrow batch has new vectors, see 
[[TestArrowBatchReader]]
+ */
+class TestRangeReaderFactory(readSchema: StructType, arrowBatchSize: Int, 
ownedBatches: Boolean)
+  extends ColumnarOnlyPartitionReaderFactory {
   override def createColumnarReader(partition: InputPartition): 
PartitionReader[ColumnarBatch] = {
     val range = partition.asInstanceOf[TestRangePartition]
-    new TestBatchReader(range.lo, range.hi, readSchema)
+    if (arrowBatchSize > 0) {
+      new TestArrowBatchReader(range.lo, range.hi, readSchema, arrowBatchSize, 
owned = ownedBatches)
+    } else {
+      new TestBatchReader(range.lo, range.hi, readSchema)
+    }
   }
 }
 
@@ -578,6 +1150,99 @@ class TestBatchReader(lo: Long, hi: Long, schema: 
StructType)
   override def close(): Unit = if (consumed) batch.close()
 }
 
+/**
+ * Returns the ids [lo, hi) and their names in Arrow batches of at most 
`batchSize` rows. It writes
+ * each batch to the same Arrow vectors, over the previous one, unless `owned`.
+ *
+ * @param largeVarTypes whether the strings are `large_utf8`, which the 
writers convert
+ * @param owned whether each batch has new Arrow vectors, which the consumer 
releases with the
+ *              batch, see [[TestOwnedBatches]]
+ */
+class TestArrowBatchReader(
+    lo: Long,
+    hi: Long,
+    schema: StructType,
+    batchSize: Int,
+    largeVarTypes: Boolean = false,
+    owned: Boolean = false)
+  extends PartitionReader[ColumnarBatch] {
+  private val allocator =
+    ArrowUtils.rootAllocator.newChildAllocator("test arrow batch reader", 0, 
Long.MaxValue)
+  private val arrowSchema = ArrowUtils.toArrowSchema(
+    schema, "UTC", errorOnDuplicatedFieldNames = true, largeVarTypes = 
largeVarTypes)
+  // The vectors of the batches, which are reused for each batch, so a 
consumer does not release
+  // them, unless the batches are owned: each one then has new vectors, which 
its consumer releases.
+  private val reusedRoot = VectorSchemaRoot.create(arrowSchema, allocator)
+  private var arrowWriter = ArrowWriter.create(reusedRoot)
+  private var batch = new ColumnarBatch(
+    reusedRoot.getFieldVectors.asScala.map(new ReusedArrowColumnVector(_): 
ColumnVector).toArray)
+  private val ownedRoots = ArrayBuffer.empty[VectorSchemaRoot]
+  private var start = lo
+
+  override def next(): Boolean = {
+    if (start >= hi) {
+      false
+    } else {
+      val end = math.min(hi, start + batchSize)
+      if (owned) {
+        val root = VectorSchemaRoot.create(arrowSchema, allocator)
+        ownedRoots += root
+        arrowWriter = ArrowWriter.create(root)
+        batch = new ColumnarBatch(
+          root.getFieldVectors.asScala.map(TestOwnedBatches.vector(_): 
ColumnVector).toArray)
+      } else {
+        arrowWriter.reset()
+      }
+      (start until end).foreach { id =>
+        arrowWriter.write(InternalRow.fromSeq(schema.fieldNames.toSeq.map {
+          case "id" => id
+          case "name" => UTF8String.fromString(s"name$id")
+        }))
+      }
+      arrowWriter.finish()
+      batch.setNumRows((end - start).toInt)
+      start = end
+      true
+    }
+  }
+
+  override def get(): ColumnarBatch = batch
+
+  override def close(): Unit = {
+    // What the consumer did not release of the owned batches leaks: it is 
recorded, and freed.
+    TestOwnedBatches.leakedBytes.addAndGet(
+      
ownedRoots.flatMap(_.getFieldVectors.asScala).map(_.getBufferSize.toLong).sum)
+    ownedRoots.foreach(_.close())
+    reusedRoot.close()
+    allocator.close()
+  }
+}
+
+/** The Arrow vectors of the owned batches of [[TestArrowBatchReader]], and 
their releases. */
+object TestOwnedBatches {
+  // The number of times that each vector was released.
+  val releases = new ConcurrentLinkedQueue[AtomicInteger]()
+  // The bytes of the owned batches that the consumers did not release.
+  val leakedBytes = new AtomicLong()
+
+  def reset(): Unit = {
+    releases.clear()
+    leakedBytes.set(0)
+  }
+
+  /** Returns a column vector of `vector`, which counts its releases. */
+  def vector(vector: ValueVector): ArrowColumnVector = {
+    val released = new AtomicInteger()
+    releases.add(released)
+    new ArrowColumnVector(vector) {
+      override def close(): Unit = {
+        released.incrementAndGet()
+        super.close()
+      }
+    }
+  }
+}
+
 /** The data written by the test writers, by table. */
 object TestColumnarSink {
   val committed = new ConcurrentHashMap[String, Seq[TestCommitMessage]]()
@@ -602,9 +1267,14 @@ object TestColumnarSink {
 case class TestCommitMessage(rows: Seq[(Long, String)], batchSizes: Seq[Int])
   extends WriterCommitMessage
 
-class TestDataWriter(fail: Boolean) extends DataWriter[ColumnarBatch] {
+/**
+ * @param keep whether to keep the data of each batch until the commit, as a 
native data writer
+ *             can, by taking ownership of the Arrow buffers of the batch
+ */
+class TestDataWriter(fail: Boolean, keep: Boolean = false) extends 
DataWriter[ColumnarBatch] {
   private val rows = ArrayBuffer.empty[(Long, String)]
   private val batchSizes = ArrayBuffer.empty[Int]
+  private val kept = ArrayBuffer.empty[TestArrowExport]
 
   override def write(batch: ColumnarBatch): Unit = {
     if (fail) {
@@ -612,21 +1282,94 @@ class TestDataWriter(fail: Boolean) extends 
DataWriter[ColumnarBatch] {
     }
     assert((0 until 
batch.numCols()).forall(batch.column(_).isInstanceOf[ArrowColumnVector]))
     batchSizes += batch.numRows()
+    if (keep) {
+      kept += TestArrowExport(batch)
+    } else {
+      read(batch)
+    }
+  }
+
+  private def read(batch: ColumnarBatch): Unit = {
     (0 until batch.numRows()).foreach { i =>
-      rows += ((batch.column(0).getLong(i), 
batch.column(1).getUTF8String(i).toString))
+      // A batch without the column of the names, such as that of a cached 
DataFrame of the ids,
+      // gets empty names.
+      val name = if (batch.numCols() > 1) 
batch.column(1).getUTF8String(i).toString else ""
+      rows += ((batch.column(0).getLong(i), name))
     }
   }
 
-  override def commit(): WriterCommitMessage = TestCommitMessage(rows.toSeq, 
batchSizes.toSeq)
+  override def commit(): WriterCommitMessage = {
+    kept.foreach(_.read(read))
+    kept.clear()
+    TestCommitMessage(rows.toSeq, batchSizes.toSeq)
+  }
 
   override def abort(): Unit = {}
 
-  override def close(): Unit = {}
+  override def close(): Unit = {
+    kept.foreach(_.close())
+    kept.clear()
+  }
+}
+
+/** A batch exported to the Arrow C data interface, which keeps its data until 
it is read. */
+class TestArrowExport(array: ArrowArray, schema: ArrowSchema) {
+  private var arrayClosed = false
+  private var schemaClosed = false
+
+  /**
+   * Imports the batch, and passes it to `f`. Each import closes the struct 
that it gets, also when
+   * it fails, so `close` then only releases the struct that is still open.
+   */
+  def read[T](f: ColumnarBatch => T): T = {
+    val arrowSchema = try Data.importSchema(ArrowUtils.rootAllocator, schema, 
null) finally {
+      schemaClosed = true
+    }
+    val root = VectorSchemaRoot.create(arrowSchema, ArrowUtils.rootAllocator)
+    try {
+      try Data.importIntoVectorSchemaRoot(ArrowUtils.rootAllocator, array, 
root, null) finally {
+        arrayClosed = true
+      }
+      f(new ColumnarBatch(
+        root.getFieldVectors.asScala.map(new ArrowColumnVector(_): 
ColumnVector).toArray,
+        root.getRowCount))
+    } finally {
+      root.close()
+    }
+  }
+
+  def close(): Unit = {
+    if (!arrayClosed) {
+      arrayClosed = true
+      array.release()
+      array.close()
+    }
+    if (!schemaClosed) {
+      schemaClosed = true
+      schema.release()
+      schema.close()
+    }
+  }
+}
+
+object TestArrowExport {
+  def apply(batch: ColumnarBatch): TestArrowExport = {
+    val vectors = (0 until batch.numCols()).map { i =>
+      
batch.column(i).asInstanceOf[ArrowColumnVector].getValueVector.asInstanceOf[FieldVector]
+    }
+    val root =
+      new VectorSchemaRoot(vectors.map(_.getField).asJava, vectors.asJava, 
batch.numRows())
+    val array = ArrowArray.allocateNew(ArrowUtils.rootAllocator)
+    val schema = ArrowSchema.allocateNew(ArrowUtils.rootAllocator)
+    Data.exportVectorSchemaRoot(ArrowUtils.rootAllocator, root, null, array, 
schema)
+    new TestArrowExport(array, schema)
+  }
 }
 
-class TestSinkWriter(table: String, overwrite: Boolean, fail: Boolean) extends 
ColumnarWriter {
+class TestSinkWriter(table: String, overwrite: Boolean, fail: Boolean, keep: 
Boolean)
+  extends ColumnarWriter {
   override def createWriter(partitionId: Int, taskId: Long): 
DataWriter[ColumnarBatch] = {
-    new TestDataWriter(fail)
+    new TestDataWriter(fail, keep)
   }
 
   override def commit(messages: Array[WriterCommitMessage]): Unit = {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 9dc836ce12e6..ad1982253cd2 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -46,9 +46,10 @@ import 
org.apache.spark.sql.connector.read.streaming.{Offset, ReadLimit, Support
 import org.apache.spark.sql.connector.write.PhysicalWriteInfoImpl
 import org.apache.spark.sql.execution.WholeStageCodegenExec
 import org.apache.spark.sql.execution.datasources.DataSource
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, 
DataSourceV2ScanRelation, MicroBatchScanExec, V2TableWriteExec}
 import 
org.apache.spark.sql.execution.datasources.v2.columnar.{ColumnarBatchScan, 
ColumnarBatchWrite, ColumnarMicroBatchStream, ColumnarScan, 
ColumnarScanBuilder, ColumnarStreamingWrite, ColumnarStreamOffset}
 import 
org.apache.spark.sql.execution.datasources.v2.python.{PythonDataSourceV2, 
UserDefinedPythonDataSource}
+import org.apache.spark.sql.execution.streaming.runtime.StreamingQueryWrapper
 import org.apache.spark.sql.execution.vectorized.ReusedArrowColumnVector
 import org.apache.spark.sql.functions.lit
 import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
@@ -655,9 +656,13 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
     write("overwrite", 10 until 12)
     assert(written() == (10 until 12).map(i => s"$i,v$i").toSet)
 
-    // Nulls and the other types of the test library.
-    range(session, "end" -> "6").write.format("native_sink").option("path", 
dir.getPath)
-      .mode("overwrite").save()
+    // Nulls and the other types of the test library, from the Arrow batches 
of a native scan,
+    // which the write gets without a transition to rows.
+    val plans = QueryTest.withQueryExecutionsCaptured(session) {
+      range(session, "end" -> "6").write.format("native_sink").option("path", 
dir.getPath)
+        .mode("overwrite").save()
+    }.flatMap(_.executedPlan.collectFirst { case write: V2TableWriteExec => 
write.query })
+    assert(plans.length == 1 && plans.head.isInstanceOf[BatchScanExec], plans)
     assert(written() == (0 until 6).map { id =>
       val name = if (id % 5 == 0) "null" else s"n$id"
       s"$id,${id % 3},$name,${format(id * 0.5)},${id % 2 == 0},$id,${id * 
1000000L}"
@@ -933,6 +938,11 @@ class NativeDataSourceSuite extends QueryTest with 
SharedSparkSession {
       .start()
     try {
       query.processAllAvailable()
+      // A micro-batch writes the Arrow batches of the scan, without a 
transition to rows.
+      val plan =
+        
query.asInstanceOf[StreamingQueryWrapper].streamingQuery.lastExecution.executedPlan
+      assert(plan.collectFirst { case write: V2TableWriteExec => write.query }
+        .exists(_.isInstanceOf[MicroBatchScanExec]), plan)
     } finally {
       query.stop()
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to