This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8c0b8f853274 [SPARK-60381][SQL] Account the late-materialization 
survivor buffer to the task's execution memory
8c0b8f853274 is described below

commit 8c0b8f85327447620f110b8019b7bda04f2d1717
Author: Peter Toth <[email protected]>
AuthorDate: Thu Oct 22 20:43:54 2026 -0700

    [SPARK-60381][SQL] Account the late-materialization survivor buffer to the 
task's execution memory
    
    ### What changes were proposed in this pull request?
    
    The late-materialization Parquet reader (SPARK-59620) buffers a row group's 
surviving key values and row ranges before it emits that row group, up to 
`spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes` per reader. 
Today the buffer is outside Spark's memory manager. This PR accounts it to the 
task's execution memory through a `MemoryConsumer`. It was on the follow-up 
list of #58895, and viirya suggested it there after the merge 
([comment](https://github.com/apache/spark/p [...]
    
    - **Reserving:** phase 1 reserves the buffer as it grows.
      - A request asks for at least what is already reserved, up to 1 MiB and 
never past the cap. So a growing buffer asks only a few times. The reservation 
stays below twice what the buffer holds while it is small, and below 1 MiB more 
once it is larger.
      - What phase 1 reserved ahead is given back before phase 2 reads.
      - A denied reservation gives splicing up, as passing the cap does, and 
phase 2 reads the key columns again.
      - If the row ranges are denied too, the filter is given up for that row 
group. A denied request that holds no key values gives the filter up at once, 
since giving splicing up would free nothing. That is always the case off heap.
    - **Releasing:** as batches are emitted, the reservation steps down once it 
is 1 MiB past what is still buffered. The rest goes when the next row group 
loads, or at close. So emitting a row group takes the memory manager's lock a 
few times, not once per batch.
    - **No spill:** the consumer's `spill()` frees nothing.
      - The buffer is bounded by the cap, and it drains as its batches are 
emitted.
      - Freeing the queued keys would make the rest of the row group read its 
key pages again and build its ranges again. Both happen outside the memory 
manager, and can take more than the spill freed. A spill would also never free 
the ranges.
      - Accounting with denial lets other consumers that yield to another 
consumer's request, such as a sorter being read, spill before this one is 
refused. A refused reader backs off.
    - **Off heap:** the consumer is `ON_HEAP` whatever the reader's memory 
mode, as `WindowSegmentTree`'s spiller is (SPARK-56546). Off-heap column 
vectors allocate native memory outside the Tungsten off-heap pool, so an 
`OFF_HEAP` consumer would charge that pool for memory it never held. With 
off-heap vectors only the row ranges are accounted.
    - **Errors:** the task's memory manager runs other consumers' spills inside 
a reservation, and can evict cached blocks to disk there. So a reservation can 
raise anything, an `IOException` included. The reader rethrows it in a form 
`FileScanRDD` cannot take for a corrupt or missing file:
      - a fatal error, or one raised by a pending task kill, goes out as the 
reader's other error paths send it, an `InternalError` in an `UnknownError`;
      - any other `Error`, such as the `SparkOutOfMemoryError` of a failed 
spill, goes out as it is;
      - any other exception goes out wrapped in a checked `SparkException`.
    - **Metrics:** four new scan metrics count the row groups the budget 
limited. Each row group counts at most once, under the last thing given up and 
what caused it:
      - `row groups with key columns read twice by storage filter (buffer cap)` 
and `(memory denied)`: splicing was given up and the filter kept;
      - `row groups read without storage filter (buffer cap)` and `(memory 
denied)`: the filter was given up.
    
      The first of each kind per split is also logged, at INFO.
    
    ### Why are the changes needed?
    
    Each reader buffers up to 64 MB by default, which the memory manager cannot 
see. Today a task holds one per scan at a time, so an executor holds about one 
per running task. Under a sorted merge over many files, as storage-partitioned 
joins do once storage filters reach them, one task would hold many at once. 
Accounted, other consumers spill earlier, and a reader that is denied memory 
gives the buffer up instead. The new metrics show when a scan loses splicing or 
the filter, and whethe [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The feature is unreleased and off by default. With it on:
    - the buffer counts against the task's execution memory;
    - a task holding the buffer is one of the tasks sharing on-heap execution 
memory;
    - the scan has four more metrics.
    
    ### How was this patch tested?
    
    New tests in `ParquetStorageFilterSuite`, most under a task whose memory 
manager the test controls:
    
    - **Holding and releasing:** the buffer is held while a spliced row group 
is emitted, and released when the next one loads. On heap that is the key 
values and the row ranges, off heap the row ranges only. Past the cap only the 
ranges stay reserved, even with a batch of keys already queued. What phase 1 
reserved ahead is gone by the first batch. A reader closed after its first 
batch releases what it holds, and a row group whose filter is given up holds 
nothing while it is read whole. E [...]
    - **Step release:** emitting a 3.6 MB row group steps the reservation down 
at exactly the batches where it is 1 MiB past what is still buffered.
    - **Denial:** memory denied for the keys gives splicing up, for the same 
bytes as a cap of that size. Memory denied for the ranges gives the filter up, 
on and off heap. The metrics tell the two causes apart, and each is logged once 
per split.
    - **Asking:** the buffer asks for less than twice what it holds while small 
and less than 1 MiB more once larger, and never asks past the cap. So within 
those margins of the free memory it spills no other consumer.
    - **Errors:** a `RuntimeException` or an `InternalError` from another 
consumer's spill, or an `IOException` from the memory manager, fails the read 
with a checked `SparkException` or an `UnknownError`. `ignoreCorruptFiles` 
swallows none of them. A fatal error nested in one goes out as itself.
    - **Counting:** the existing cap tests now count each row group under what 
it gave up, once. A row group that gave splicing up and then the filter counts 
as giving the filter up.
    
    Each was checked by mutation. These all make at least one of them fail:
    
    - ignoring a denied reservation, or one denied again without key values;
    - asking again off heap after a denial;
    - accounting the key values off heap;
    - not releasing at close;
    - asking past the cap;
    - releasing at every batch rather than in steps;
    - keeping what phase 1 reserved ahead through phase 2;
    - keeping the ranges reserved once the filter is given up;
    - reserving off heap;
    - letting an error from another consumer's spill out unwrapped;
    - counting a row group as giving splicing up before it may give the filter 
up too;
    - counting a denied key or range reservation as passing the cap;
    - logging every time rather than once per split;
    - not taking an emitted batch off the queued total, or not clearing that 
total when splicing is given up.
    
    `ParquetStorageFilterSuite` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code with Claude Opus 5.5
    
    Closes #59639 from peter-toth/SPARK-60381-survivor-buffer-memory-consumer.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../org/apache/spark/sql/internal/SQLConf.scala    |  14 +-
 .../LateMaterializationParquetRecordReader.java    | 302 +++++++++++--
 .../datasources/parquet/ParquetStorageFilter.scala |  60 ++-
 .../parquet/ParquetStorageFilterSuite.scala        | 495 ++++++++++++++++++---
 .../datasources/parquet/ParquetTestAccess.scala    |  14 +-
 5 files changed, 778 insertions(+), 107 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index a1cbd5691a4f..570657bec19e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2051,7 +2051,12 @@ object SQLConf {
         s"the key values are native memory outside ${MEMORY_OFFHEAP_SIZE.key}, 
so they come " +
         s"out of ${EXECUTOR_MEMORY_OVERHEAD.key}. Past the limit the reader 
drops the buffered " +
         "key values and reads the key columns again for the surviving rows. If 
the row ranges " +
-        "alone pass it, the reader reads the row group with no filter 
applied.")
+        "alone pass it, the reader reads the row group with no filter applied. 
The buffer is " +
+        "also counted against the task's execution memory, except key values 
held in off-heap " +
+        "column vectors, and a reservation the memory manager denies is 
handled the same way. " +
+        "The scan's metrics count the row groups each of these happens to, by 
cause. While it " +
+        "holds a buffer, the task is one of those sharing on-heap execution 
memory, as with any " +
+        "operator that reserves it.")
       .version("5.0.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .bytesConf(ByteUnit.BYTE)
@@ -2074,9 +2079,10 @@ object SQLConf {
         "as in the files pyarrow's write_table writes by default, or with that 
conf set to " +
         "false, the reader can only skip whole row groups, and it reads the 
key columns twice in " +
         "the others. With a page index, it can be slower when the matching 
rows are spread " +
-        "across most pages. The matching key values of a row group are 
buffered while it is " +
-        s"read, up to 
${PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key}, " +
-        "outside Spark's memory manager.")
+        "across most pages. The matching key values and row ranges of a row 
group are buffered " +
+        "while it is read, up to " +
+        s"${PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key}, 
and counted " +
+        "against the task's execution memory, except key values held in 
off-heap column vectors.")
       .version("5.0.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
index 5d0e0206b6ee..007f1250c1e3 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
@@ -27,6 +27,7 @@ import java.util.List;
 import java.util.Objects;
 import java.util.PrimitiveIterator;
 import java.util.Set;
+import java.util.function.Function;
 
 import org.apache.parquet.column.ColumnDescriptor;
 import org.apache.parquet.column.page.PageReadStore;
@@ -35,17 +36,22 @@ import org.apache.parquet.hadoop.metadata.BlockMetaData;
 import org.apache.parquet.hadoop.metadata.ColumnPath;
 import 
org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
 
+import org.apache.spark.TaskContext;
 import org.apache.spark.internal.LogKeys;
 import org.apache.spark.internal.MDC;
 import org.apache.spark.internal.SparkLogger;
 import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryConsumer;
 import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.memory.TaskMemoryManager;
 import org.apache.spark.sql.catalyst.expressions.BasePredicate;
 import org.apache.spark.sql.catalyst.InternalRow;
 import org.apache.spark.sql.execution.datasources.FileFormat;
+import org.apache.spark.sql.execution.metric.SQLMetric;
 import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
 import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
 import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.internal.SQLConf;
 import org.apache.spark.sql.types.DataType;
 import org.apache.spark.sql.types.DataTypes;
 import org.apache.spark.sql.types.DecimalType;
@@ -83,6 +89,9 @@ import org.apache.spark.sql.vectorized.ColumnarBatch;
  * <p>A spliced key slot holds a different vector in each batch, and {@link 
#nextBatch()} frees the
  * previous one. A caller must fetch {@code batch.column(i)} again after every 
{@code nextBatch()}.
  *
+ * <p>The survivor buffer is accounted to the task's execution memory, the key 
values only on heap,
+ * see {@link SurvivorMemory}.
+ *
  * <p>When the scan has the column {@code 
FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME}, this
  * reader fills it. A batch never spans two row groups, so one value covers 
each batch. It is true
  * where the filter was applied, whose rows are then exactly the survivors, 
and false where it was
@@ -99,10 +108,10 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
 
   /**
    * What one buffered variable-length value costs, as a multiple of its own 
bytes. The byte child
-   * holds a null byte per byte, and {@link WritableColumnVector#reserve} 
doubles it when it grows.
-   * It over-counts on purpose, since nothing can spill this buffer. That 
holds at the default
-   * growth. A {@code hugeVectorReserveRatio} above 2 makes it under-count past
-   * {@code hugeVectorThreshold}, which is off by default.
+   * holds a null byte per byte, and {@link WritableColumnVector#reserve} 
doubles it when it grows,
+   * so this is the most the value takes on heap, and what the task's memory 
manager is asked for.
+   * That holds at the default growth. A {@code hugeVectorReserveRatio} above 
2 makes it
+   * under-count past {@code hugeVectorThreshold}, which is off by default.
    */
   private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
 
@@ -118,6 +127,13 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
    */
   private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
 
+  /**
+   * The most the survivor buffer reserves beyond what it holds, see
+   * {@link #reserveSurvivorMemory}. The task's memory manager may spill other 
consumers, or evict
+   * cached blocks, to grant a request in full, so an ask past what is free 
can cost them that much.
+   */
+  private static final long SURVIVOR_MEMORY_STEP_BYTES = 1L << 20;
+
   /**
    * The filter this reader applies. {@link #keyColumns} is null for a file it 
is not applied to row
    * by row.
@@ -183,15 +199,18 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
   private WritableColumnVector checkedVector;
 
   /**
-   * Splicing state. {@link #survivorBatches} holds one array of survivor 
vectors per batch to emit,
-   * indexed like {@link #keyColumns}, and {@link #currentKeyAccumulators} the 
array still filling.
-   * Together they hold every survivor of one row group, outside any 
MemoryConsumer.
+   * Splicing state. {@link #survivorBatches} holds one {@link SurvivorBatch} 
per batch to emit, and
+   * {@link #currentKeyAccumulators} the vectors still filling, indexed like 
{@link #keyColumns}.
+   * Together they hold the survivors of one row group not yet emitted.
    */
-  private final ArrayDeque<WritableColumnVector[]> survivorBatches = new 
ArrayDeque<>();
+  private final ArrayDeque<SurvivorBatch> survivorBatches = new ArrayDeque<>();
+  /** What the batches in {@link #survivorBatches} are charged together. */
+  private long queuedKeyCharge;
   /** The survivor vectors still filling, or null when there are none. */
   private WritableColumnVector[] currentKeyAccumulators;
-  /** Row count of {@link #currentKeyAccumulators}. */
+  /** Row count and charge of {@link #currentKeyAccumulators}. */
   private int currentKeyAccumulatorRowCount;
+  private long currentKeyAccumulatorCharge;
   /**
    * The survivor vectors the current batch is built on. The next batch frees 
them, or
    * {@link #close()} when no next batch comes.
@@ -202,6 +221,25 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
   /** The array the emitted batch is built over, whose key slots each batch 
rewrites. */
   private ColumnVector[] spliceBatchColumns;
 
+  /** What the survivor ranges of the row group being read or emitted cost to 
hold. */
+  private long rangeCharge;
+  /**
+   * The survivor buffer's account with the task's memory manager, or null 
without one, or for a
+   * file the filter is not applied to row by row.
+   */
+  private SurvivorMemory survivorMemory;
+  /**
+   * What {@link #survivorMemory} holds. Only this reader changes it, so it is 
kept here rather
+   * than read off the consumer's atomic counter per survivor.
+   */
+  private long reservedBytes;
+  /**
+   * What the budget gave splicing up for in the row group being loaded, or 
null. Phase 1 leaves it
+   * for {@link #loadNextRowGroup} to record once phase 2 keeps the filter, so 
a row group counts
+   * once.
+   */
+  private BudgetGiveUp splicingGiveUp;
+
   public LateMaterializationParquetRecordReader(
       ZoneId convertTz,
       String datetimeRebaseMode,
@@ -241,11 +279,17 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
         }
       } finally {
         try {
+          // So `abandonSplicing()` releases everything at once.
+          rangeCharge = 0L;
           closeSplicingState();
         } finally {
-          if (checkedVector != null) {
-            checkedVector.close();
-            checkedVector = null;
+          try {
+            if (checkedVector != null) {
+              checkedVector.close();
+              checkedVector = null;
+            }
+          } finally {
+            releaseSurvivorMemoryTo(0L);
           }
         }
       }
@@ -337,17 +381,21 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
   public boolean nextBatch() throws IOException {
     closeAll(publishedKeyVectors);
     publishedKeyVectors = null;
+    trimSurvivorMemory();
     if (!super.nextBatch()) return false;
     if (keyColumns != null) {
       if (spliceCurrentRowGroup) {
-        publishedKeyVectors = survivorBatches.pollFirst();
-        if (publishedKeyVectors == null) {
+        SurvivorBatch next = survivorBatches.pollFirst();
+        if (next == null) {
           // Unreachable, since the queue holds exactly the survivors the emit 
loop counts. Without
           // this check the key slots would show stale values.
           throw ParquetStorageFilter.internalError(String.format(
               "Storage-filter survivor queue of row group %d in %s ran out 
with %d rows still "
                   + "to emit", nextBlockIndex - 1, fileReader.getFile(), 
columnarBatch.numRows()));
         }
+        publishedKeyVectors = next.vectors();
+        // Still reserved, until a later trim, the next row group's load, or 
close() releases it.
+        queuedKeyCharge -= next.charge();
       }
       for (int k = 0; k < keyColumns.length; k++) {
         int slot = keyColumns[k].batchSlot();
@@ -445,6 +493,12 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
     }
     keyFixedBytesPerRow = fixedBytesPerRow;
     keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+    // A test's task can have no memory manager.
+    TaskContext context = TaskContext.get();
+    TaskMemoryManager taskMemoryManager = context == null ? null : 
context.taskMemoryManager();
+    if (taskMemoryManager != null) {
+      survivorMemory = new SurvivorMemory(taskMemoryManager);
+    }
     // Last, since a non-null `keyColumns` is what says the filter is applied 
to this file.
     keyColumns = keys;
   }
@@ -460,6 +514,8 @@ public class LateMaterializationParquetRecordReader extends 
VectorizedParquetRec
     // assignment comes only after the next row group's phase 1, whose budget 
does not count them.
     closeDataPages();
     releaseRowGroupReaders();
+    rangeCharge = 0L;
+    releaseSurvivorMemoryTo(heldCharge());
     StorageFilterMetrics m = storageFilter.metrics();
     List<BlockMetaData> blocks = fileReader.getRowGroups();
     while (nextBlockIndex < blocks.size()) {
@@ -500,7 +556,10 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
       // Null means phase 1 gave the filter up, and phase 2 then reads every 
row the pushed filter
       // kept.
       boolean filterGivenUp = survivors == null;
-      if (filterGivenUp) abandonSplicing();
+      if (filterGivenUp) abandonFilter();
+      // Phase 1 may have reserved up to a step past what it buffered, which 
phase 2's reads should
+      // not keep from other consumers.
+      releaseSurvivorMemoryTo(heldCharge());
       RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
       long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
       // The byte metrics' baseline, the non-key bytes a plain read asks for 
over the rows the
@@ -532,15 +591,16 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
               MDC.of(LogKeys.REASON, e.getMessage()));
         }
         filterGivenUp = true;
-        abandonSplicing();
+        abandonFilter();
         finalRanges = pushedFilterRanges;
         finalRowCount = baselineRows;
         fileReader.setRequestedSchema(requestedColumns);
         dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
       }
       requireRowCount(dataPages, finalRowCount, blockIdx);
-      // A row group whose filter was given up reports nothing (see 
`StorageFilterMetrics`).
+      // A row group whose filter was given up counts no saving, nor a 
splicing give-up.
       if (!filterGivenUp) {
+        if (splicingGiveUp != null) recordBudgetGiveUp(splicingGiveUp);
         // Over the columns phase 2 read, the key columns included once 
splicing is given up. A
         // spliced row group that pruned nothing read exactly the baseline.
         long phase2Bytes = spliceCurrentRowGroup && finalRowCount == 
baselineRows
@@ -640,7 +700,7 @@ public class LateMaterializationParquetRecordReader extends 
VectorizedParquetRec
    * <p>Returns null when the filter is given up for this row group:
    * <ul>
    *   <li>at the first survivor of a row group that cannot be read in 
part;</li>
-   *   <li>when the surviving ranges pass the budget;</li>
+   *   <li>when the surviving ranges pass the budget, or the task's memory 
manager denies them;</li>
    *   <li>on an error decoding the key pages or applying the filter.</li>
    * </ul>
    */
@@ -663,11 +723,10 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
 
     PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
     RowRanges.Builder finalRangesBuilder = RowRanges.builder();
-    // The budget weighs both the bytes buffered for splicing and the 
survivors' ranges.
-    long splicedBytes = 0L;
-    long rangeBytes = 0L;
     long previousSurvivor = -2L;
+    // The budget weighs both the key values buffered for splicing and the 
survivors' ranges.
     long cap = storageFilter.maxSplicedRowGroupBytes();
+    splicingGiveUp = null;
     // Out of the loop, since resolving it is a volatile read.
     BasePredicate predicate = storageFilter.preparedPredicate();
     while (remaining > 0) {
@@ -712,12 +771,12 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
           return null;
         }
         finalRangesBuilder.addSelectedRow(blockRow);
-        if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+        if (blockRow != previousSurvivor + 1) rangeCharge += ROW_RANGE_BYTES;
         previousSurvivor = blockRow;
         if (spliceCurrentRowGroup) {
           try {
             if (currentKeyAccumulators == null) allocateKeyAccumulators();
-            splicedBytes += appendSurvivorRowToAccumulators(r);
+            appendSurvivorRowToAccumulators(r);
           } catch (RuntimeException e) {
             // The buffer is optional, so failing to fill it gives splicing 
up, not the filter. That
             // covers a vector that cannot grow, and a key value the predicate 
did not read on this
@@ -727,15 +786,32 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
             // where a plain read fails.
             reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
             abandonSplicing();
-            splicedBytes = 0L;
           }
         }
-        if (splicedBytes + rangeBytes > cap) {
-          // The buffer goes first, which is a no-op once splicing is given up.
-          abandonSplicing();
-          splicedBytes = 0L;
+        if (bufferedKeyCharge() + rangeCharge > cap) {
+          // The buffer goes first.
+          if (spliceCurrentRowGroup) {
+            splicingGiveUp = BudgetGiveUp.SPLICING_ON_CAP;
+            abandonSplicing();
+          }
           // The ranges would be thrown away, so stop.
-          if (rangeBytes > cap) return null;
+          if (rangeCharge > cap) {
+            recordBudgetGiveUp(BudgetGiveUp.FILTER_ON_CAP);
+            return null;
+          }
+        }
+        // A reservation the task's memory manager denies is handled as 
passing the cap is. With no
+        // key values in it, giving splicing up frees nothing, so the ranges 
are denied too.
+        long held = heldCharge();
+        boolean denied = !reserveSurvivorMemory(held);
+        if (denied && held > rangeCharge) {
+          splicingGiveUp = BudgetGiveUp.SPLICING_ON_MEMORY;
+          abandonSplicing();
+          denied = !reserveSurvivorMemory(heldCharge());
+        }
+        if (denied) {
+          recordBudgetGiveUp(BudgetGiveUp.FILTER_ON_MEMORY);
+          return null;
         }
       }
       remaining -= num;
@@ -797,10 +873,9 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
 
   /**
    * Appends row {@code srcRow} of every key column to the accumulators, which 
the caller has
-   * allocated, and hands them to the queue once they are full. Returns the 
bytes the row added,
-   * which the caller weighs against its budget.
+   * allocated, adds its bytes to their charge, and hands them to the queue 
once they are full.
    */
-  private long appendSurvivorRowToAccumulators(int srcRow) {
+  private void appendSurvivorRowToAccumulators(int srcRow) {
     long charged = 0L;
     final int dstRow = currentKeyAccumulatorRowCount;
     final WritableColumnVector[] accs = currentKeyAccumulators;
@@ -820,16 +895,19 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
         }
       }
     }
+    currentKeyAccumulatorCharge += keyFixedBytesPerRow + charged;
     currentKeyAccumulatorRowCount = dstRow + 1;
     if (currentKeyAccumulatorRowCount == capacity) pushAccumulators();
-    return keyFixedBytesPerRow + charged;
   }
 
   /** Hands the accumulators to the queue the emit path takes its batches 
from. */
   private void pushAccumulators() {
-    survivorBatches.addLast(currentKeyAccumulators);
+    survivorBatches.addLast(
+        new SurvivorBatch(currentKeyAccumulators, 
currentKeyAccumulatorCharge));
+    queuedKeyCharge += currentKeyAccumulatorCharge;
     currentKeyAccumulators = null;
     currentKeyAccumulatorRowCount = 0;
+    currentKeyAccumulatorCharge = 0L;
   }
 
   /** Releases the pages phase 2 read for the row group just emitted, if any. 
*/
@@ -867,6 +945,28 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
           + "because buffering a surviving key value raised an error. The 
storage filter is still "
           + "applied";
 
+  private static final String SPLICING_GIVEN_UP_ON_CAP =
+      "Reading the key columns of a row group of {} a second time rather than 
buffering them, "
+          + "because the buffer would pass "
+          + 
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES().key()
+          + ". The storage filter is still applied";
+
+  private static final String SPLICING_GIVEN_UP_ON_MEMORY =
+      "Reading the key columns of a row group of {} a second time rather than 
buffering them, "
+          + "because the task's memory manager denied the memory to buffer 
them. The storage "
+          + "filter is still applied";
+
+  private static final String ROW_GROUP_GIVEN_UP_ON_CAP =
+      "Reading a row group of {} without the storage filter, because its 
surviving row ranges "
+          + "alone would pass "
+          + 
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES().key()
+          + ". The filter is still applied above the scan, so the answer is 
unchanged";
+
+  private static final String ROW_GROUP_GIVEN_UP_ON_MEMORY =
+      "Reading a row group of {} without the storage filter, because the 
task's memory manager "
+          + "denied the memory to hold its surviving row ranges. The filter is 
still applied "
+          + "above the scan, so the answer is unchanged";
+
   /**
    * Gives the filter up on an error, after rethrowing a task kill or a fatal 
error the reader must
    * not absorb, see {@code ParquetStorageFilter.rethrowIfMustPropagate}. 
Every catch of
@@ -887,19 +987,43 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
     LOG.warn(message, t, MDC.of(LogKeys.PATH, fileReader.getFile()));
   }
 
+  /**
+   * Counts a row group the survivor budget limited, under the last thing it 
gave up, so each row
+   * group counts once. The first of each kind per split is logged, at INFO 
rather than WARN, since
+   * that is the budget doing its job rather than an error.
+   */
+  private void recordBudgetGiveUp(BudgetGiveUp giveUp) {
+    giveUp.counter.apply(storageFilter.metrics()).add(1L);
+    if (reportedMessages.add(giveUp.message)) {
+      LOG.info(giveUp.message, MDC.of(LogKeys.PATH, fileReader.getFile()));
+    }
+  }
+
   /**
    * Gives up splicing for the row group being evaluated and frees its 
buffered survivors. Phase 2
    * then reads every projected column, so the rows are unaffected.
    */
   private void abandonSplicing() {
-    for (WritableColumnVector[] batch : survivorBatches) {
-      closeAll(batch);
+    for (SurvivorBatch batch : survivorBatches) {
+      closeAll(batch.vectors());
     }
     survivorBatches.clear();
+    queuedKeyCharge = 0L;
     closeAll(currentKeyAccumulators);
     currentKeyAccumulators = null;
     currentKeyAccumulatorRowCount = 0;
+    currentKeyAccumulatorCharge = 0L;
     spliceCurrentRowGroup = false;
+    releaseSurvivorMemoryTo(heldCharge());
+  }
+
+  /**
+   * Frees everything the survivor buffer holds for a row group whose filter 
is given up, its
+   * ranges included, since phase 2 then reads every row the pushed filter 
kept.
+   */
+  private void abandonFilter() {
+    rangeCharge = 0L;
+    abandonSplicing();
   }
 
   /** Closes every survivor vector this reader still holds, the published ones 
included. */
@@ -912,6 +1036,65 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
     }
   }
 
+  /** The bytes of key values buffered for splicing, which the cap weighs even 
off heap. */
+  private long bufferedKeyCharge() {
+    return queuedKeyCharge + currentKeyAccumulatorCharge;
+  }
+
+  /** The accounted bytes the survivor buffer holds, see {@link 
SurvivorMemory}. */
+  private long heldCharge() {
+    return MEMORY_MODE == MemoryMode.ON_HEAP ? rangeCharge + 
bufferedKeyCharge() : rangeCharge;
+  }
+
+  /**
+   * Grows the reservation to cover {@code held} bytes, which the caller has 
kept within the cap.
+   * It asks for at least what it already holds, up to {@link 
#SURVIVOR_MEMORY_STEP_BYTES} and the
+   * cap, so a growing buffer asks only a few times and the reservation never 
passes the cap. False
+   * when it is denied.
+   */
+  private boolean reserveSurvivorMemory(long held) {
+    if (survivorMemory == null) return true;
+    long missing = held - reservedBytes;
+    if (missing <= 0) return true;
+    long headroom = storageFilter.maxSplicedRowGroupBytes() - reservedBytes;
+    long ask = Math.max(missing,
+        Math.min(Math.min(reservedBytes, SURVIVOR_MEMORY_STEP_BYTES), 
headroom));
+    long granted;
+    try {
+      granted = survivorMemory.acquireMemory(ask);
+    } catch (Exception | InternalError e) {
+      // Another consumer's spill and the eviction of cached blocks run 
inside, so this can raise
+      // anything, see `ParquetStorageFilter.memoryReservationFailed`. Any 
other `Error`, such as
+      // the `SparkOutOfMemoryError` of a failed spill, goes out as it is, 
since `FileScanRDD`
+      // cannot take it for a corrupt file.
+      throw ParquetStorageFilter.memoryReservationFailed(e);
+    }
+    if (granted < missing) {
+      if (granted > 0) survivorMemory.freeMemory(granted);
+      return false;
+    }
+    reservedBytes += granted;
+    return true;
+  }
+
+  /** Shrinks the reservation to {@code held} bytes. */
+  private void releaseSurvivorMemoryTo(long held) {
+    long excess = reservedBytes - held;
+    if (excess <= 0) return;
+    survivorMemory.freeMemory(excess);
+    reservedBytes = held;
+  }
+
+  /**
+   * Shrinks the reservation to what the buffer holds once that is a step 
less, so emitting a row
+   * group releases its memory a few times rather than per batch. Each release 
takes the memory
+   * manager's lock and wakes every task waiting on it. The rest goes when the 
next row group loads.
+   */
+  private void trimSurvivorMemory() {
+    long held = heldCharge();
+    if (reservedBytes - held >= SURVIVOR_MEMORY_STEP_BYTES) 
releaseSurvivorMemoryTo(held);
+  }
+
   /** Closes every non-null vector of {@code vectors}, which may itself be 
null. */
   private static void closeAll(ColumnVector[] vectors) {
     if (vectors == null) return;
@@ -920,6 +1103,53 @@ public class LateMaterializationParquetRecordReader 
extends VectorizedParquetRec
     }
   }
 
+  /**
+   * The survivor buffer's account with the task's memory manager, so other 
consumers see what it
+   * holds. The charge grows per row, while survivor vectors are allocated a 
batch at a time, so up
+   * to a batch per key column is held beyond it. A spill frees nothing. The 
buffer is bounded by
+   * {@code maxSplicedRowGroupBytes} and drains as its batches are emitted, 
and freeing it would
+   * have the rest of the row group read its key pages again, which the memory 
manager does not see
+   * either. A denied reservation is handled where it is made, as passing that 
bound is. ON_HEAP
+   * whatever the reader's memory mode, since off-heap vectors allocate native 
memory outside the
+   * Tungsten off-heap pool. With them only the row ranges, which are always 
on heap, are
+   * accounted. The task's memory manager keeps every consumer until the task 
ends, so a task holds
+   * one per file it reserved for.
+   */
+  private static final class SurvivorMemory extends MemoryConsumer {
+    SurvivorMemory(TaskMemoryManager taskMemoryManager) {
+      super(taskMemoryManager, MemoryMode.ON_HEAP);
+    }
+
+    @Override
+    public long spill(long size, MemoryConsumer trigger) {
+      return 0L;
+    }
+  }
+
+  /** One batch's survivor vectors, indexed like {@link #keyColumns}, and what 
they are charged. */
+  private record SurvivorBatch(WritableColumnVector[] vectors, long charge) {}
+
+  /**
+   * What the survivor budget gave up for a row group and the limit that made 
it, with the metric
+   * that counts it and the message logged for it.
+   */
+  private enum BudgetGiveUp {
+    SPLICING_ON_CAP(StorageFilterMetrics::rowGroupsSplicingGivenUpByCap, 
SPLICING_GIVEN_UP_ON_CAP),
+    SPLICING_ON_MEMORY(
+        StorageFilterMetrics::rowGroupsSplicingGivenUpByMemory, 
SPLICING_GIVEN_UP_ON_MEMORY),
+    FILTER_ON_CAP(StorageFilterMetrics::rowGroupsFilterGivenUpByCap, 
ROW_GROUP_GIVEN_UP_ON_CAP),
+    FILTER_ON_MEMORY(
+        StorageFilterMetrics::rowGroupsFilterGivenUpByMemory, 
ROW_GROUP_GIVEN_UP_ON_MEMORY);
+
+    final Function<StorageFilterMetrics, SQLMetric> counter;
+    final String message;
+
+    BudgetGiveUp(Function<StorageFilterMetrics, SQLMetric> counter, String 
message) {
+      this.counter = counter;
+      this.message = message;
+    }
+  }
+
   /** One key column this file has, as init resolves it. */
   private record KeyColumn(
       int batchSlot,
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
index 5f3fbc7801b7..1cbe41083b40 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
@@ -30,12 +30,12 @@ import org.apache.spark.util.Utils
 
 /**
  * The SQL metrics the reader updates while applying a 
[[ParquetStorageFilter]], created by
- * `ParquetFileFormat.storageFilterMetrics`. They count only what the filter 
saved against a read of
- * the same projection without one. A row group whose filter was given up 
counts nothing, so a scan
- * that read more than with the feature off can still show a saving. Nor do 
they take off the offset
- * indexes phase 2 reads when no data filter is pushed, which a plain read 
does not. Bytes are the
- * compressed bytes the reader did not ask parquet for, which a vectored read 
can still fetch when
- * it merges nearby ranges.
+ * `ParquetFileFormat.storageFilterMetrics`. The first five count only what 
the filter saved against
+ * a read of the same projection without one. A row group whose filter was 
given up counts nothing
+ * there, so a scan that read more than with the feature off can still show a 
saving. Nor do they
+ * take off the offset indexes phase 2 reads when no data filter is pushed, 
which a plain read does
+ * not. Bytes are the compressed bytes the reader did not ask parquet for, 
which a vectored read can
+ * still fetch when it merges nearby ranges.
  *
  *  - [[rowGroupsSkipped]] counts row groups whose non-key columns were never 
read.
  *  - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed 
data filter kept.
@@ -47,13 +47,26 @@ import org.apache.spark.util.Utils
  *
  * A row excluded within a row group may still have been read, as part of a 
page that held a
  * survivor.
+ *
+ * The other four count row groups the survivor buffer's budget limited, each 
at most once, under
+ * the last thing it gave up and the limit that made it: 
`maxSplicedRowGroupBytes`, or the task's
+ * memory manager denying the reservation.
+ *
+ *  - [[rowGroupsSplicingGivenUpByCap]] and 
[[rowGroupsSplicingGivenUpByMemory]] count row groups
+ *    that kept the filter but read their key columns a second time.
+ *  - [[rowGroupsFilterGivenUpByCap]] and [[rowGroupsFilterGivenUpByMemory]] 
count row groups read
+ *    without the filter.
  */
 case class StorageFilterMetrics(
     rowGroupsSkipped: SQLMetric,
     rowsExcludedByRowGroup: SQLMetric,
     rowsExcludedWithinRowGroup: SQLMetric,
     bytesAvoidedByRowGroup: SQLMetric,
-    bytesAvoidedByPageFiltering: SQLMetric) {
+    bytesAvoidedByPageFiltering: SQLMetric,
+    rowGroupsSplicingGivenUpByCap: SQLMetric,
+    rowGroupsSplicingGivenUpByMemory: SQLMetric,
+    rowGroupsFilterGivenUpByCap: SQLMetric,
+    rowGroupsFilterGivenUpByMemory: SQLMetric) {
 
   /** Counts a row group whose non-key columns the filter kept the reader from 
reading at all. */
   def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
@@ -71,6 +84,10 @@ object StorageFilterMetrics {
   val ROWS_EXCLUDED_WITHIN_ROW_GROUP = 
"storageFilterRowsExcludedWithinRowGroup"
   val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
   val BYTES_AVOIDED_BY_PAGE_FILTERING = 
"storageFilterBytesAvoidedByPageFiltering"
+  val ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP = 
"storageFilterRowGroupsSplicingGivenUpByCap"
+  val ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY = 
"storageFilterRowGroupsSplicingGivenUpByMemory"
+  val ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP = 
"storageFilterRowGroupsFilterGivenUpByCap"
+  val ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY = 
"storageFilterRowGroupsFilterGivenUpByMemory"
 
   /** A fresh set of counters, keyed for the scan, with the labels the SQL UI 
shows. */
   def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
@@ -83,7 +100,15 @@ object StorageFilterMetrics {
     BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
       sparkContext, "bytes avoided by storage filter (whole row group)"),
     BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
-      sparkContext, "bytes avoided by storage filter (page filtering)"))
+      sparkContext, "bytes avoided by storage filter (page filtering)"),
+    ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP -> SQLMetrics.createMetric(
+      sparkContext, "row groups with key columns read twice by storage filter 
(buffer cap)"),
+    ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY -> SQLMetrics.createMetric(
+      sparkContext, "row groups with key columns read twice by storage filter 
(memory denied)"),
+    ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP -> SQLMetrics.createMetric(
+      sparkContext, "row groups read without storage filter (buffer cap)"),
+    ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY -> SQLMetrics.createMetric(
+      sparkContext, "row groups read without storage filter (memory denied)"))
 
   /** The counters the scan carries back, which [[create]] made. */
   def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics = 
StorageFilterMetrics(
@@ -91,7 +116,11 @@ object StorageFilterMetrics {
     rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
     rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
     bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
-    bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+    bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING),
+    rowGroupsSplicingGivenUpByCap = 
metrics(ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP),
+    rowGroupsSplicingGivenUpByMemory = 
metrics(ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY),
+    rowGroupsFilterGivenUpByCap = metrics(ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP),
+    rowGroupsFilterGivenUpByMemory = 
metrics(ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY))
 }
 
 /**
@@ -295,6 +324,19 @@ object ParquetStorageFilter {
     }
   }
 
+  /**
+   * Throws what reserving the survivor buffer's memory raised, in a form 
`FileScanRDD` cannot take
+   * for a corrupt or missing file. The task's memory manager runs other 
consumers' spills inside
+   * the reservation, and can evict cached blocks to disk there, so it can 
raise anything, an
+   * `IOException` included. What [[rethrowIfMustPropagate]] rethrows goes out 
as it does there, and
+   * anything else wrapped in a checked [[SparkException]]. Typed as returning 
a `RuntimeException`
+   * so Java code can write `throw memoryReservationFailed(e)`.
+   */
+  def memoryReservationFailed(e: Throwable): RuntimeException = {
+    rethrowIfMustPropagate(e, fromRead = false)
+    throw new SparkException("A storage filter failed to reserve memory", e)
+  }
+
   /**
    * Throws `SparkException.internalError(message)` for a broken invariant of 
this reader. It is
    * checked, so `FileScanRDD` cannot take it for a corrupt file. Typed as 
returning a
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
index 2179ac12be11..07d0f9dfae5b 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
@@ -17,10 +17,10 @@
 
 package org.apache.spark.sql.execution.datasources.parquet
 
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, 
RandomAccessFile}
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File, 
IOException, RandomAccessFile}
 import java.net.URI
 import java.time.LocalTime
-import java.util.Locale
+import java.util.{Locale, Properties}
 import java.util.concurrent.atomic.AtomicLong
 
 import scala.collection.mutable
@@ -47,8 +47,9 @@ import org.mockito.ArgumentMatchers.{any, anyInt}
 import org.mockito.Mockito.{doAnswer, doThrow, spy}
 import org.mockito.invocation.InvocationOnMock
 
-import org.apache.spark.{SparkEnv, SparkException, TaskContext, 
TaskKilledException}
+import org.apache.spark.{SparkEnv, SparkException, TaskContext, 
TaskContextImpl, TaskKilledException}
 import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.memory.{MemoryConsumer, MemoryManager, MemoryMode, 
TaskMemoryManager, TestMemoryConsumer, TestMemoryManager}
 import org.apache.spark.paths.SparkPath
 import org.apache.spark.sql.{classic, sources, DataFrame, QueryTest, Row, 
SparkSession}
 import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
@@ -73,7 +74,7 @@ import org.apache.spark.util.sketch.BloomFilter
 /**
  * Tests [[LateMaterializationParquetRecordReader]] driven by a 
[[ParquetStorageFilter]]. Writes
  * small multi-row-group parquet files, wires a hand-built filter into the 
reader, and asserts
- * correctness and the five storage-filter metrics.
+ * correctness and the storage-filter metrics.
  */
 class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
   with AdaptiveSparkPlanHelper {
@@ -197,18 +198,21 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       s"a cap one byte under it must give splicing up; got $underCharge 
against $uncapped")
   }
 
-  // Reads `path` whole through the format under a splice cap and a batch 
size, and returns the rows
-  // it emitted with the bytes it transferred.
+  // Reads `path` whole through the format under a splice cap and a batch 
size, as a task of
+  // `context` if one is given, and returns the rows it emitted with the bytes 
it transferred.
   private def readUnderCap(
       path: String,
       schema: StructType,
       storageFilters: Seq[Expression],
       maxSplicedBytes: String,
       batchSize: Int,
-      metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) = 
withSQLConf(
+      metrics: Map[String, SQLMetric] = metricMap(),
+      context: Option[TaskContext] = None): (Int, Long) = withSQLConf(
       SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key 
-> maxSplicedBytes,
       SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
-    readCounting(path, formatReader(schema, Nil, storageFilters, 
countingHadoopConf(), metrics))
+    def read(): (Int, Long) = readCounting(
+      path, formatReader(schema, Nil, storageFilters, countingHadoopConf(), 
metrics))
+    context.fold(read())(asTask(_)(read()))
   }
 
   // Reads `path` whole through `readerFn`, and returns the long column at 
`ordinal` of every row.
@@ -254,6 +258,12 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
   // What the reader logs when filling the survivor buffer fails and it gives 
splicing up.
   private val splicingGivenUpOnError = "buffering a surviving key value raised 
an error"
 
+  // What the reader logs when the survivor buffer's budget gives splicing or 
the filter up.
+  private val splicingGivenUpOnCap = "because the buffer would pass"
+  private val splicingGivenUpOnMemory = "denied the memory to buffer them"
+  private val filterGivenUpOnCap = "its surviving row ranges alone would pass"
+  private val filterGivenUpOnMemory = "denied the memory to hold its surviving 
row ranges"
+
   // What the reader logs when applying the filter to a row, or decoding the 
key pages, raises an
   // error and it gives the filter up for that row group.
   private val evaluationGivenUp = "applying it to a row raised an error"
@@ -273,6 +283,14 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
   private def allMetrics(): StorageFilterMetrics =
     StorageFilterMetrics.fromMap(metricMap())
 
+  // The row groups the survivor budget limited, as counted in `metrics`: 
splicing given up by the
+  // cap and by the memory manager, then the filter given up by each.
+  private def budgetGiveUps(metrics: Map[String, SQLMetric]): Seq[Long] = Seq(
+    StorageFilterMetrics.ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP,
+    StorageFilterMetrics.ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY,
+    StorageFilterMetrics.ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP,
+    
StorageFilterMetrics.ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY).map(metrics(_).value)
+
   // `ParquetStorageFilter.create` takes what the production path always 
supplies; most tests care
   // about neither the metrics nor the cap.
   private def createFilter(
@@ -1081,17 +1099,24 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     withTempDir { dir =>
       val path = writePagedFile(dir)
 
-      def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+      // Also returns the row groups the budget limited, see `budgetGiveUps`.
+      def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long, 
Seq[Long]) = {
         val metrics = metricMap()
         val (rows, bytes) = readUnderCap(path, kvSchema(), 
Seq(keyAtLeast(threshold)),
           maxSplicedBytes, batchSize = 16, metrics)
-        (rows, bytes, 
metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+        (rows, bytes, 
metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value,
+          budgetGiveUps(metrics))
       }
 
-      val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+      val (splicedRows, splicedBytes, splicedAvoided, splicedGiveUps) = 
run("64MB", 350L)
       // `k >= 350` keeps one contiguous range of 40 bytes, and a long key 
buffers 9 bytes per
       // survivor, so a 100-byte cap is passed at the seventh.
-      val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+      val ((plainRows, plainBytes, plainAvoided, plainGiveUps), logged) =
+        withLogged(splicingGivenUpOnCap)(run("100b", 350L))
+      assert(splicedGiveUps == Seq(0L, 0L, 0L, 0L), s"got $splicedGiveUps")
+      assert(plainGiveUps == Seq(1L, 0L, 0L, 0L),
+        s"the row group must count as giving splicing up on the cap; got 
$plainGiveUps")
+      assert(logged == 1, s"and log it; got $logged")
       assert(splicedRows == 51, s"the filter keeps keys 350..400; got 
$splicedRows")
       assert(plainRows == splicedRows,
         s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
@@ -1107,8 +1132,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       // Eight survivors buffer 72 bytes and fall in one range of 40, so 
neither half reaches the
       // 100-byte cap on its own while the sum passes it at the seventh. 
Weighing them separately
       // would keep splicing here, and report the larger saving for it.
-      val (_, _, eightSpliced) = run("64MB", 393L)
-      val (_, _, eightPlain) = run("100b", 393L)
+      val (_, _, eightSpliced, _) = run("64MB", 393L)
+      val (_, _, eightPlain, _) = run("100b", 393L)
       assert(eightSpliced > 0, s"and avoid something with eight survivors; got 
$eightSpliced")
       assert(eightPlain < eightSpliced,
         s"the sum of the two halves must cross the cap; plain=$eightPlain 
spliced=$eightSpliced")
@@ -1116,7 +1141,7 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       // The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103 
bytes, which a cap of
       // 103 holds and one of 102 does not. A charge one byte off either way 
flips one of the two.
       assertSpliceCharge(103L, rows = 7) { cap =>
-        val (rows, bytes, _) = run(cap, 394L)
+        val (rows, bytes, _, _) = run(cap, 394L)
         (rows, bytes)
       }
     }
@@ -1143,7 +1168,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       val rows = rows400
       val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
       assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row 
groups")
-      val m = allMetrics()
+      val metrics = metricMap()
+      val m = StorageFilterMetrics.fromMap(metrics)
       val k = BoundReference(0, LongType, nullable = true)
       def between(lo: Long, hi: Long): Expression =
         And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, 
Literal(hi)))
@@ -1179,6 +1205,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
           s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got 
$splicedByRowGroup")
         assert(m.rowGroupsSkipped.value == 1,
           s"expected the row group with no survivor skipped; got 
${m.rowGroupsSkipped.value}")
+        assert(budgetGiveUps(metrics) == Seq(1L, 0L, 0L, 0L),
+          s"and the one over the cap counted once; got 
${budgetGiveUps(metrics)}")
       } finally {
         reader.close()
       }
@@ -1626,12 +1654,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     // `UnknownError`. Off heap, the scratch vectors the failed decode used 
must be freed, and the
     // second row group's phase 1 runs over the ones that replace them.
     withTempDir { dir =>
+      val path = twoRowGroupsOf20(dir)
       val rows = (0L until 40L).map(i => (i, i.toString))
-      val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
-        rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
-        options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
-      assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 
20L),
-        "the fixture must be two row groups of 20 rows")
       Seq(
         new RuntimeException("Cannot reserve", new 
OutOfMemoryError("reserve")),
         new InternalError("codec")).foreach { error =>
@@ -1849,14 +1873,20 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       val everyOtherRow = EqualTo(
         Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), 
Literal(0L))
 
-      def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
-        val filter =
-          createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+      // Also returns the row groups the budget limited, see `budgetGiveUps`.
+      def readWithCap(cap: Long, capacity: Int = 4096): (Seq[(Long, String)], 
Seq[Long]) = {
+        val metrics = metricMap()
+        val filter = createFilter(
+          Seq(everyOtherRow), fileSchema, 
StorageFilterMetrics.fromMap(metrics), cap)
         val (emitted, reader) = readAll(path, filter, capacity)
-        try emitted finally reader.close()
+        try {
+          (emitted, budgetGiveUps(metrics))
+        } finally {
+          reader.close()
+        }
       }
 
-      assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+      assert(readWithCap(64L * 1024 * 1024) == (rows.filter(_._1 % 2 == 0), 
Seq(0L, 0L, 0L, 0L)),
         "with room for the ranges, every surviving row comes back with its own 
value")
       // Each survivor costs 9 bytes of long key and 40 of its own range. The 
sum passes the cap of
       // 1024 at survivor 21, which releases the buffer, and the ranges alone 
at survivor 26, which
@@ -1865,9 +1895,12 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       // well. Both have to end with the queue empty and the filter given up. 
Keeping one without
       // the other would leave phase 2 reading non-key columns while emit 
spliced keys from a queue
       // holding only the first survivors.
-      assert(readWithCap(1024L) == rows,
+      // The row group counts once, as giving the filter up, though it gave 
splicing up first.
+      val (pastCap, logged) = 
withLogged(filterGivenUpOnCap)(readWithCap(1024L))
+      assert(pastCap == (rows, Seq(0L, 0L, 1L, 0L)),
         "past the cap the filter is given up and the row group comes back 
whole")
-      assert(readWithCap(1024L, capacity = 16) == rows,
+      assert(logged == 1, s"and that is logged; got $logged")
+      assert(readWithCap(1024L, capacity = 16) == (rows, Seq(0L, 0L, 1L, 0L)),
         "and the same when the buffer has been rolled over first")
     }
   }
@@ -1884,7 +1917,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
       val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
         EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
       val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), 
LessThanOrEqual(k, Literal(320L)))
-      val m = allMetrics()
+      val metrics = metricMap()
+      val m = StorageFilterMetrics.fromMap(metrics)
       val filter = createFilter(
         Seq(Or(scatteredInFirst, runInLast)),
         kvSchema(),
@@ -1898,6 +1932,8 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
           s"expected the given-up row group whole and the last filtered; got 
${result.size} rows")
         assert(m.rowGroupsSkipped.value == 2,
           s"and the two emptied row groups skipped; got 
${m.rowGroupsSkipped.value}")
+        assert(budgetGiveUps(metrics) == Seq(0L, 0L, 1L, 0L),
+          s"and the first counted once; got ${budgetGiveUps(metrics)}")
       } finally {
         reader.close()
       }
@@ -1950,20 +1986,21 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     kvSchema(valueType = 
valueType).add(FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME, BooleanType)
 
   // Reads `path` through the format, the way a scan with `storageFilters` 
over `schema` does, under
-  // the SQL confs in `conf`. Returns each row's key, as `keyOf` reads it, 
with what the checked
-  // column says for the row.
+  // the SQL confs in `conf`, and as a task of `context` if one is given. The 
confs are set outside
+  // the task, which sees a read-only one. Returns each row's key, as `keyOf` 
reads it, with what
+  // the checked column says for the row.
   private def readChecked(
       path: String,
       schema: StructType,
       storageFilters: Seq[Expression],
       conf: Map[String, String] = Map.empty,
-      keyOf: (ColumnarBatch, Int) => Long = (b, i) => b.column(0).getLong(i))
-      : Seq[(Long, Boolean)] = withSQLConf(conf.toSeq: _*) {
+      keyOf: (ColumnarBatch, Int) => Long = (b, i) => b.column(0).getLong(i),
+      context: Option[TaskContext] = None): Seq[(Long, Boolean)] = 
withSQLConf(conf.toSeq: _*) {
     val checked = 
schema.fieldIndex(FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME)
     val file = PartitionedFile(
       InternalRow.empty, SparkPath.fromPathString(path), 0, new 
File(path).length())
-    drain(formatReader(schema, Nil, storageFilters)(file)) { it =>
-      it.flatMap { case batch: ColumnarBatch =>
+    def read(): Seq[(Long, Boolean)] = drain(formatReader(schema, Nil, 
storageFilters)(file)) {
+      it => it.flatMap { case batch: ColumnarBatch =>
         val marks = batch.column(checked)
         (0 until batch.numRows()).map { i =>
           assert(!marks.isNullAt(i), "the reader must mark every row")
@@ -1971,12 +2008,44 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
         }
       }.toSeq
     }
+    context.fold(read())(asTask(_)(read()))
   }
 
   // The keys in `keys`, each with the mark `checked`.
   private def marked(keys: Seq[Long], checked: Boolean): Seq[(Long, Boolean)] =
     keys.map(_ -> checked)
 
+  // 40 rows of a long key and a string value in two row groups of 20. A 
filter of `k >= 10` keeps
+  // one range of 10 rows in the first and all 20 rows of the second. A 
buffered long key costs 9
+  // bytes and a range 40, so the first row group holds 130 bytes and the 
second 220.
+  private def twoRowGroupsOf20(dir: File): String = {
+    val path = writeSingleParquetFile(dir, (0L until 40L).map(i => (i, 
i.toString)).toDF("k", "v"),
+      rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+      options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+    assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 
20L),
+      "the fixture must be two row groups of 20 rows")
+    path
+  }
+
+  // Three row groups of 100 long keys and values, `(i, i)` for i from 1 to 
300.
+  private val threeRowGroupsOf100: Seq[Seq[(Long, Long)]] =
+    (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+
+  // Keeps the rows whose key, at ordinal 0, is even.
+  private val evenKey: Expression =
+    EqualTo(Remainder(BoundReference(0, LongType, nullable = true), 
Literal(2L)), Literal(0L))
+
+  // `threeRowGroupsOf100` with the middle row group's value offset index 
damaged. Phase 2 needs it
+  // to read that column over the survivors, and the retry then reads the row 
group whole. Returns
+  // the path, with the rows a read under `evenKey` returns.
+  private def damagedMiddleRowGroup(dir: File): (String, Seq[(Long, Boolean)]) 
= {
+    val path = writeParquetFileByHand(dir, threeRowGroupsOf100, (_, _) => true)
+    breakOffsetIndex(path, blockIdx = 1, columnIdx = 1)
+    val keys = (block: Int) => threeRowGroupsOf100(block).map(_._1)
+    val even = (block: Int) => keys(block).filter(_ % 2 == 0)
+    (path, marked(even(0), checked = true) ++ marked(keys(1), false) ++ 
marked(even(2), true))
+  }
+
   test("the checked column is true exactly for the rows of a row group the 
filter was applied to") {
     // The post-scan Filter skips the storage filter on a row marked true, so 
true must mean the
     // filter kept that row. A row group the filter was given up on comes back 
whole and must be
@@ -1984,12 +2053,7 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     // give-up in the first leaves the second its own. Batches of 4 rows split 
each row group, and
     // every batch of it must be marked alike.
     withTempDir { dir =>
-      val rows = (0L until 40L).map(i => (i, i.toString))
-      val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
-        rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
-        options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
-      assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L, 
20L),
-        "the fixture must be two row groups of 20 rows")
+      val path = twoRowGroupsOf20(dir)
       val schema = checkedSchema()
       val k = BoundReference(0, LongType, nullable = true)
       def filtersOn(key: Expression): Seq[Expression] = 
Seq(GreaterThanOrEqual(key, Literal(10L)))
@@ -2023,31 +2087,44 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     // A row group the footer declines, with no offset index, and one whose 
offset index parquet
     // then fails to parse both come back whole, the latter through phase 2's 
retry.
     withTempDir { dir =>
-      val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 
100L)).map(i => (i, i)))
       val schema = checkedSchema(valueType = LongType)
-      val evenKeys = Seq(EqualTo(
-        Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), 
Literal(0L)))
-      def keys(block: Int): Seq[Long] = blocks(block).map(_._1)
-
-      val noIndex = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_, 
_) => false)
-      assert(readChecked(noIndex, schema, evenKeys) == marked((0 until 
3).flatMap(keys), false),
+      val noIndex =
+        writeParquetFileByHand(dir, threeRowGroupsOf100, hasOffsetIndex = (_, 
_) => false)
+      assert(readChecked(noIndex, schema, Seq(evenKey)) ==
+        marked(threeRowGroupsOf100.flatten.map(_._1), false),
         "every row group of a file with no offset index must come back whole 
and marked false")
 
-      // The middle row group's value index is damaged. Phase 2 needs it to 
read that column over
-      // the survivors, and the retry then reads the row group whole.
-      val damaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
-      breakOffsetIndex(damaged, blockIdx = 1, columnIdx = 1)
+      val (damaged, expected) = damagedMiddleRowGroup(dir)
       val (result, warnings) = withLogged(partialReadWarning) {
-        readChecked(damaged, schema, evenKeys)
+        readChecked(damaged, schema, Seq(evenKey))
       }
       assert(warnings == 1, s"the read must have gone through the retry; got 
$warnings")
-      val even = (block: Int) => keys(block).filter(_ % 2 == 0)
-      assert(result ==
-        marked(even(0), checked = true) ++ marked(keys(1), false) ++ 
marked(even(2), true),
+      assert(result == expected,
         "the damaged row group must come back whole and marked false, and the 
others filtered")
     }
   }
 
+  test("a row group whose filter phase 2 gives up does not count as giving 
splicing up") {
+    // At a cap of 2,100 bytes, each row group's 50 scattered even keys pass 
it at the 43rd
+    // survivor, at 43 x 49 = 2,107 bytes, while their 2,000 bytes of ranges 
fit. So each gives
+    // splicing up and keeps the filter. The middle row group's damaged value 
index then has phase
+    // 2 give the filter up as well, which the splicing give-up must not be 
counted for. That row
+    // group then holds nothing while it is emitted.
+    withTempDir { dir =>
+      val (damaged, expected) = damagedMiddleRowGroup(dir)
+      val schema = checkedSchema(valueType = LongType)
+      val metrics = metricMap()
+      readUnderCap(damaged, schema, Seq(evenKey), "2100b", batchSize = 4, 
metrics)
+      assert(budgetGiveUps(metrics) == Seq(2L, 0L, 0L, 0L),
+        s"only the two row groups that kept the filter count; got 
${budgetGiveUps(metrics)}")
+      val (rows, held) = readInTask(damaged, memoryTask(), cap = "2100b", 
filter = evenKey,
+        schema = schema)
+      assert(rows == expected, s"got $rows")
+      assert(held == Seq.fill(13)(2000L) ++ Seq.fill(25)(0L) ++ 
Seq.fill(13)(2000L),
+        s"each batch must hold its row group's ranges, and none for the one 
read whole; got $held")
+    }
+  }
+
   test("the checked column for a file the filter is not applied to row by 
row") {
     withTempDir { dir =>
       val schema = checkedSchema()
@@ -2119,6 +2196,312 @@ class ParquetStorageFilterSuite extends QueryTest with 
SharedSparkSession
     }
   }
 
+  // A task memory manager that calls `onAcquire` before each acquisition but 
a test consumer's,
+  // with what is asked for, what the task holds then, and the consumer's 
memory mode.
+  private class ObservedTaskMemoryManager(
+      memoryManager: MemoryManager,
+      onAcquire: (Long, Long, MemoryMode) => Unit) extends 
TaskMemoryManager(memoryManager, 0) {
+    override def acquireExecutionMemory(required: Long, consumer: 
MemoryConsumer): Long = {
+      if (!consumer.isInstanceOf[TestMemoryConsumer]) {
+        onAcquire(required, getMemoryConsumptionForThisTask, consumer.getMode)
+      }
+      super.acquireExecutionMemory(required, consumer)
+    }
+  }
+
+  // A task whose memory manager grants at most `limit` bytes, and calls 
`onAcquire` as
+  // `ObservedTaskMemoryManager` does.
+  private def memoryTask(
+      limit: Long = Long.MaxValue,
+      onAcquire: (Long, Long, MemoryMode) => Unit = (_, _, _) => ()): 
TaskContextImpl = {
+    val memoryManager = new TestMemoryManager(spark.sparkContext.conf)
+    memoryManager.limit(limit)
+    new TaskContextImpl(stageId = 0, stageAttemptNumber = 0, partitionId = 0, 
taskAttemptId = 0,
+      attemptNumber = 0, numPartitions = 1,
+      taskMemoryManager = new ObservedTaskMemoryManager(memoryManager, 
onAcquire),
+      localProperties = new Properties, metricsSystem = null)
+  }
+
+  // Reads `path` under `filter` in batches of `batchSize` and a cap of `cap`, 
as a task of
+  // `context`. Returns the rows, and what the task holds as each batch is 
read.
+  private def readInTask(
+      path: String,
+      context: TaskContext,
+      offHeap: Boolean = false,
+      batchSize: Int = 4,
+      cap: String = "64MB",
+      filter: Expression = keyAtLeast(10L),
+      schema: StructType = checkedSchema()): (Seq[(Long, Boolean)], Seq[Long]) 
= {
+    var held = Seq.empty[Long]
+    val rows = readChecked(path, schema, Seq(filter), conf = Map(
+        SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString,
+        SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> offHeap.toString,
+        
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> cap),
+      keyOf = (b, i) => {
+        if (i == 0) held :+= 
context.taskMemoryManager().getMemoryConsumptionForThisTask
+        b.column(0).getLong(i)
+      },
+      context = Some(context))
+    (rows, held)
+  }
+
+  // Reads `path` under `k >= 10` while another consumer holds all of the 
task's memory but
+  // `margin`, and checks that the read neither drops rows nor spills that 
consumer.
+  private def readBesideOther(
+      path: String,
+      rows: Long,
+      margin: Long,
+      cap: String = "64MB"): Unit = {
+    val held = 1000000L
+    val context = memoryTask(held + margin)
+    val other = new TestMemoryConsumer(context.taskMemoryManager)
+    other.use(held)
+    val (read, _) = readInTask(path, context, batchSize = 4096, cap = cap)
+    val name = s"$rows rows with a margin of $margin under a cap of $cap"
+    assert(read == marked(10L until rows, checked = true), s"$name: got 
${read.size} rows")
+    assert(other.getUsed == held, s"$name: the read must not spill another 
consumer")
+  }
+
+  // One row group of 400,000 rows of a long key and a string value. Under `k 
>= 10` its 399,990
+  // survivors fall in one range and buffer 3,599,950 bytes, several steps of 
the reservation.
+  private def oneRowGroupOf400000(dir: File): String = {
+    val path = writeSingleParquetFile(dir,
+      spark.range(400000L).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+      rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE)
+    assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row 
group")
+    path
+  }
+
+  test("the survivor buffer is accounted to the task's execution memory") {
+    // What a row group buffers is held while it is emitted, and released when 
the next one loads.
+    // Phase 1 reserves a little ahead of what it buffers, and gives that back 
before phase 2. Off
+    // heap, only the ranges are accounted, since the vectors' native memory 
is not the memory
+    // manager's.
+    withTempDir { dir =>
+      val path = twoRowGroupsOf20(dir)
+      // Past a cap of 100 bytes, at the seventh survivor of each row group, 
the keys go, a full
+      // batch of them already queued, and only the range is held.
+      Seq((false, "64MB", Seq(130L, 130L, 130L, 220L, 220L, 220L, 220L, 220L)),
+          (true, "64MB", Seq.fill(8)(40L)),
+          (false, "100b", Seq.fill(8)(40L))).foreach { case (offHeap, cap, 
expected) =>
+        var modes = Set.empty[MemoryMode]
+        val context = memoryTask(onAcquire = (_, _, mode) => modes += mode)
+        val (rows, held) = readInTask(path, context, offHeap, cap = cap)
+        val name = s"off heap $offHeap under a cap of $cap"
+        assert(rows == marked(10L until 40L, checked = true), s"$name: got 
$rows")
+        // The off-heap pool does not hold the vectors' native memory, nor the 
ranges.
+        assert(modes == Set(MemoryMode.ON_HEAP), s"$name: every ask must be on 
heap; got $modes")
+        assert(held == expected,
+          s"$name: each batch must hold its row group's reservation until the 
next row group " +
+            s"loads; got $held")
+        assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0,
+          s"$name: not released")
+      }
+      // On heap, each row group asks first for its first survivor's 49 bytes, 
with nothing held
+      // since the previous one's release, and then for as much again as it 
holds, three and four
+      // times for the 130 and 220 bytes it reaches.
+      var asked = Seq.empty[(Long, Long)]
+      readInTask(path, memoryTask(onAcquire = (size, held, _) => asked :+= 
((size, held))))
+      assert(asked == Seq((49L, 0L), (49L, 49L), (98L, 98L), (49L, 0L), (49L, 
49L), (98L, 98L),
+        (196L, 196L)), s"got $asked")
+      // A reader closed after its first batch, as under a limit, releases 
what it still holds.
+      val context = memoryTask()
+      val file = PartitionedFile(
+        InternalRow.empty, SparkPath.fromPathString(path), 0, new 
File(path).length())
+      val heldAtFirst = 
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+        asTask(context) {
+          drain(formatReader(checkedSchema(), Nil, 
Seq(keyAtLeast(10L)))(file)) { it =>
+            it.next()
+            context.taskMemoryManager.getMemoryConsumptionForThisTask
+          }
+        }
+      }
+      assert(heldAtFirst == 130L,
+        s"the first batch must hold the first row group; got $heldAtFirst")
+      assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0,
+        "a reader closed early must release what it holds")
+    }
+  }
+
+  test("emitting a row group releases its memory in steps") {
+    // Each batch emitted frees its survivors, but the reservation is cut only 
once it is a step
+    // past what is still buffered, so a large row group gives its memory back 
a few times rather
+    // than per batch.
+    withTempDir { dir =>
+      val path = oneRowGroupOf400000(dir)
+      var asked = Seq.empty[Long]
+      val context = memoryTask(onAcquire = (size, _, _) => asked :+= size)
+      val (rows, reserved) = readInTask(path, context, batchSize = 4096)
+      assert(rows.size == 399990, s"got ${rows.size} rows")
+      // What batch `b` still holds, its own survivors and those after it at 9 
bytes each, and the
+      // range. A batch frees 36,864 bytes, so the reservation steps down 
every 29 batches.
+      def held(b: Int): Long = 40L + 9L * (399990L - 4096L * b)
+      val expected = (0 until 98).map(b => held(b - b % 29))
+      assert(reserved == expected, s"expected the reservation to step down at 
batches 29, 58 " +
+        s"and 87; got ${reserved.distinct}")
+      assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0, 
"not released")
+      // A growing buffer asks a few times. First for the first survivor, then 
for as much again as
+      // it holds up to 802,816 bytes, and then 1 MiB at a time.
+      assert(asked == Seq(49L) ++ (0 to 14).map(49L << _) ++ Seq.fill(2)(1L << 
20),
+        s"got ${asked.size} asks: ${asked.take(20)}")
+      // The 3,599,950 bytes it ends up buffering are not asked for twice 
over. Asking for as much
+      // again as it holds would spill the other consumer.
+      readBesideOther(path, rows = 400000L, margin = 3599950L + (1L << 20))
+    }
+  }
+
+  test("memory the task's memory manager denies gives splicing up, and then 
the filter") {
+    withTempDir { dir =>
+      val path = twoRowGroupsOf20(dir)
+      // Each read must leave the task holding nothing.
+      def released(context: TaskContext, available: Long): Unit =
+        assert(context.taskMemoryManager().getMemoryConsumptionForThisTask == 
0,
+          s"a read with $available bytes free must release what it reserved")
+      def rows(available: Long, offHeap: Boolean = false): Seq[(Long, 
Boolean)] = {
+        val context = memoryTask(available)
+        val (read, _) = readInTask(path, context, offHeap)
+        released(context, available)
+        read
+      }
+      // The bytes a read transfers, with the row groups the budget limited, 
see `budgetGiveUps`.
+      def bytes(
+          available: Long,
+          cap: String = "64MB",
+          filter: Expression = keyAtLeast(10L)): (Long, Seq[Long]) = {
+        val metrics = metricMap()
+        val context = memoryTask(available)
+        val (_, read) = readUnderCap(path, checkedSchema(), Seq(filter), cap,
+          batchSize = 4, metrics, Some(context))
+        released(context, available)
+        (read, budgetGiveUps(metrics))
+      }
+      val plain = rows(Long.MaxValue)
+      assert(plain == marked(10L until 40L, checked = true), s"got $plain")
+      // 105 bytes hold the 40 of the first row group's range and 7 of its 
keys, but not the 8th.
+      // So splicing is given up there, and in the second row group, as a cap 
of 105 bytes gives
+      // it up. The read must cost exactly what the cap does, the key column 
read again, while the
+      // filter stays. The metrics tell the two causes apart.
+      val keysDenied = rows(105L)
+      assert(keysDenied == plain, s"denied keys must keep the filter; got 
$keysDenied")
+      // 50 bytes take the first survivor's 49, and then grant 1 of the 9 the 
second one misses,
+      // which goes back.
+      assert(rows(50L) == plain, "a partly granted reservation must keep the 
filter")
+      // 130 bytes grant the seventh survivor 32 of the 98 it asks for, which 
covers the 5 it
+      // misses, so the first row group keeps splicing. The second is denied 
at its eleventh.
+      val (_, partlyGranted) = bytes(130L)
+      assert(partlyGranted == Seq(0L, 1L, 0L, 0L),
+        s"only the second row group must be denied; got $partlyGranted")
+      val (uncappedBytes, uncapped) = bytes(Long.MaxValue)
+      assert(uncapped == Seq(0L, 0L, 0L, 0L), s"got $uncapped")
+      val (cappedBytes, capped) = bytes(Long.MaxValue, cap = "105b")
+      assert(cappedBytes > uncappedBytes, "a cap of 105 bytes must give 
splicing up")
+      assert(capped == Seq(2L, 0L, 0L, 0L), s"both row groups pass the cap; 
got $capped")
+      val ((keysDeniedBytes, denied), keysLogged) =
+        withLogged(splicingGivenUpOnMemory)(bytes(105L))
+      assert(keysDeniedBytes == cappedBytes,
+        s"denied keys must read what the cap reads; got $keysDeniedBytes 
against $cappedBytes")
+      assert(denied == Seq(0L, 2L, 0L, 0L), s"both row groups are denied; got 
$denied")
+      assert(keysLogged == 1, s"which is logged once per split; got 
$keysLogged")
+      // Nothing at all does not hold the ranges either, so the filter is 
given up. Off heap only
+      // the ranges are asked for, so 40 bytes keep the filter and nothing 
gives it up.
+      assert(rows(0L) == marked(0L until 40L, checked = false),
+        "denied ranges must give the filter up")
+      val ((_, rangesDenied), rangesLogged) = 
withLogged(filterGivenUpOnMemory)(bytes(0L))
+      assert(rangesDenied == Seq(0L, 0L, 0L, 2L),
+        s"both row groups count as giving the filter up, not splicing; got 
$rangesDenied")
+      assert(rangesLogged == 1, s"which is logged once per split; got 
$rangesLogged")
+      assert(rows(40L, offHeap = true) == plain, "off heap, ranges that fit 
must keep the filter")
+      assert(rows(0L, offHeap = true) == marked(0L until 40L, checked = false),
+        "off heap, denied ranges must give the filter up")
+      // Since off heap a denied request holds no key values, each row group 
asks only once.
+      var offHeapAsks = Seq.empty[Long]
+      readInTask(path, memoryTask(0L, (size, _, _) => offHeapAsks :+= size), 
offHeap = true)
+      assert(offHeapAsks == Seq(40L, 40L), s"got $offHeapAsks")
+      // Under `k >= 19` the first row group's one survivor is its last. 
Denied with its key and
+      // again without it, it must give the filter up.
+      val (_, lastDenied) = bytes(0L, filter = keyAtLeast(19L))
+      assert(lastDenied == Seq(0L, 0L, 0L, 2L), s"got $lastDenied")
+    }
+  }
+
+  test("a row group whose filter phase 1 gives up holds no memory while it is 
read whole") {
+    // Under `evenKey` each survivor of `twoRowGroupsOf20` is a range of its 
own and 9 bytes of
+    // key. At a cap of 300 bytes the seventh gives splicing up and the eighth 
the filter, with 280
+    // bytes of ranges reserved. With 100 bytes free the third is denied with 
its key and then
+    // without it, with 98 bytes reserved. Either way the row group is then 
read whole.
+    withTempDir { dir =>
+      val path = twoRowGroupsOf20(dir)
+      Seq(("300b", Long.MaxValue, Seq(0L, 0L, 2L, 0L)), ("64MB", 100L, Seq(0L, 
0L, 0L, 2L)))
+        .foreach { case (cap, available, expected) =>
+          val name = s"under a cap of $cap with $available bytes free"
+          val metrics = metricMap()
+          readUnderCap(path, checkedSchema(), Seq(evenKey), cap, batchSize = 
4, metrics,
+            Some(memoryTask(available)))
+          assert(budgetGiveUps(metrics) == expected, s"$name: got 
${budgetGiveUps(metrics)}")
+          val (rows, held) = readInTask(path, memoryTask(available), cap = 
cap, filter = evenKey)
+          assert(rows == marked(0L until 40L, checked = false), s"$name: got 
$rows")
+          assert(held == Seq.fill(10)(0L), s"$name: nothing may stay reserved; 
got $held")
+        }
+    }
+  }
+
+  test("an error raised while reserving fails the read in a form 
ignoreCorruptFiles keeps") {
+    // Reserving the buffer can have the task's memory manager spill another 
consumer, which can
+    // raise anything. A `RuntimeException`, which a kill's interrupt during a 
spill also becomes,
+    // goes out checked, and an `InternalError` in a fatal `UnknownError`. 
Under
+    // `ignoreCorruptFiles` either would otherwise be taken for a corrupt 
file, and the rest of the
+    // file dropped. A fatal error in the cause chain goes out as itself.
+    withTempDir { dir =>
+      val path = twoRowGroupsOf20(dir)
+      // Reads beside another consumer whose spill throws `error`, and returns 
what the read throws.
+      def readBesideFailing(error: Throwable): Throwable = {
+        val context = memoryTask(1000L)
+        val failing = new TestMemoryConsumer(context.taskMemoryManager) {
+          override def spill(size: Long, trigger: MemoryConsumer): Long = 
throw error
+        }
+        failing.use(1000L)
+        intercept[Throwable](readInTask(path, context))
+      }
+      Seq(new RuntimeException("spill"), new InternalError("spill")).foreach { 
error =>
+        val thrown = readBesideFailing(error)
+        val expected = if (error.isInstanceOf[InternalError]) {
+          classOf[UnknownError]
+        } else {
+          classOf[SparkException]
+        }
+        assert(thrown.getClass == expected && (thrown.getCause eq error),
+          s"expected $error inside ${expected.getSimpleName}; got $thrown")
+        assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), 
s"got $thrown")
+      }
+      val deep = new OutOfMemoryError("deep")
+      val thrownDeep = readBesideFailing(new RuntimeException("spill", deep))
+      assert(thrownDeep eq deep, s"expected $deep itself; got $thrownDeep")
+      // Evicting a cached block to disk can raise an `IOException` there too, 
which the task's
+      // memory manager passes on as it is.
+      val full = new IOException("No space left on device")
+      val evicting = memoryTask(onAcquire = (_, _, _) => throw full)
+      val thrown = intercept[Throwable](readInTask(path, evicting))
+      assert(thrown.getClass == classOf[SparkException] && (thrown.getCause eq 
full),
+        s"expected $full inside a SparkException; got $thrown")
+      assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got 
$thrown")
+    }
+  }
+
+  test("the survivor buffer asks for little more memory than it holds") {
+    // Another consumer holds all but what the buffer needs plus a margin. The 
task's memory manager
+    // spills that consumer to grant a reservation past what is free, so the 
buffer must not ask
+    // for more than its margin: less than as much again as it holds while 
small, less than 1 MiB
+    // more once it is past that, see "emitting a row group releases its 
memory in steps", and
+    // never past the cap.
+    withTempDir { dir =>
+      // The buffer never holds more than 220 bytes, so it asks for less than 
440.
+      val path = twoRowGroupsOf20(dir)
+      readBesideOther(path, rows = 40L, margin = 440L)
+      // Doubling from 196 would reserve 392, which a cap of 300 bytes stops 
at 300.
+      readBesideOther(path, rows = 40L, margin = 300L, cap = "300b")
+    }
+  }
+
   test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan 
Filter") {
     // The reader drops the rows a storage filter rejects and the post-scan 
Filter takes its word on
     // the rows it checked, so the reader's evaluation has to be the one the 
Filter would make. A
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
index f78970766681..1f0a261019dd 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
@@ -232,11 +232,21 @@ object ParquetTestAccess {
     f
   }
 
+  private lazy val survivorBatchVectorsMethod = {
+    val m = SparkClassUtils.classForName[Any](
+      classOf[LateMaterializationParquetRecordReader].getName + 
"$SurvivorBatch")
+      .getDeclaredMethod("vectors")
+    m.setAccessible(true)
+    m
+  }
+
   /** The survivor sets phase 1 queued and the reader has not emitted yet, 
oldest first. */
   def queuedSurvivorSets(
       reader: LateMaterializationParquetRecordReader): 
Seq[Seq[WritableColumnVector]] =
-    survivorBatchesField.get(reader)
-      
.asInstanceOf[java.util.ArrayDeque[Array[WritableColumnVector]]].asScala.map(_.toSeq).toSeq
+    
survivorBatchesField.get(reader).asInstanceOf[java.util.ArrayDeque[AnyRef]].asScala.map
 {
+      batch => 
survivorBatchVectorsMethod.invoke(batch).asInstanceOf[Array[WritableColumnVector]]
+        .toSeq
+    }.toSeq
 
   private lazy val columnVectorsField = {
     val f = 
classOf[VectorizedParquetRecordReader].getDeclaredField("columnVectors")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to