This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8c0b8f853274 [SPARK-60381][SQL] Account the late-materialization
survivor buffer to the task's execution memory
8c0b8f853274 is described below
commit 8c0b8f85327447620f110b8019b7bda04f2d1717
Author: Peter Toth <[email protected]>
AuthorDate: Thu Oct 22 20:43:54 2026 -0700
[SPARK-60381][SQL] Account the late-materialization survivor buffer to the
task's execution memory
### What changes were proposed in this pull request?
The late-materialization Parquet reader (SPARK-59620) buffers a row group's
surviving key values and row ranges before it emits that row group, up to
`spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes` per reader.
Today the buffer is outside Spark's memory manager. This PR accounts it to the
task's execution memory through a `MemoryConsumer`. It was on the follow-up
list of #58895, and viirya suggested it there after the merge
([comment](https://github.com/apache/spark/p [...]
- **Reserving:** phase 1 reserves the buffer as it grows.
- A request asks for at least what is already reserved, up to 1 MiB and
never past the cap. So a growing buffer asks only a few times. The reservation
stays below twice what the buffer holds while it is small, and below 1 MiB more
once it is larger.
- What phase 1 reserved ahead is given back before phase 2 reads.
- A denied reservation gives splicing up, as passing the cap does, and
phase 2 reads the key columns again.
- If the row ranges are denied too, the filter is given up for that row
group. A denied request that holds no key values gives the filter up at once,
since giving splicing up would free nothing. That is always the case off heap.
- **Releasing:** as batches are emitted, the reservation steps down once it
is 1 MiB past what is still buffered. The rest goes when the next row group
loads, or at close. So emitting a row group takes the memory manager's lock a
few times, not once per batch.
- **No spill:** the consumer's `spill()` frees nothing.
- The buffer is bounded by the cap, and it drains as its batches are
emitted.
- Freeing the queued keys would make the rest of the row group read its
key pages again and build its ranges again. Both happen outside the memory
manager, and can take more than the spill freed. A spill would also never free
the ranges.
- Accounting with denial lets other consumers that yield to another
consumer's request, such as a sorter being read, spill before this one is
refused. A refused reader backs off.
- **Off heap:** the consumer is `ON_HEAP` whatever the reader's memory
mode, as `WindowSegmentTree`'s spiller is (SPARK-56546). Off-heap column
vectors allocate native memory outside the Tungsten off-heap pool, so an
`OFF_HEAP` consumer would charge that pool for memory it never held. With
off-heap vectors only the row ranges are accounted.
- **Errors:** the task's memory manager runs other consumers' spills inside
a reservation, and can evict cached blocks to disk there. So a reservation can
raise anything, an `IOException` included. The reader rethrows it in a form
`FileScanRDD` cannot take for a corrupt or missing file:
- a fatal error, or one raised by a pending task kill, goes out as the
reader's other error paths send it, an `InternalError` in an `UnknownError`;
- any other `Error`, such as the `SparkOutOfMemoryError` of a failed
spill, goes out as it is;
- any other exception goes out wrapped in a checked `SparkException`.
- **Metrics:** four new scan metrics count the row groups the budget
limited. Each row group counts at most once, under the last thing given up and
what caused it:
- `row groups with key columns read twice by storage filter (buffer cap)`
and `(memory denied)`: splicing was given up and the filter kept;
- `row groups read without storage filter (buffer cap)` and `(memory
denied)`: the filter was given up.
The first of each kind per split is also logged, at INFO.
### Why are the changes needed?
Each reader buffers up to 64 MB by default, which the memory manager cannot
see. Today a task holds one per scan at a time, so an executor holds about one
per running task. Under a sorted merge over many files, as storage-partitioned
joins do once storage filters reach them, one task would hold many at once.
Accounted, other consumers spill earlier, and a reader that is denied memory
gives the buffer up instead. The new metrics show when a scan loses splicing or
the filter, and whethe [...]
### Does this PR introduce _any_ user-facing change?
No. The feature is unreleased and off by default. With it on:
- the buffer counts against the task's execution memory;
- a task holding the buffer is one of the tasks sharing on-heap execution
memory;
- the scan has four more metrics.
### How was this patch tested?
New tests in `ParquetStorageFilterSuite`, most under a task whose memory
manager the test controls:
- **Holding and releasing:** the buffer is held while a spliced row group
is emitted, and released when the next one loads. On heap that is the key
values and the row ranges, off heap the row ranges only. Past the cap only the
ranges stay reserved, even with a batch of keys already queued. What phase 1
reserved ahead is gone by the first batch. A reader closed after its first
batch releases what it holds, and a row group whose filter is given up holds
nothing while it is read whole. E [...]
- **Step release:** emitting a 3.6 MB row group steps the reservation down
at exactly the batches where it is 1 MiB past what is still buffered.
- **Denial:** memory denied for the keys gives splicing up, for the same
bytes as a cap of that size. Memory denied for the ranges gives the filter up,
on and off heap. The metrics tell the two causes apart, and each is logged once
per split.
- **Asking:** the buffer asks for less than twice what it holds while small
and less than 1 MiB more once larger, and never asks past the cap. So within
those margins of the free memory it spills no other consumer.
- **Errors:** a `RuntimeException` or an `InternalError` from another
consumer's spill, or an `IOException` from the memory manager, fails the read
with a checked `SparkException` or an `UnknownError`. `ignoreCorruptFiles`
swallows none of them. A fatal error nested in one goes out as itself.
- **Counting:** the existing cap tests now count each row group under what
it gave up, once. A row group that gave splicing up and then the filter counts
as giving the filter up.
Each was checked by mutation. These all make at least one of them fail:
- ignoring a denied reservation, or one denied again without key values;
- asking again off heap after a denial;
- accounting the key values off heap;
- not releasing at close;
- asking past the cap;
- releasing at every batch rather than in steps;
- keeping what phase 1 reserved ahead through phase 2;
- keeping the ranges reserved once the filter is given up;
- reserving off heap;
- letting an error from another consumer's spill out unwrapped;
- counting a row group as giving splicing up before it may give the filter
up too;
- counting a denied key or range reservation as passing the cap;
- logging every time rather than once per split;
- not taking an emitted batch off the queued total, or not clearing that
total when splicing is given up.
`ParquetStorageFilterSuite` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #59639 from peter-toth/SPARK-60381-survivor-buffer-memory-consumer.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/sql/internal/SQLConf.scala | 14 +-
.../LateMaterializationParquetRecordReader.java | 302 +++++++++++--
.../datasources/parquet/ParquetStorageFilter.scala | 60 ++-
.../parquet/ParquetStorageFilterSuite.scala | 495 ++++++++++++++++++---
.../datasources/parquet/ParquetTestAccess.scala | 14 +-
5 files changed, 778 insertions(+), 107 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index a1cbd5691a4f..570657bec19e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2051,7 +2051,12 @@ object SQLConf {
s"the key values are native memory outside ${MEMORY_OFFHEAP_SIZE.key},
so they come " +
s"out of ${EXECUTOR_MEMORY_OVERHEAD.key}. Past the limit the reader
drops the buffered " +
"key values and reads the key columns again for the surviving rows. If
the row ranges " +
- "alone pass it, the reader reads the row group with no filter
applied.")
+ "alone pass it, the reader reads the row group with no filter applied.
The buffer is " +
+ "also counted against the task's execution memory, except key values
held in off-heap " +
+ "column vectors, and a reservation the memory manager denies is
handled the same way. " +
+ "The scan's metrics count the row groups each of these happens to, by
cause. While it " +
+ "holds a buffer, the task is one of those sharing on-heap execution
memory, as with any " +
+ "operator that reserves it.")
.version("5.0.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.bytesConf(ByteUnit.BYTE)
@@ -2074,9 +2079,10 @@ object SQLConf {
"as in the files pyarrow's write_table writes by default, or with that
conf set to " +
"false, the reader can only skip whole row groups, and it reads the
key columns twice in " +
"the others. With a page index, it can be slower when the matching
rows are spread " +
- "across most pages. The matching key values of a row group are
buffered while it is " +
- s"read, up to
${PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key}, " +
- "outside Spark's memory manager.")
+ "across most pages. The matching key values and row ranges of a row
group are buffered " +
+ "while it is read, up to " +
+ s"${PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key},
and counted " +
+ "against the task's execution memory, except key values held in
off-heap column vectors.")
.version("5.0.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
index 5d0e0206b6ee..007f1250c1e3 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java
@@ -27,6 +27,7 @@ import java.util.List;
import java.util.Objects;
import java.util.PrimitiveIterator;
import java.util.Set;
+import java.util.function.Function;
import org.apache.parquet.column.ColumnDescriptor;
import org.apache.parquet.column.page.PageReadStore;
@@ -35,17 +36,22 @@ import org.apache.parquet.hadoop.metadata.BlockMetaData;
import org.apache.parquet.hadoop.metadata.ColumnPath;
import
org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException;
+import org.apache.spark.TaskContext;
import org.apache.spark.internal.LogKeys;
import org.apache.spark.internal.MDC;
import org.apache.spark.internal.SparkLogger;
import org.apache.spark.internal.SparkLoggerFactory;
+import org.apache.spark.memory.MemoryConsumer;
import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.memory.TaskMemoryManager;
import org.apache.spark.sql.catalyst.expressions.BasePredicate;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.execution.datasources.FileFormat;
+import org.apache.spark.sql.execution.metric.SQLMetric;
import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.internal.SQLConf;
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.DecimalType;
@@ -83,6 +89,9 @@ import org.apache.spark.sql.vectorized.ColumnarBatch;
* <p>A spliced key slot holds a different vector in each batch, and {@link
#nextBatch()} frees the
* previous one. A caller must fetch {@code batch.column(i)} again after every
{@code nextBatch()}.
*
+ * <p>The survivor buffer is accounted to the task's execution memory, the key
values only on heap,
+ * see {@link SurvivorMemory}.
+ *
* <p>When the scan has the column {@code
FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME}, this
* reader fills it. A batch never spans two row groups, so one value covers
each batch. It is true
* where the filter was applied, whose rows are then exactly the survivors,
and false where it was
@@ -99,10 +108,10 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
/**
* What one buffered variable-length value costs, as a multiple of its own
bytes. The byte child
- * holds a null byte per byte, and {@link WritableColumnVector#reserve}
doubles it when it grows.
- * It over-counts on purpose, since nothing can spill this buffer. That
holds at the default
- * growth. A {@code hugeVectorReserveRatio} above 2 makes it under-count past
- * {@code hugeVectorThreshold}, which is off by default.
+ * holds a null byte per byte, and {@link WritableColumnVector#reserve}
doubles it when it grows,
+ * so this is the most the value takes on heap, and what the task's memory
manager is asked for.
+ * That holds at the default growth. A {@code hugeVectorReserveRatio} above
2 makes it
+ * under-count past {@code hugeVectorThreshold}, which is off by default.
*/
private static final long VARIABLE_LENGTH_BYTES_FACTOR = 4L;
@@ -118,6 +127,13 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
*/
private static final int DEFAULT_CHILD_BYTES_PER_ROW = 4;
+ /**
+ * The most the survivor buffer reserves beyond what it holds, see
+ * {@link #reserveSurvivorMemory}. The task's memory manager may spill other
consumers, or evict
+ * cached blocks, to grant a request in full, so an ask past what is free
can cost them that much.
+ */
+ private static final long SURVIVOR_MEMORY_STEP_BYTES = 1L << 20;
+
/**
* The filter this reader applies. {@link #keyColumns} is null for a file it
is not applied to row
* by row.
@@ -183,15 +199,18 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
private WritableColumnVector checkedVector;
/**
- * Splicing state. {@link #survivorBatches} holds one array of survivor
vectors per batch to emit,
- * indexed like {@link #keyColumns}, and {@link #currentKeyAccumulators} the
array still filling.
- * Together they hold every survivor of one row group, outside any
MemoryConsumer.
+ * Splicing state. {@link #survivorBatches} holds one {@link SurvivorBatch}
per batch to emit, and
+ * {@link #currentKeyAccumulators} the vectors still filling, indexed like
{@link #keyColumns}.
+ * Together they hold the survivors of one row group not yet emitted.
*/
- private final ArrayDeque<WritableColumnVector[]> survivorBatches = new
ArrayDeque<>();
+ private final ArrayDeque<SurvivorBatch> survivorBatches = new ArrayDeque<>();
+ /** What the batches in {@link #survivorBatches} are charged together. */
+ private long queuedKeyCharge;
/** The survivor vectors still filling, or null when there are none. */
private WritableColumnVector[] currentKeyAccumulators;
- /** Row count of {@link #currentKeyAccumulators}. */
+ /** Row count and charge of {@link #currentKeyAccumulators}. */
private int currentKeyAccumulatorRowCount;
+ private long currentKeyAccumulatorCharge;
/**
* The survivor vectors the current batch is built on. The next batch frees
them, or
* {@link #close()} when no next batch comes.
@@ -202,6 +221,25 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
/** The array the emitted batch is built over, whose key slots each batch
rewrites. */
private ColumnVector[] spliceBatchColumns;
+ /** What the survivor ranges of the row group being read or emitted cost to
hold. */
+ private long rangeCharge;
+ /**
+ * The survivor buffer's account with the task's memory manager, or null
without one, or for a
+ * file the filter is not applied to row by row.
+ */
+ private SurvivorMemory survivorMemory;
+ /**
+ * What {@link #survivorMemory} holds. Only this reader changes it, so it is
kept here rather
+ * than read off the consumer's atomic counter per survivor.
+ */
+ private long reservedBytes;
+ /**
+ * What the budget gave splicing up for in the row group being loaded, or
null. Phase 1 leaves it
+ * for {@link #loadNextRowGroup} to record once phase 2 keeps the filter, so
a row group counts
+ * once.
+ */
+ private BudgetGiveUp splicingGiveUp;
+
public LateMaterializationParquetRecordReader(
ZoneId convertTz,
String datetimeRebaseMode,
@@ -241,11 +279,17 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
}
} finally {
try {
+ // So `abandonSplicing()` releases everything at once.
+ rangeCharge = 0L;
closeSplicingState();
} finally {
- if (checkedVector != null) {
- checkedVector.close();
- checkedVector = null;
+ try {
+ if (checkedVector != null) {
+ checkedVector.close();
+ checkedVector = null;
+ }
+ } finally {
+ releaseSurvivorMemoryTo(0L);
}
}
}
@@ -337,17 +381,21 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
public boolean nextBatch() throws IOException {
closeAll(publishedKeyVectors);
publishedKeyVectors = null;
+ trimSurvivorMemory();
if (!super.nextBatch()) return false;
if (keyColumns != null) {
if (spliceCurrentRowGroup) {
- publishedKeyVectors = survivorBatches.pollFirst();
- if (publishedKeyVectors == null) {
+ SurvivorBatch next = survivorBatches.pollFirst();
+ if (next == null) {
// Unreachable, since the queue holds exactly the survivors the emit
loop counts. Without
// this check the key slots would show stale values.
throw ParquetStorageFilter.internalError(String.format(
"Storage-filter survivor queue of row group %d in %s ran out
with %d rows still "
+ "to emit", nextBlockIndex - 1, fileReader.getFile(),
columnarBatch.numRows()));
}
+ publishedKeyVectors = next.vectors();
+ // Still reserved, until a later trim, the next row group's load, or
close() releases it.
+ queuedKeyCharge -= next.charge();
}
for (int k = 0; k < keyColumns.length; k++) {
int slot = keyColumns[k].batchSlot();
@@ -445,6 +493,12 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
}
keyFixedBytesPerRow = fixedBytesPerRow;
keyOnlyColumns = Arrays.stream(keys).map(KeyColumn::descriptor).toList();
+ // A test's task can have no memory manager.
+ TaskContext context = TaskContext.get();
+ TaskMemoryManager taskMemoryManager = context == null ? null :
context.taskMemoryManager();
+ if (taskMemoryManager != null) {
+ survivorMemory = new SurvivorMemory(taskMemoryManager);
+ }
// Last, since a non-null `keyColumns` is what says the filter is applied
to this file.
keyColumns = keys;
}
@@ -460,6 +514,8 @@ public class LateMaterializationParquetRecordReader extends
VectorizedParquetRec
// assignment comes only after the next row group's phase 1, whose budget
does not count them.
closeDataPages();
releaseRowGroupReaders();
+ rangeCharge = 0L;
+ releaseSurvivorMemoryTo(heldCharge());
StorageFilterMetrics m = storageFilter.metrics();
List<BlockMetaData> blocks = fileReader.getRowGroups();
while (nextBlockIndex < blocks.size()) {
@@ -500,7 +556,10 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
// Null means phase 1 gave the filter up, and phase 2 then reads every
row the pushed filter
// kept.
boolean filterGivenUp = survivors == null;
- if (filterGivenUp) abandonSplicing();
+ if (filterGivenUp) abandonFilter();
+ // Phase 1 may have reserved up to a step past what it buffered, which
phase 2's reads should
+ // not keep from other consumers.
+ releaseSurvivorMemoryTo(heldCharge());
RowRanges finalRanges = filterGivenUp ? pushedFilterRanges : survivors;
long finalRowCount = filterGivenUp ? baselineRows : survivors.rowCount();
// The byte metrics' baseline, the non-key bytes a plain read asks for
over the rows the
@@ -532,15 +591,16 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
MDC.of(LogKeys.REASON, e.getMessage()));
}
filterGivenUp = true;
- abandonSplicing();
+ abandonFilter();
finalRanges = pushedFilterRanges;
finalRowCount = baselineRows;
fileReader.setRequestedSchema(requestedColumns);
dataPages = fileReader.readFilteredRowGroup(blockIdx, finalRanges);
}
requireRowCount(dataPages, finalRowCount, blockIdx);
- // A row group whose filter was given up reports nothing (see
`StorageFilterMetrics`).
+ // A row group whose filter was given up counts no saving, nor a
splicing give-up.
if (!filterGivenUp) {
+ if (splicingGiveUp != null) recordBudgetGiveUp(splicingGiveUp);
// Over the columns phase 2 read, the key columns included once
splicing is given up. A
// spliced row group that pruned nothing read exactly the baseline.
long phase2Bytes = spliceCurrentRowGroup && finalRowCount ==
baselineRows
@@ -640,7 +700,7 @@ public class LateMaterializationParquetRecordReader extends
VectorizedParquetRec
* <p>Returns null when the filter is given up for this row group:
* <ul>
* <li>at the first survivor of a row group that cannot be read in
part;</li>
- * <li>when the surviving ranges pass the budget;</li>
+ * <li>when the surviving ranges pass the budget, or the task's memory
manager denies them;</li>
* <li>on an error decoding the key pages or applying the filter.</li>
* </ul>
*/
@@ -663,11 +723,10 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator();
RowRanges.Builder finalRangesBuilder = RowRanges.builder();
- // The budget weighs both the bytes buffered for splicing and the
survivors' ranges.
- long splicedBytes = 0L;
- long rangeBytes = 0L;
long previousSurvivor = -2L;
+ // The budget weighs both the key values buffered for splicing and the
survivors' ranges.
long cap = storageFilter.maxSplicedRowGroupBytes();
+ splicingGiveUp = null;
// Out of the loop, since resolving it is a volatile read.
BasePredicate predicate = storageFilter.preparedPredicate();
while (remaining > 0) {
@@ -712,12 +771,12 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
return null;
}
finalRangesBuilder.addSelectedRow(blockRow);
- if (blockRow != previousSurvivor + 1) rangeBytes += ROW_RANGE_BYTES;
+ if (blockRow != previousSurvivor + 1) rangeCharge += ROW_RANGE_BYTES;
previousSurvivor = blockRow;
if (spliceCurrentRowGroup) {
try {
if (currentKeyAccumulators == null) allocateKeyAccumulators();
- splicedBytes += appendSurvivorRowToAccumulators(r);
+ appendSurvivorRowToAccumulators(r);
} catch (RuntimeException e) {
// The buffer is optional, so failing to fill it gives splicing
up, not the filter. That
// covers a vector that cannot grow, and a key value the predicate
did not read on this
@@ -727,15 +786,32 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
// where a plain read fails.
reportOnce(e, SPLICING_GIVEN_UP_ON_ERROR);
abandonSplicing();
- splicedBytes = 0L;
}
}
- if (splicedBytes + rangeBytes > cap) {
- // The buffer goes first, which is a no-op once splicing is given up.
- abandonSplicing();
- splicedBytes = 0L;
+ if (bufferedKeyCharge() + rangeCharge > cap) {
+ // The buffer goes first.
+ if (spliceCurrentRowGroup) {
+ splicingGiveUp = BudgetGiveUp.SPLICING_ON_CAP;
+ abandonSplicing();
+ }
// The ranges would be thrown away, so stop.
- if (rangeBytes > cap) return null;
+ if (rangeCharge > cap) {
+ recordBudgetGiveUp(BudgetGiveUp.FILTER_ON_CAP);
+ return null;
+ }
+ }
+ // A reservation the task's memory manager denies is handled as
passing the cap is. With no
+ // key values in it, giving splicing up frees nothing, so the ranges
are denied too.
+ long held = heldCharge();
+ boolean denied = !reserveSurvivorMemory(held);
+ if (denied && held > rangeCharge) {
+ splicingGiveUp = BudgetGiveUp.SPLICING_ON_MEMORY;
+ abandonSplicing();
+ denied = !reserveSurvivorMemory(heldCharge());
+ }
+ if (denied) {
+ recordBudgetGiveUp(BudgetGiveUp.FILTER_ON_MEMORY);
+ return null;
}
}
remaining -= num;
@@ -797,10 +873,9 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
/**
* Appends row {@code srcRow} of every key column to the accumulators, which
the caller has
- * allocated, and hands them to the queue once they are full. Returns the
bytes the row added,
- * which the caller weighs against its budget.
+ * allocated, adds its bytes to their charge, and hands them to the queue
once they are full.
*/
- private long appendSurvivorRowToAccumulators(int srcRow) {
+ private void appendSurvivorRowToAccumulators(int srcRow) {
long charged = 0L;
final int dstRow = currentKeyAccumulatorRowCount;
final WritableColumnVector[] accs = currentKeyAccumulators;
@@ -820,16 +895,19 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
}
}
}
+ currentKeyAccumulatorCharge += keyFixedBytesPerRow + charged;
currentKeyAccumulatorRowCount = dstRow + 1;
if (currentKeyAccumulatorRowCount == capacity) pushAccumulators();
- return keyFixedBytesPerRow + charged;
}
/** Hands the accumulators to the queue the emit path takes its batches
from. */
private void pushAccumulators() {
- survivorBatches.addLast(currentKeyAccumulators);
+ survivorBatches.addLast(
+ new SurvivorBatch(currentKeyAccumulators,
currentKeyAccumulatorCharge));
+ queuedKeyCharge += currentKeyAccumulatorCharge;
currentKeyAccumulators = null;
currentKeyAccumulatorRowCount = 0;
+ currentKeyAccumulatorCharge = 0L;
}
/** Releases the pages phase 2 read for the row group just emitted, if any.
*/
@@ -867,6 +945,28 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
+ "because buffering a surviving key value raised an error. The
storage filter is still "
+ "applied";
+ private static final String SPLICING_GIVEN_UP_ON_CAP =
+ "Reading the key columns of a row group of {} a second time rather than
buffering them, "
+ + "because the buffer would pass "
+ +
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES().key()
+ + ". The storage filter is still applied";
+
+ private static final String SPLICING_GIVEN_UP_ON_MEMORY =
+ "Reading the key columns of a row group of {} a second time rather than
buffering them, "
+ + "because the task's memory manager denied the memory to buffer
them. The storage "
+ + "filter is still applied";
+
+ private static final String ROW_GROUP_GIVEN_UP_ON_CAP =
+ "Reading a row group of {} without the storage filter, because its
surviving row ranges "
+ + "alone would pass "
+ +
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES().key()
+ + ". The filter is still applied above the scan, so the answer is
unchanged";
+
+ private static final String ROW_GROUP_GIVEN_UP_ON_MEMORY =
+ "Reading a row group of {} without the storage filter, because the
task's memory manager "
+ + "denied the memory to hold its surviving row ranges. The filter is
still applied "
+ + "above the scan, so the answer is unchanged";
+
/**
* Gives the filter up on an error, after rethrowing a task kill or a fatal
error the reader must
* not absorb, see {@code ParquetStorageFilter.rethrowIfMustPropagate}.
Every catch of
@@ -887,19 +987,43 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
LOG.warn(message, t, MDC.of(LogKeys.PATH, fileReader.getFile()));
}
+ /**
+ * Counts a row group the survivor budget limited, under the last thing it
gave up, so each row
+ * group counts once. The first of each kind per split is logged, at INFO
rather than WARN, since
+ * that is the budget doing its job rather than an error.
+ */
+ private void recordBudgetGiveUp(BudgetGiveUp giveUp) {
+ giveUp.counter.apply(storageFilter.metrics()).add(1L);
+ if (reportedMessages.add(giveUp.message)) {
+ LOG.info(giveUp.message, MDC.of(LogKeys.PATH, fileReader.getFile()));
+ }
+ }
+
/**
* Gives up splicing for the row group being evaluated and frees its
buffered survivors. Phase 2
* then reads every projected column, so the rows are unaffected.
*/
private void abandonSplicing() {
- for (WritableColumnVector[] batch : survivorBatches) {
- closeAll(batch);
+ for (SurvivorBatch batch : survivorBatches) {
+ closeAll(batch.vectors());
}
survivorBatches.clear();
+ queuedKeyCharge = 0L;
closeAll(currentKeyAccumulators);
currentKeyAccumulators = null;
currentKeyAccumulatorRowCount = 0;
+ currentKeyAccumulatorCharge = 0L;
spliceCurrentRowGroup = false;
+ releaseSurvivorMemoryTo(heldCharge());
+ }
+
+ /**
+ * Frees everything the survivor buffer holds for a row group whose filter
is given up, its
+ * ranges included, since phase 2 then reads every row the pushed filter
kept.
+ */
+ private void abandonFilter() {
+ rangeCharge = 0L;
+ abandonSplicing();
}
/** Closes every survivor vector this reader still holds, the published ones
included. */
@@ -912,6 +1036,65 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
}
}
+ /** The bytes of key values buffered for splicing, which the cap weighs even
off heap. */
+ private long bufferedKeyCharge() {
+ return queuedKeyCharge + currentKeyAccumulatorCharge;
+ }
+
+ /** The accounted bytes the survivor buffer holds, see {@link
SurvivorMemory}. */
+ private long heldCharge() {
+ return MEMORY_MODE == MemoryMode.ON_HEAP ? rangeCharge +
bufferedKeyCharge() : rangeCharge;
+ }
+
+ /**
+ * Grows the reservation to cover {@code held} bytes, which the caller has
kept within the cap.
+ * It asks for at least what it already holds, up to {@link
#SURVIVOR_MEMORY_STEP_BYTES} and the
+ * cap, so a growing buffer asks only a few times and the reservation never
passes the cap. False
+ * when it is denied.
+ */
+ private boolean reserveSurvivorMemory(long held) {
+ if (survivorMemory == null) return true;
+ long missing = held - reservedBytes;
+ if (missing <= 0) return true;
+ long headroom = storageFilter.maxSplicedRowGroupBytes() - reservedBytes;
+ long ask = Math.max(missing,
+ Math.min(Math.min(reservedBytes, SURVIVOR_MEMORY_STEP_BYTES),
headroom));
+ long granted;
+ try {
+ granted = survivorMemory.acquireMemory(ask);
+ } catch (Exception | InternalError e) {
+ // Another consumer's spill and the eviction of cached blocks run
inside, so this can raise
+ // anything, see `ParquetStorageFilter.memoryReservationFailed`. Any
other `Error`, such as
+ // the `SparkOutOfMemoryError` of a failed spill, goes out as it is,
since `FileScanRDD`
+ // cannot take it for a corrupt file.
+ throw ParquetStorageFilter.memoryReservationFailed(e);
+ }
+ if (granted < missing) {
+ if (granted > 0) survivorMemory.freeMemory(granted);
+ return false;
+ }
+ reservedBytes += granted;
+ return true;
+ }
+
+ /** Shrinks the reservation to {@code held} bytes. */
+ private void releaseSurvivorMemoryTo(long held) {
+ long excess = reservedBytes - held;
+ if (excess <= 0) return;
+ survivorMemory.freeMemory(excess);
+ reservedBytes = held;
+ }
+
+ /**
+ * Shrinks the reservation to what the buffer holds once that is a step
less, so emitting a row
+ * group releases its memory a few times rather than per batch. Each release
takes the memory
+ * manager's lock and wakes every task waiting on it. The rest goes when the
next row group loads.
+ */
+ private void trimSurvivorMemory() {
+ long held = heldCharge();
+ if (reservedBytes - held >= SURVIVOR_MEMORY_STEP_BYTES)
releaseSurvivorMemoryTo(held);
+ }
+
/** Closes every non-null vector of {@code vectors}, which may itself be
null. */
private static void closeAll(ColumnVector[] vectors) {
if (vectors == null) return;
@@ -920,6 +1103,53 @@ public class LateMaterializationParquetRecordReader
extends VectorizedParquetRec
}
}
+ /**
+ * The survivor buffer's account with the task's memory manager, so other
consumers see what it
+ * holds. The charge grows per row, while survivor vectors are allocated a
batch at a time, so up
+ * to a batch per key column is held beyond it. A spill frees nothing. The
buffer is bounded by
+ * {@code maxSplicedRowGroupBytes} and drains as its batches are emitted,
and freeing it would
+ * have the rest of the row group read its key pages again, which the memory
manager does not see
+ * either. A denied reservation is handled where it is made, as passing that
bound is. ON_HEAP
+ * whatever the reader's memory mode, since off-heap vectors allocate native
memory outside the
+ * Tungsten off-heap pool. With them only the row ranges, which are always
on heap, are
+ * accounted. The task's memory manager keeps every consumer until the task
ends, so a task holds
+ * one per file it reserved for.
+ */
+ private static final class SurvivorMemory extends MemoryConsumer {
+ SurvivorMemory(TaskMemoryManager taskMemoryManager) {
+ super(taskMemoryManager, MemoryMode.ON_HEAP);
+ }
+
+ @Override
+ public long spill(long size, MemoryConsumer trigger) {
+ return 0L;
+ }
+ }
+
+ /** One batch's survivor vectors, indexed like {@link #keyColumns}, and what
they are charged. */
+ private record SurvivorBatch(WritableColumnVector[] vectors, long charge) {}
+
+ /**
+ * What the survivor budget gave up for a row group and the limit that made
it, with the metric
+ * that counts it and the message logged for it.
+ */
+ private enum BudgetGiveUp {
+ SPLICING_ON_CAP(StorageFilterMetrics::rowGroupsSplicingGivenUpByCap,
SPLICING_GIVEN_UP_ON_CAP),
+ SPLICING_ON_MEMORY(
+ StorageFilterMetrics::rowGroupsSplicingGivenUpByMemory,
SPLICING_GIVEN_UP_ON_MEMORY),
+ FILTER_ON_CAP(StorageFilterMetrics::rowGroupsFilterGivenUpByCap,
ROW_GROUP_GIVEN_UP_ON_CAP),
+ FILTER_ON_MEMORY(
+ StorageFilterMetrics::rowGroupsFilterGivenUpByMemory,
ROW_GROUP_GIVEN_UP_ON_MEMORY);
+
+ final Function<StorageFilterMetrics, SQLMetric> counter;
+ final String message;
+
+ BudgetGiveUp(Function<StorageFilterMetrics, SQLMetric> counter, String
message) {
+ this.counter = counter;
+ this.message = message;
+ }
+ }
+
/** One key column this file has, as init resolves it. */
private record KeyColumn(
int batchSlot,
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
index 5f3fbc7801b7..1cbe41083b40 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala
@@ -30,12 +30,12 @@ import org.apache.spark.util.Utils
/**
* The SQL metrics the reader updates while applying a
[[ParquetStorageFilter]], created by
- * `ParquetFileFormat.storageFilterMetrics`. They count only what the filter
saved against a read of
- * the same projection without one. A row group whose filter was given up
counts nothing, so a scan
- * that read more than with the feature off can still show a saving. Nor do
they take off the offset
- * indexes phase 2 reads when no data filter is pushed, which a plain read
does not. Bytes are the
- * compressed bytes the reader did not ask parquet for, which a vectored read
can still fetch when
- * it merges nearby ranges.
+ * `ParquetFileFormat.storageFilterMetrics`. The first five count only what
the filter saved against
+ * a read of the same projection without one. A row group whose filter was
given up counts nothing
+ * there, so a scan that read more than with the feature off can still show a
saving. Nor do they
+ * take off the offset indexes phase 2 reads when no data filter is pushed,
which a plain read does
+ * not. Bytes are the compressed bytes the reader did not ask parquet for,
which a vectored read can
+ * still fetch when it merges nearby ranges.
*
* - [[rowGroupsSkipped]] counts row groups whose non-key columns were never
read.
* - [[rowsExcludedByRowGroup]] sums those row groups' rows that the pushed
data filter kept.
@@ -47,13 +47,26 @@ import org.apache.spark.util.Utils
*
* A row excluded within a row group may still have been read, as part of a
page that held a
* survivor.
+ *
+ * The other four count row groups the survivor buffer's budget limited, each
at most once, under
+ * the last thing it gave up and the limit that made it:
`maxSplicedRowGroupBytes`, or the task's
+ * memory manager denying the reservation.
+ *
+ * - [[rowGroupsSplicingGivenUpByCap]] and
[[rowGroupsSplicingGivenUpByMemory]] count row groups
+ * that kept the filter but read their key columns a second time.
+ * - [[rowGroupsFilterGivenUpByCap]] and [[rowGroupsFilterGivenUpByMemory]]
count row groups read
+ * without the filter.
*/
case class StorageFilterMetrics(
rowGroupsSkipped: SQLMetric,
rowsExcludedByRowGroup: SQLMetric,
rowsExcludedWithinRowGroup: SQLMetric,
bytesAvoidedByRowGroup: SQLMetric,
- bytesAvoidedByPageFiltering: SQLMetric) {
+ bytesAvoidedByPageFiltering: SQLMetric,
+ rowGroupsSplicingGivenUpByCap: SQLMetric,
+ rowGroupsSplicingGivenUpByMemory: SQLMetric,
+ rowGroupsFilterGivenUpByCap: SQLMetric,
+ rowGroupsFilterGivenUpByMemory: SQLMetric) {
/** Counts a row group whose non-key columns the filter kept the reader from
reading at all. */
def recordRowGroupSkipped(excludedRows: Long, avoidedBytes: Long): Unit = {
@@ -71,6 +84,10 @@ object StorageFilterMetrics {
val ROWS_EXCLUDED_WITHIN_ROW_GROUP =
"storageFilterRowsExcludedWithinRowGroup"
val BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup"
val BYTES_AVOIDED_BY_PAGE_FILTERING =
"storageFilterBytesAvoidedByPageFiltering"
+ val ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP =
"storageFilterRowGroupsSplicingGivenUpByCap"
+ val ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY =
"storageFilterRowGroupsSplicingGivenUpByMemory"
+ val ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP =
"storageFilterRowGroupsFilterGivenUpByCap"
+ val ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY =
"storageFilterRowGroupsFilterGivenUpByMemory"
/** A fresh set of counters, keyed for the scan, with the labels the SQL UI
shows. */
def create(sparkContext: SparkContext): Map[String, SQLMetric] = Map(
@@ -83,7 +100,15 @@ object StorageFilterMetrics {
BYTES_AVOIDED_BY_ROW_GROUP -> SQLMetrics.createSizeMetric(
sparkContext, "bytes avoided by storage filter (whole row group)"),
BYTES_AVOIDED_BY_PAGE_FILTERING -> SQLMetrics.createSizeMetric(
- sparkContext, "bytes avoided by storage filter (page filtering)"))
+ sparkContext, "bytes avoided by storage filter (page filtering)"),
+ ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP -> SQLMetrics.createMetric(
+ sparkContext, "row groups with key columns read twice by storage filter
(buffer cap)"),
+ ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY -> SQLMetrics.createMetric(
+ sparkContext, "row groups with key columns read twice by storage filter
(memory denied)"),
+ ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP -> SQLMetrics.createMetric(
+ sparkContext, "row groups read without storage filter (buffer cap)"),
+ ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY -> SQLMetrics.createMetric(
+ sparkContext, "row groups read without storage filter (memory denied)"))
/** The counters the scan carries back, which [[create]] made. */
def fromMap(metrics: Map[String, SQLMetric]): StorageFilterMetrics =
StorageFilterMetrics(
@@ -91,7 +116,11 @@ object StorageFilterMetrics {
rowsExcludedByRowGroup = metrics(ROWS_EXCLUDED_BY_ROW_GROUP),
rowsExcludedWithinRowGroup = metrics(ROWS_EXCLUDED_WITHIN_ROW_GROUP),
bytesAvoidedByRowGroup = metrics(BYTES_AVOIDED_BY_ROW_GROUP),
- bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING))
+ bytesAvoidedByPageFiltering = metrics(BYTES_AVOIDED_BY_PAGE_FILTERING),
+ rowGroupsSplicingGivenUpByCap =
metrics(ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP),
+ rowGroupsSplicingGivenUpByMemory =
metrics(ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY),
+ rowGroupsFilterGivenUpByCap = metrics(ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP),
+ rowGroupsFilterGivenUpByMemory =
metrics(ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY))
}
/**
@@ -295,6 +324,19 @@ object ParquetStorageFilter {
}
}
+ /**
+ * Throws what reserving the survivor buffer's memory raised, in a form
`FileScanRDD` cannot take
+ * for a corrupt or missing file. The task's memory manager runs other
consumers' spills inside
+ * the reservation, and can evict cached blocks to disk there, so it can
raise anything, an
+ * `IOException` included. What [[rethrowIfMustPropagate]] rethrows goes out
as it does there, and
+ * anything else wrapped in a checked [[SparkException]]. Typed as returning
a `RuntimeException`
+ * so Java code can write `throw memoryReservationFailed(e)`.
+ */
+ def memoryReservationFailed(e: Throwable): RuntimeException = {
+ rethrowIfMustPropagate(e, fromRead = false)
+ throw new SparkException("A storage filter failed to reserve memory", e)
+ }
+
/**
* Throws `SparkException.internalError(message)` for a broken invariant of
this reader. It is
* checked, so `FileScanRDD` cannot take it for a corrupt file. Typed as
returning a
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
index 2179ac12be11..07d0f9dfae5b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala
@@ -17,10 +17,10 @@
package org.apache.spark.sql.execution.datasources.parquet
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File,
RandomAccessFile}
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File,
IOException, RandomAccessFile}
import java.net.URI
import java.time.LocalTime
-import java.util.Locale
+import java.util.{Locale, Properties}
import java.util.concurrent.atomic.AtomicLong
import scala.collection.mutable
@@ -47,8 +47,9 @@ import org.mockito.ArgumentMatchers.{any, anyInt}
import org.mockito.Mockito.{doAnswer, doThrow, spy}
import org.mockito.invocation.InvocationOnMock
-import org.apache.spark.{SparkEnv, SparkException, TaskContext,
TaskKilledException}
+import org.apache.spark.{SparkEnv, SparkException, TaskContext,
TaskContextImpl, TaskKilledException}
import org.apache.spark.internal.config.KILL_ON_FATAL_ERROR_DEPTH
+import org.apache.spark.memory.{MemoryConsumer, MemoryManager, MemoryMode,
TaskMemoryManager, TestMemoryConsumer, TestMemoryManager}
import org.apache.spark.paths.SparkPath
import org.apache.spark.sql.{classic, sources, DataFrame, QueryTest, Row,
SparkSession}
import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
@@ -73,7 +74,7 @@ import org.apache.spark.util.sketch.BloomFilter
/**
* Tests [[LateMaterializationParquetRecordReader]] driven by a
[[ParquetStorageFilter]]. Writes
* small multi-row-group parquet files, wires a hand-built filter into the
reader, and asserts
- * correctness and the five storage-filter metrics.
+ * correctness and the storage-filter metrics.
*/
class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession
with AdaptiveSparkPlanHelper {
@@ -197,18 +198,21 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
s"a cap one byte under it must give splicing up; got $underCharge
against $uncapped")
}
- // Reads `path` whole through the format under a splice cap and a batch
size, and returns the rows
- // it emitted with the bytes it transferred.
+ // Reads `path` whole through the format under a splice cap and a batch
size, as a task of
+ // `context` if one is given, and returns the rows it emitted with the bytes
it transferred.
private def readUnderCap(
path: String,
schema: StructType,
storageFilters: Seq[Expression],
maxSplicedBytes: String,
batchSize: Int,
- metrics: Map[String, SQLMetric] = metricMap()): (Int, Long) =
withSQLConf(
+ metrics: Map[String, SQLMetric] = metricMap(),
+ context: Option[TaskContext] = None): (Int, Long) = withSQLConf(
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key
-> maxSplicedBytes,
SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString) {
- readCounting(path, formatReader(schema, Nil, storageFilters,
countingHadoopConf(), metrics))
+ def read(): (Int, Long) = readCounting(
+ path, formatReader(schema, Nil, storageFilters, countingHadoopConf(),
metrics))
+ context.fold(read())(asTask(_)(read()))
}
// Reads `path` whole through `readerFn`, and returns the long column at
`ordinal` of every row.
@@ -254,6 +258,12 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// What the reader logs when filling the survivor buffer fails and it gives
splicing up.
private val splicingGivenUpOnError = "buffering a surviving key value raised
an error"
+ // What the reader logs when the survivor buffer's budget gives splicing or
the filter up.
+ private val splicingGivenUpOnCap = "because the buffer would pass"
+ private val splicingGivenUpOnMemory = "denied the memory to buffer them"
+ private val filterGivenUpOnCap = "its surviving row ranges alone would pass"
+ private val filterGivenUpOnMemory = "denied the memory to hold its surviving
row ranges"
+
// What the reader logs when applying the filter to a row, or decoding the
key pages, raises an
// error and it gives the filter up for that row group.
private val evaluationGivenUp = "applying it to a row raised an error"
@@ -273,6 +283,14 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
private def allMetrics(): StorageFilterMetrics =
StorageFilterMetrics.fromMap(metricMap())
+ // The row groups the survivor budget limited, as counted in `metrics`:
splicing given up by the
+ // cap and by the memory manager, then the filter given up by each.
+ private def budgetGiveUps(metrics: Map[String, SQLMetric]): Seq[Long] = Seq(
+ StorageFilterMetrics.ROW_GROUPS_SPLICING_GIVEN_UP_BY_CAP,
+ StorageFilterMetrics.ROW_GROUPS_SPLICING_GIVEN_UP_BY_MEMORY,
+ StorageFilterMetrics.ROW_GROUPS_FILTER_GIVEN_UP_BY_CAP,
+
StorageFilterMetrics.ROW_GROUPS_FILTER_GIVEN_UP_BY_MEMORY).map(metrics(_).value)
+
// `ParquetStorageFilter.create` takes what the production path always
supplies; most tests care
// about neither the metrics nor the cap.
private def createFilter(
@@ -1081,17 +1099,24 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
withTempDir { dir =>
val path = writePagedFile(dir)
- def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long) = {
+ // Also returns the row groups the budget limited, see `budgetGiveUps`.
+ def run(maxSplicedBytes: String, threshold: Long): (Int, Long, Long,
Seq[Long]) = {
val metrics = metricMap()
val (rows, bytes) = readUnderCap(path, kvSchema(),
Seq(keyAtLeast(threshold)),
maxSplicedBytes, batchSize = 16, metrics)
- (rows, bytes,
metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value)
+ (rows, bytes,
metrics(StorageFilterMetrics.BYTES_AVOIDED_BY_PAGE_FILTERING).value,
+ budgetGiveUps(metrics))
}
- val (splicedRows, splicedBytes, splicedAvoided) = run("64MB", 350L)
+ val (splicedRows, splicedBytes, splicedAvoided, splicedGiveUps) =
run("64MB", 350L)
// `k >= 350` keeps one contiguous range of 40 bytes, and a long key
buffers 9 bytes per
// survivor, so a 100-byte cap is passed at the seventh.
- val (plainRows, plainBytes, plainAvoided) = run("100b", 350L)
+ val ((plainRows, plainBytes, plainAvoided, plainGiveUps), logged) =
+ withLogged(splicingGivenUpOnCap)(run("100b", 350L))
+ assert(splicedGiveUps == Seq(0L, 0L, 0L, 0L), s"got $splicedGiveUps")
+ assert(plainGiveUps == Seq(1L, 0L, 0L, 0L),
+ s"the row group must count as giving splicing up on the cap; got
$plainGiveUps")
+ assert(logged == 1, s"and log it; got $logged")
assert(splicedRows == 51, s"the filter keeps keys 350..400; got
$splicedRows")
assert(plainRows == splicedRows,
s"rows differ past the cap: plain=$plainRows spliced=$splicedRows")
@@ -1107,8 +1132,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// Eight survivors buffer 72 bytes and fall in one range of 40, so
neither half reaches the
// 100-byte cap on its own while the sum passes it at the seventh.
Weighing them separately
// would keep splicing here, and report the larger saving for it.
- val (_, _, eightSpliced) = run("64MB", 393L)
- val (_, _, eightPlain) = run("100b", 393L)
+ val (_, _, eightSpliced, _) = run("64MB", 393L)
+ val (_, _, eightPlain, _) = run("100b", 393L)
assert(eightSpliced > 0, s"and avoid something with eight survivors; got
$eightSpliced")
assert(eightPlain < eightSpliced,
s"the sum of the two halves must cross the cap; plain=$eightPlain
spliced=$eightSpliced")
@@ -1116,7 +1141,7 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// The exact charge. Seven survivors in one range cost 7 * 9 + 40 = 103
bytes, which a cap of
// 103 holds and one of 102 does not. A charge one byte off either way
flips one of the two.
assertSpliceCharge(103L, rows = 7) { cap =>
- val (rows, bytes, _) = run(cap, 394L)
+ val (rows, bytes, _, _) = run(cap, 394L)
(rows, bytes)
}
}
@@ -1143,7 +1168,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
val rows = rows400
val path = writeParquetFile(dir, rows, rowGroupSize = 256L)
assert(footerOf(path).getBlocks.size == 4, "the fixture must be four row
groups")
- val m = allMetrics()
+ val metrics = metricMap()
+ val m = StorageFilterMetrics.fromMap(metrics)
val k = BoundReference(0, LongType, nullable = true)
def between(lo: Long, hi: Long): Expression =
And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k,
Literal(hi)))
@@ -1179,6 +1205,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
s"rows 1-100 and 201-300 must splice and rows 101-200 must not; got
$splicedByRowGroup")
assert(m.rowGroupsSkipped.value == 1,
s"expected the row group with no survivor skipped; got
${m.rowGroupsSkipped.value}")
+ assert(budgetGiveUps(metrics) == Seq(1L, 0L, 0L, 0L),
+ s"and the one over the cap counted once; got
${budgetGiveUps(metrics)}")
} finally {
reader.close()
}
@@ -1626,12 +1654,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// `UnknownError`. Off heap, the scratch vectors the failed decode used
must be freed, and the
// second row group's phase 1 runs over the ones that replace them.
withTempDir { dir =>
+ val path = twoRowGroupsOf20(dir)
val rows = (0L until 40L).map(i => (i, i.toString))
- val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
- rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
- options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
- assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L,
20L),
- "the fixture must be two row groups of 20 rows")
Seq(
new RuntimeException("Cannot reserve", new
OutOfMemoryError("reserve")),
new InternalError("codec")).foreach { error =>
@@ -1849,14 +1873,20 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
val everyOtherRow = EqualTo(
Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)),
Literal(0L))
- def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = {
- val filter =
- createFilter(Seq(everyOtherRow), fileSchema, allMetrics(), cap)
+ // Also returns the row groups the budget limited, see `budgetGiveUps`.
+ def readWithCap(cap: Long, capacity: Int = 4096): (Seq[(Long, String)],
Seq[Long]) = {
+ val metrics = metricMap()
+ val filter = createFilter(
+ Seq(everyOtherRow), fileSchema,
StorageFilterMetrics.fromMap(metrics), cap)
val (emitted, reader) = readAll(path, filter, capacity)
- try emitted finally reader.close()
+ try {
+ (emitted, budgetGiveUps(metrics))
+ } finally {
+ reader.close()
+ }
}
- assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0),
+ assert(readWithCap(64L * 1024 * 1024) == (rows.filter(_._1 % 2 == 0),
Seq(0L, 0L, 0L, 0L)),
"with room for the ranges, every surviving row comes back with its own
value")
// Each survivor costs 9 bytes of long key and 40 of its own range. The
sum passes the cap of
// 1024 at survivor 21, which releases the buffer, and the ranges alone
at survivor 26, which
@@ -1865,9 +1895,12 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// well. Both have to end with the queue empty and the filter given up.
Keeping one without
// the other would leave phase 2 reading non-key columns while emit
spliced keys from a queue
// holding only the first survivors.
- assert(readWithCap(1024L) == rows,
+ // The row group counts once, as giving the filter up, though it gave
splicing up first.
+ val (pastCap, logged) =
withLogged(filterGivenUpOnCap)(readWithCap(1024L))
+ assert(pastCap == (rows, Seq(0L, 0L, 1L, 0L)),
"past the cap the filter is given up and the row group comes back
whole")
- assert(readWithCap(1024L, capacity = 16) == rows,
+ assert(logged == 1, s"and that is logged; got $logged")
+ assert(readWithCap(1024L, capacity = 16) == (rows, Seq(0L, 0L, 1L, 0L)),
"and the same when the buffer has been rolled over first")
}
}
@@ -1884,7 +1917,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)),
EqualTo(Remainder(k, Literal(2L)), Literal(0L)))
val runInLast = And(GreaterThanOrEqual(k, Literal(301L)),
LessThanOrEqual(k, Literal(320L)))
- val m = allMetrics()
+ val metrics = metricMap()
+ val m = StorageFilterMetrics.fromMap(metrics)
val filter = createFilter(
Seq(Or(scatteredInFirst, runInLast)),
kvSchema(),
@@ -1898,6 +1932,8 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
s"expected the given-up row group whole and the last filtered; got
${result.size} rows")
assert(m.rowGroupsSkipped.value == 2,
s"and the two emptied row groups skipped; got
${m.rowGroupsSkipped.value}")
+ assert(budgetGiveUps(metrics) == Seq(0L, 0L, 1L, 0L),
+ s"and the first counted once; got ${budgetGiveUps(metrics)}")
} finally {
reader.close()
}
@@ -1950,20 +1986,21 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
kvSchema(valueType =
valueType).add(FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME, BooleanType)
// Reads `path` through the format, the way a scan with `storageFilters`
over `schema` does, under
- // the SQL confs in `conf`. Returns each row's key, as `keyOf` reads it,
with what the checked
- // column says for the row.
+ // the SQL confs in `conf`, and as a task of `context` if one is given. The
confs are set outside
+ // the task, which sees a read-only one. Returns each row's key, as `keyOf`
reads it, with what
+ // the checked column says for the row.
private def readChecked(
path: String,
schema: StructType,
storageFilters: Seq[Expression],
conf: Map[String, String] = Map.empty,
- keyOf: (ColumnarBatch, Int) => Long = (b, i) => b.column(0).getLong(i))
- : Seq[(Long, Boolean)] = withSQLConf(conf.toSeq: _*) {
+ keyOf: (ColumnarBatch, Int) => Long = (b, i) => b.column(0).getLong(i),
+ context: Option[TaskContext] = None): Seq[(Long, Boolean)] =
withSQLConf(conf.toSeq: _*) {
val checked =
schema.fieldIndex(FileFormat.STORAGE_FILTER_CHECKED_COLUMN_NAME)
val file = PartitionedFile(
InternalRow.empty, SparkPath.fromPathString(path), 0, new
File(path).length())
- drain(formatReader(schema, Nil, storageFilters)(file)) { it =>
- it.flatMap { case batch: ColumnarBatch =>
+ def read(): Seq[(Long, Boolean)] = drain(formatReader(schema, Nil,
storageFilters)(file)) {
+ it => it.flatMap { case batch: ColumnarBatch =>
val marks = batch.column(checked)
(0 until batch.numRows()).map { i =>
assert(!marks.isNullAt(i), "the reader must mark every row")
@@ -1971,12 +2008,44 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
}
}.toSeq
}
+ context.fold(read())(asTask(_)(read()))
}
// The keys in `keys`, each with the mark `checked`.
private def marked(keys: Seq[Long], checked: Boolean): Seq[(Long, Boolean)] =
keys.map(_ -> checked)
+ // 40 rows of a long key and a string value in two row groups of 20. A
filter of `k >= 10` keeps
+ // one range of 10 rows in the first and all 20 rows of the second. A
buffered long key costs 9
+ // bytes and a range 40, so the first row group holds 130 bytes and the
second 220.
+ private def twoRowGroupsOf20(dir: File): String = {
+ val path = writeSingleParquetFile(dir, (0L until 40L).map(i => (i,
i.toString)).toDF("k", "v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
+ options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
+ assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L,
20L),
+ "the fixture must be two row groups of 20 rows")
+ path
+ }
+
+ // Three row groups of 100 long keys and values, `(i, i)` for i from 1 to
300.
+ private val threeRowGroupsOf100: Seq[Seq[(Long, Long)]] =
+ (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 + 100L)).map(i => (i, i)))
+
+ // Keeps the rows whose key, at ordinal 0, is even.
+ private val evenKey: Expression =
+ EqualTo(Remainder(BoundReference(0, LongType, nullable = true),
Literal(2L)), Literal(0L))
+
+ // `threeRowGroupsOf100` with the middle row group's value offset index
damaged. Phase 2 needs it
+ // to read that column over the survivors, and the retry then reads the row
group whole. Returns
+ // the path, with the rows a read under `evenKey` returns.
+ private def damagedMiddleRowGroup(dir: File): (String, Seq[(Long, Boolean)])
= {
+ val path = writeParquetFileByHand(dir, threeRowGroupsOf100, (_, _) => true)
+ breakOffsetIndex(path, blockIdx = 1, columnIdx = 1)
+ val keys = (block: Int) => threeRowGroupsOf100(block).map(_._1)
+ val even = (block: Int) => keys(block).filter(_ % 2 == 0)
+ (path, marked(even(0), checked = true) ++ marked(keys(1), false) ++
marked(even(2), true))
+ }
+
test("the checked column is true exactly for the rows of a row group the
filter was applied to") {
// The post-scan Filter skips the storage filter on a row marked true, so
true must mean the
// filter kept that row. A row group the filter was given up on comes back
whole and must be
@@ -1984,12 +2053,7 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// give-up in the first leaves the second its own. Batches of 4 rows split
each row group, and
// every batch of it must be marked alike.
withTempDir { dir =>
- val rows = (0L until 40L).map(i => (i, i.toString))
- val path = writeSingleParquetFile(dir, rows.toDF("k", "v"),
- rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE,
- options = Map(ParquetOutputFormat.BLOCK_ROW_COUNT_LIMIT -> "20"))
- assert(footerOf(path).getBlocks.asScala.map(_.getRowCount) == Seq(20L,
20L),
- "the fixture must be two row groups of 20 rows")
+ val path = twoRowGroupsOf20(dir)
val schema = checkedSchema()
val k = BoundReference(0, LongType, nullable = true)
def filtersOn(key: Expression): Seq[Expression] =
Seq(GreaterThanOrEqual(key, Literal(10L)))
@@ -2023,31 +2087,44 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
// A row group the footer declines, with no offset index, and one whose
offset index parquet
// then fails to parse both come back whole, the latter through phase 2's
retry.
withTempDir { dir =>
- val blocks = (0 until 3).map(b => ((b * 100 + 1L) to (b * 100 +
100L)).map(i => (i, i)))
val schema = checkedSchema(valueType = LongType)
- val evenKeys = Seq(EqualTo(
- Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)),
Literal(0L)))
- def keys(block: Int): Seq[Long] = blocks(block).map(_._1)
-
- val noIndex = writeParquetFileByHand(dir, blocks, hasOffsetIndex = (_,
_) => false)
- assert(readChecked(noIndex, schema, evenKeys) == marked((0 until
3).flatMap(keys), false),
+ val noIndex =
+ writeParquetFileByHand(dir, threeRowGroupsOf100, hasOffsetIndex = (_,
_) => false)
+ assert(readChecked(noIndex, schema, Seq(evenKey)) ==
+ marked(threeRowGroupsOf100.flatten.map(_._1), false),
"every row group of a file with no offset index must come back whole
and marked false")
- // The middle row group's value index is damaged. Phase 2 needs it to
read that column over
- // the survivors, and the retry then reads the row group whole.
- val damaged = writeParquetFileByHand(dir, blocks, (_, _) => true)
- breakOffsetIndex(damaged, blockIdx = 1, columnIdx = 1)
+ val (damaged, expected) = damagedMiddleRowGroup(dir)
val (result, warnings) = withLogged(partialReadWarning) {
- readChecked(damaged, schema, evenKeys)
+ readChecked(damaged, schema, Seq(evenKey))
}
assert(warnings == 1, s"the read must have gone through the retry; got
$warnings")
- val even = (block: Int) => keys(block).filter(_ % 2 == 0)
- assert(result ==
- marked(even(0), checked = true) ++ marked(keys(1), false) ++
marked(even(2), true),
+ assert(result == expected,
"the damaged row group must come back whole and marked false, and the
others filtered")
}
}
+ test("a row group whose filter phase 2 gives up does not count as giving
splicing up") {
+ // At a cap of 2,100 bytes, each row group's 50 scattered even keys pass
it at the 43rd
+ // survivor, at 43 x 49 = 2,107 bytes, while their 2,000 bytes of ranges
fit. So each gives
+ // splicing up and keeps the filter. The middle row group's damaged value
index then has phase
+ // 2 give the filter up as well, which the splicing give-up must not be
counted for. That row
+ // group then holds nothing while it is emitted.
+ withTempDir { dir =>
+ val (damaged, expected) = damagedMiddleRowGroup(dir)
+ val schema = checkedSchema(valueType = LongType)
+ val metrics = metricMap()
+ readUnderCap(damaged, schema, Seq(evenKey), "2100b", batchSize = 4,
metrics)
+ assert(budgetGiveUps(metrics) == Seq(2L, 0L, 0L, 0L),
+ s"only the two row groups that kept the filter count; got
${budgetGiveUps(metrics)}")
+ val (rows, held) = readInTask(damaged, memoryTask(), cap = "2100b",
filter = evenKey,
+ schema = schema)
+ assert(rows == expected, s"got $rows")
+ assert(held == Seq.fill(13)(2000L) ++ Seq.fill(25)(0L) ++
Seq.fill(13)(2000L),
+ s"each batch must hold its row group's ranges, and none for the one
read whole; got $held")
+ }
+ }
+
test("the checked column for a file the filter is not applied to row by
row") {
withTempDir { dir =>
val schema = checkedSchema()
@@ -2119,6 +2196,312 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
}
}
+ // A task memory manager that calls `onAcquire` before each acquisition but
a test consumer's,
+ // with what is asked for, what the task holds then, and the consumer's
memory mode.
+ private class ObservedTaskMemoryManager(
+ memoryManager: MemoryManager,
+ onAcquire: (Long, Long, MemoryMode) => Unit) extends
TaskMemoryManager(memoryManager, 0) {
+ override def acquireExecutionMemory(required: Long, consumer:
MemoryConsumer): Long = {
+ if (!consumer.isInstanceOf[TestMemoryConsumer]) {
+ onAcquire(required, getMemoryConsumptionForThisTask, consumer.getMode)
+ }
+ super.acquireExecutionMemory(required, consumer)
+ }
+ }
+
+ // A task whose memory manager grants at most `limit` bytes, and calls
`onAcquire` as
+ // `ObservedTaskMemoryManager` does.
+ private def memoryTask(
+ limit: Long = Long.MaxValue,
+ onAcquire: (Long, Long, MemoryMode) => Unit = (_, _, _) => ()):
TaskContextImpl = {
+ val memoryManager = new TestMemoryManager(spark.sparkContext.conf)
+ memoryManager.limit(limit)
+ new TaskContextImpl(stageId = 0, stageAttemptNumber = 0, partitionId = 0,
taskAttemptId = 0,
+ attemptNumber = 0, numPartitions = 1,
+ taskMemoryManager = new ObservedTaskMemoryManager(memoryManager,
onAcquire),
+ localProperties = new Properties, metricsSystem = null)
+ }
+
+ // Reads `path` under `filter` in batches of `batchSize` and a cap of `cap`,
as a task of
+ // `context`. Returns the rows, and what the task holds as each batch is
read.
+ private def readInTask(
+ path: String,
+ context: TaskContext,
+ offHeap: Boolean = false,
+ batchSize: Int = 4,
+ cap: String = "64MB",
+ filter: Expression = keyAtLeast(10L),
+ schema: StructType = checkedSchema()): (Seq[(Long, Boolean)], Seq[Long])
= {
+ var held = Seq.empty[Long]
+ val rows = readChecked(path, schema, Seq(filter), conf = Map(
+ SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> batchSize.toString,
+ SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> offHeap.toString,
+
SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> cap),
+ keyOf = (b, i) => {
+ if (i == 0) held :+=
context.taskMemoryManager().getMemoryConsumptionForThisTask
+ b.column(0).getLong(i)
+ },
+ context = Some(context))
+ (rows, held)
+ }
+
+ // Reads `path` under `k >= 10` while another consumer holds all of the
task's memory but
+ // `margin`, and checks that the read neither drops rows nor spills that
consumer.
+ private def readBesideOther(
+ path: String,
+ rows: Long,
+ margin: Long,
+ cap: String = "64MB"): Unit = {
+ val held = 1000000L
+ val context = memoryTask(held + margin)
+ val other = new TestMemoryConsumer(context.taskMemoryManager)
+ other.use(held)
+ val (read, _) = readInTask(path, context, batchSize = 4096, cap = cap)
+ val name = s"$rows rows with a margin of $margin under a cap of $cap"
+ assert(read == marked(10L until rows, checked = true), s"$name: got
${read.size} rows")
+ assert(other.getUsed == held, s"$name: the read must not spill another
consumer")
+ }
+
+ // One row group of 400,000 rows of a long key and a string value. Under `k
>= 10` its 399,990
+ // survivors fall in one range and buffer 3,599,950 bytes, several steps of
the reservation.
+ private def oneRowGroupOf400000(dir: File): String = {
+ val path = writeSingleParquetFile(dir,
+ spark.range(400000L).selectExpr("id AS k", "CAST(id AS STRING) AS v"),
+ rowGroupSize = ParquetWriter.DEFAULT_BLOCK_SIZE)
+ assert(footerOf(path).getBlocks.size == 1, "the fixture must be one row
group")
+ path
+ }
+
+ test("the survivor buffer is accounted to the task's execution memory") {
+ // What a row group buffers is held while it is emitted, and released when
the next one loads.
+ // Phase 1 reserves a little ahead of what it buffers, and gives that back
before phase 2. Off
+ // heap, only the ranges are accounted, since the vectors' native memory
is not the memory
+ // manager's.
+ withTempDir { dir =>
+ val path = twoRowGroupsOf20(dir)
+ // Past a cap of 100 bytes, at the seventh survivor of each row group,
the keys go, a full
+ // batch of them already queued, and only the range is held.
+ Seq((false, "64MB", Seq(130L, 130L, 130L, 220L, 220L, 220L, 220L, 220L)),
+ (true, "64MB", Seq.fill(8)(40L)),
+ (false, "100b", Seq.fill(8)(40L))).foreach { case (offHeap, cap,
expected) =>
+ var modes = Set.empty[MemoryMode]
+ val context = memoryTask(onAcquire = (_, _, mode) => modes += mode)
+ val (rows, held) = readInTask(path, context, offHeap, cap = cap)
+ val name = s"off heap $offHeap under a cap of $cap"
+ assert(rows == marked(10L until 40L, checked = true), s"$name: got
$rows")
+ // The off-heap pool does not hold the vectors' native memory, nor the
ranges.
+ assert(modes == Set(MemoryMode.ON_HEAP), s"$name: every ask must be on
heap; got $modes")
+ assert(held == expected,
+ s"$name: each batch must hold its row group's reservation until the
next row group " +
+ s"loads; got $held")
+ assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0,
+ s"$name: not released")
+ }
+ // On heap, each row group asks first for its first survivor's 49 bytes,
with nothing held
+ // since the previous one's release, and then for as much again as it
holds, three and four
+ // times for the 130 and 220 bytes it reaches.
+ var asked = Seq.empty[(Long, Long)]
+ readInTask(path, memoryTask(onAcquire = (size, held, _) => asked :+=
((size, held))))
+ assert(asked == Seq((49L, 0L), (49L, 49L), (98L, 98L), (49L, 0L), (49L,
49L), (98L, 98L),
+ (196L, 196L)), s"got $asked")
+ // A reader closed after its first batch, as under a limit, releases
what it still holds.
+ val context = memoryTask()
+ val file = PartitionedFile(
+ InternalRow.empty, SparkPath.fromPathString(path), 0, new
File(path).length())
+ val heldAtFirst =
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "4") {
+ asTask(context) {
+ drain(formatReader(checkedSchema(), Nil,
Seq(keyAtLeast(10L)))(file)) { it =>
+ it.next()
+ context.taskMemoryManager.getMemoryConsumptionForThisTask
+ }
+ }
+ }
+ assert(heldAtFirst == 130L,
+ s"the first batch must hold the first row group; got $heldAtFirst")
+ assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0,
+ "a reader closed early must release what it holds")
+ }
+ }
+
+ test("emitting a row group releases its memory in steps") {
+ // Each batch emitted frees its survivors, but the reservation is cut only
once it is a step
+ // past what is still buffered, so a large row group gives its memory back
a few times rather
+ // than per batch.
+ withTempDir { dir =>
+ val path = oneRowGroupOf400000(dir)
+ var asked = Seq.empty[Long]
+ val context = memoryTask(onAcquire = (size, _, _) => asked :+= size)
+ val (rows, reserved) = readInTask(path, context, batchSize = 4096)
+ assert(rows.size == 399990, s"got ${rows.size} rows")
+ // What batch `b` still holds, its own survivors and those after it at 9
bytes each, and the
+ // range. A batch frees 36,864 bytes, so the reservation steps down
every 29 batches.
+ def held(b: Int): Long = 40L + 9L * (399990L - 4096L * b)
+ val expected = (0 until 98).map(b => held(b - b % 29))
+ assert(reserved == expected, s"expected the reservation to step down at
batches 29, 58 " +
+ s"and 87; got ${reserved.distinct}")
+ assert(context.taskMemoryManager.getMemoryConsumptionForThisTask == 0,
"not released")
+ // A growing buffer asks a few times. First for the first survivor, then
for as much again as
+ // it holds up to 802,816 bytes, and then 1 MiB at a time.
+ assert(asked == Seq(49L) ++ (0 to 14).map(49L << _) ++ Seq.fill(2)(1L <<
20),
+ s"got ${asked.size} asks: ${asked.take(20)}")
+ // The 3,599,950 bytes it ends up buffering are not asked for twice
over. Asking for as much
+ // again as it holds would spill the other consumer.
+ readBesideOther(path, rows = 400000L, margin = 3599950L + (1L << 20))
+ }
+ }
+
+ test("memory the task's memory manager denies gives splicing up, and then
the filter") {
+ withTempDir { dir =>
+ val path = twoRowGroupsOf20(dir)
+ // Each read must leave the task holding nothing.
+ def released(context: TaskContext, available: Long): Unit =
+ assert(context.taskMemoryManager().getMemoryConsumptionForThisTask ==
0,
+ s"a read with $available bytes free must release what it reserved")
+ def rows(available: Long, offHeap: Boolean = false): Seq[(Long,
Boolean)] = {
+ val context = memoryTask(available)
+ val (read, _) = readInTask(path, context, offHeap)
+ released(context, available)
+ read
+ }
+ // The bytes a read transfers, with the row groups the budget limited,
see `budgetGiveUps`.
+ def bytes(
+ available: Long,
+ cap: String = "64MB",
+ filter: Expression = keyAtLeast(10L)): (Long, Seq[Long]) = {
+ val metrics = metricMap()
+ val context = memoryTask(available)
+ val (_, read) = readUnderCap(path, checkedSchema(), Seq(filter), cap,
+ batchSize = 4, metrics, Some(context))
+ released(context, available)
+ (read, budgetGiveUps(metrics))
+ }
+ val plain = rows(Long.MaxValue)
+ assert(plain == marked(10L until 40L, checked = true), s"got $plain")
+ // 105 bytes hold the 40 of the first row group's range and 7 of its
keys, but not the 8th.
+ // So splicing is given up there, and in the second row group, as a cap
of 105 bytes gives
+ // it up. The read must cost exactly what the cap does, the key column
read again, while the
+ // filter stays. The metrics tell the two causes apart.
+ val keysDenied = rows(105L)
+ assert(keysDenied == plain, s"denied keys must keep the filter; got
$keysDenied")
+ // 50 bytes take the first survivor's 49, and then grant 1 of the 9 the
second one misses,
+ // which goes back.
+ assert(rows(50L) == plain, "a partly granted reservation must keep the
filter")
+ // 130 bytes grant the seventh survivor 32 of the 98 it asks for, which
covers the 5 it
+ // misses, so the first row group keeps splicing. The second is denied
at its eleventh.
+ val (_, partlyGranted) = bytes(130L)
+ assert(partlyGranted == Seq(0L, 1L, 0L, 0L),
+ s"only the second row group must be denied; got $partlyGranted")
+ val (uncappedBytes, uncapped) = bytes(Long.MaxValue)
+ assert(uncapped == Seq(0L, 0L, 0L, 0L), s"got $uncapped")
+ val (cappedBytes, capped) = bytes(Long.MaxValue, cap = "105b")
+ assert(cappedBytes > uncappedBytes, "a cap of 105 bytes must give
splicing up")
+ assert(capped == Seq(2L, 0L, 0L, 0L), s"both row groups pass the cap;
got $capped")
+ val ((keysDeniedBytes, denied), keysLogged) =
+ withLogged(splicingGivenUpOnMemory)(bytes(105L))
+ assert(keysDeniedBytes == cappedBytes,
+ s"denied keys must read what the cap reads; got $keysDeniedBytes
against $cappedBytes")
+ assert(denied == Seq(0L, 2L, 0L, 0L), s"both row groups are denied; got
$denied")
+ assert(keysLogged == 1, s"which is logged once per split; got
$keysLogged")
+ // Nothing at all does not hold the ranges either, so the filter is
given up. Off heap only
+ // the ranges are asked for, so 40 bytes keep the filter and nothing
gives it up.
+ assert(rows(0L) == marked(0L until 40L, checked = false),
+ "denied ranges must give the filter up")
+ val ((_, rangesDenied), rangesLogged) =
withLogged(filterGivenUpOnMemory)(bytes(0L))
+ assert(rangesDenied == Seq(0L, 0L, 0L, 2L),
+ s"both row groups count as giving the filter up, not splicing; got
$rangesDenied")
+ assert(rangesLogged == 1, s"which is logged once per split; got
$rangesLogged")
+ assert(rows(40L, offHeap = true) == plain, "off heap, ranges that fit
must keep the filter")
+ assert(rows(0L, offHeap = true) == marked(0L until 40L, checked = false),
+ "off heap, denied ranges must give the filter up")
+ // Since off heap a denied request holds no key values, each row group
asks only once.
+ var offHeapAsks = Seq.empty[Long]
+ readInTask(path, memoryTask(0L, (size, _, _) => offHeapAsks :+= size),
offHeap = true)
+ assert(offHeapAsks == Seq(40L, 40L), s"got $offHeapAsks")
+ // Under `k >= 19` the first row group's one survivor is its last.
Denied with its key and
+ // again without it, it must give the filter up.
+ val (_, lastDenied) = bytes(0L, filter = keyAtLeast(19L))
+ assert(lastDenied == Seq(0L, 0L, 0L, 2L), s"got $lastDenied")
+ }
+ }
+
+ test("a row group whose filter phase 1 gives up holds no memory while it is
read whole") {
+ // Under `evenKey` each survivor of `twoRowGroupsOf20` is a range of its
own and 9 bytes of
+ // key. At a cap of 300 bytes the seventh gives splicing up and the eighth
the filter, with 280
+ // bytes of ranges reserved. With 100 bytes free the third is denied with
its key and then
+ // without it, with 98 bytes reserved. Either way the row group is then
read whole.
+ withTempDir { dir =>
+ val path = twoRowGroupsOf20(dir)
+ Seq(("300b", Long.MaxValue, Seq(0L, 0L, 2L, 0L)), ("64MB", 100L, Seq(0L,
0L, 0L, 2L)))
+ .foreach { case (cap, available, expected) =>
+ val name = s"under a cap of $cap with $available bytes free"
+ val metrics = metricMap()
+ readUnderCap(path, checkedSchema(), Seq(evenKey), cap, batchSize =
4, metrics,
+ Some(memoryTask(available)))
+ assert(budgetGiveUps(metrics) == expected, s"$name: got
${budgetGiveUps(metrics)}")
+ val (rows, held) = readInTask(path, memoryTask(available), cap =
cap, filter = evenKey)
+ assert(rows == marked(0L until 40L, checked = false), s"$name: got
$rows")
+ assert(held == Seq.fill(10)(0L), s"$name: nothing may stay reserved;
got $held")
+ }
+ }
+ }
+
+ test("an error raised while reserving fails the read in a form
ignoreCorruptFiles keeps") {
+ // Reserving the buffer can have the task's memory manager spill another
consumer, which can
+ // raise anything. A `RuntimeException`, which a kill's interrupt during a
spill also becomes,
+ // goes out checked, and an `InternalError` in a fatal `UnknownError`.
Under
+ // `ignoreCorruptFiles` either would otherwise be taken for a corrupt
file, and the rest of the
+ // file dropped. A fatal error in the cause chain goes out as itself.
+ withTempDir { dir =>
+ val path = twoRowGroupsOf20(dir)
+ // Reads beside another consumer whose spill throws `error`, and returns
what the read throws.
+ def readBesideFailing(error: Throwable): Throwable = {
+ val context = memoryTask(1000L)
+ val failing = new TestMemoryConsumer(context.taskMemoryManager) {
+ override def spill(size: Long, trigger: MemoryConsumer): Long =
throw error
+ }
+ failing.use(1000L)
+ intercept[Throwable](readInTask(path, context))
+ }
+ Seq(new RuntimeException("spill"), new InternalError("spill")).foreach {
error =>
+ val thrown = readBesideFailing(error)
+ val expected = if (error.isInstanceOf[InternalError]) {
+ classOf[UnknownError]
+ } else {
+ classOf[SparkException]
+ }
+ assert(thrown.getClass == expected && (thrown.getCause eq error),
+ s"expected $error inside ${expected.getSimpleName}; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown),
s"got $thrown")
+ }
+ val deep = new OutOfMemoryError("deep")
+ val thrownDeep = readBesideFailing(new RuntimeException("spill", deep))
+ assert(thrownDeep eq deep, s"expected $deep itself; got $thrownDeep")
+ // Evicting a cached block to disk can raise an `IOException` there too,
which the task's
+ // memory manager passes on as it is.
+ val full = new IOException("No space left on device")
+ val evicting = memoryTask(onAcquire = (_, _, _) => throw full)
+ val thrown = intercept[Throwable](readInTask(path, evicting))
+ assert(thrown.getClass == classOf[SparkException] && (thrown.getCause eq
full),
+ s"expected $full inside a SparkException; got $thrown")
+ assert(!DataSourceUtils.shouldIgnoreCorruptFileException(thrown), s"got
$thrown")
+ }
+ }
+
+ test("the survivor buffer asks for little more memory than it holds") {
+ // Another consumer holds all but what the buffer needs plus a margin. The
task's memory manager
+ // spills that consumer to grant a reservation past what is free, so the
buffer must not ask
+ // for more than its margin: less than as much again as it holds while
small, less than 1 MiB
+ // more once it is past that, see "emitting a row group releases its
memory in steps", and
+ // never past the cap.
+ withTempDir { dir =>
+ // The buffer never holds more than 220 bytes, so it asks for less than
440.
+ val path = twoRowGroupsOf20(dir)
+ readBesideOther(path, rows = 40L, margin = 440L)
+ // Doubling from 196 would reserve 392, which a cap of 300 bytes stops
at 300.
+ readBesideOther(path, rows = 40L, margin = 300L, cap = "300b")
+ }
+ }
+
test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan
Filter") {
// The reader drops the rows a storage filter rejects and the post-scan
Filter takes its word on
// the rows it checked, so the reader's evaluation has to be the one the
Filter would make. A
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
index f78970766681..1f0a261019dd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTestAccess.scala
@@ -232,11 +232,21 @@ object ParquetTestAccess {
f
}
+ private lazy val survivorBatchVectorsMethod = {
+ val m = SparkClassUtils.classForName[Any](
+ classOf[LateMaterializationParquetRecordReader].getName +
"$SurvivorBatch")
+ .getDeclaredMethod("vectors")
+ m.setAccessible(true)
+ m
+ }
+
/** The survivor sets phase 1 queued and the reader has not emitted yet,
oldest first. */
def queuedSurvivorSets(
reader: LateMaterializationParquetRecordReader):
Seq[Seq[WritableColumnVector]] =
- survivorBatchesField.get(reader)
-
.asInstanceOf[java.util.ArrayDeque[Array[WritableColumnVector]]].asScala.map(_.toSeq).toSeq
+
survivorBatchesField.get(reader).asInstanceOf[java.util.ArrayDeque[AnyRef]].asScala.map
{
+ batch =>
survivorBatchVectorsMethod.invoke(batch).asInstanceOf[Array[WritableColumnVector]]
+ .toSeq
+ }.toSeq
private lazy val columnVectorsField = {
val f =
classOf[VectorizedParquetRecordReader].getDeclaredField("columnVectors")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]