Qing Fu created SPARK-59509:
-------------------------------
Summary: Fix record undercount in UnsafeSorterSpillMerger when
merging in-memory records
Key: SPARK-59509
URL: https://issues.apache.org/jira/browse/SPARK-59509
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Qing Fu
{{UnsafeSorterSpillMerger.addSpillIfNotEmpty}} reads the incoming iterator's
record count *after* calling {{loadNext()}}:
{code:java}
public void addSpillIfNotEmpty(UnsafeSorterIterator spillReader) throws
IOException {
if (spillReader.hasNext()) {
spillReader.loadNext();
priorityQueue.add(spillReader);
numRecords += spillReader.getNumRecords();
}
}
{code}
The two {{UnsafeSorterIterator}} implementations that reach this method
disagree about what {{getNumRecords()}} means:
* {{UnsafeSorterSpillReader.getNumRecords()}} returns the count stored in the
spill file header. It never changes, however many times {{loadNext()}} is
called.
* {{UnsafeExternalSorter.SpillableIterator.getNumRecords()}} returns the number
of records _remaining_, and decrements on every {{loadNext()}}.
{{UnsafeExternalSorter.getSortedIterator()}} adds its in-memory
{{SpillableIterator}} to the merger alongside the spill readers, so the merged
iterator reports one record fewer than it will actually produce, per in-memory
iterator.
This is latent on master: the merged iterator is consumed with a plain {{while
(hasNext())}} loop, and no caller reads the declared count, so there is no
user-visible symptom today. It is still an incorrect value, and it is a trap
for any future caller that hands a merged iterator to
{{UnsafeSorterSpillWriter}}, which enforces the declared count and fails with
{{IllegalStateException: Number of records written exceeded numRecordsToWrite}}.
The fix is to read the count before {{loadNext()}}.
Reproducer: two spills plus five records still in memory.
{{getSortedIterator().getNumRecords()}} returns 14, and the iterator then
yields 15 records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
