dossett commented on code in PR #3927:
URL: https://github.com/apache/parquet-java/pull/3927#discussion_r4566160701
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/MemoryManager.java:
##########
@@ -93,44 +114,65 @@ synchronized void addWriter(InternalParquetRecordWriter<?>
writer, Long allocati
* @param writer the writer that has been closed
*/
synchronized void removeWriter(InternalParquetRecordWriter<?> writer) {
- writerList.remove(writer);
+ Long allocation = writerList.remove(writer);
+ if (allocation == null) {
+ return;
+ }
+ totalAllocations -= allocation;
+ allocationCounts.compute(allocation, (key, count) -> (count == null ||
count <= 1) ? null : count - 1);
if (!writerList.isEmpty()) {
- updateAllocation();
+ recomputeScale();
+ } else {
+ // No live writers are left: reset so a stale shrunken scale cannot be
observed or used
+ // while the manager sits idle (writers read scale lazily, so it is the
sole live state).
+ scale = 1.0;
}
}
/**
- * Update the allocated size of each writer based on the current allocations
and pool size.
+ * Recompute the scale factor applied to every writer's row group size from
the current total
+ * allocation and pool size. Writers do not store a pushed threshold; each
one reads {@link
+ * #getScale()} lazily, so this only has to update the handful of fields
below instead of
+ * visiting every writer in {@code writerList}.
*/
- private void updateAllocation() {
- long totalAllocations = 0;
- for (Long allocation : writerList.values()) {
- totalAllocations += allocation;
- }
+ private void recomputeScale() {
+ double newScale;
if (totalAllocations <= totalMemoryPool) {
- scale = 1.0;
+ newScale = 1.0;
} else {
- scale = (double) totalMemoryPool / totalAllocations;
+ newScale = (double) totalMemoryPool / totalAllocations;
LOG.warn(String.format(
"Total allocation exceeds %.2f%% (%,d bytes) of heap memory\n"
+ "Scaling row group sizes to %.2f%% for %d writers",
- 100 * memoryPoolRatio, totalMemoryPool, 100 * scale,
writerList.size()));
+ 100 * memoryPoolRatio, totalMemoryPool, 100 * newScale,
writerList.size()));
for (Runnable callBack : callBacks.values()) {
// we do not really want to start a new thread here.
callBack.run();
}
Review Comment:
[Approved CODEX generated comment]
Could we recalculate `newScale` after the callbacks? A callback can close a
writer and reenter `removeWriter()` (the monitor is reentrant), changing
`totalAllocations`. The outer call then uses its stale local `newScale` for the
minimum-allocation check and overwrites the scale calculated by the nested
removal.
For a pool of 100 and two allocations of 60, a callback removing the first
writer leaves total allocation at 60. The scale should return to 1.0, but this
code publishes 0.8333; with a minimum allocation of 55 it incorrectly rejects
the remaining writer. An isolated Docker probe using the exact before/after
`MemoryManager` sources and stub writer registrations reproduced both cases;
this recalculation fixes them.
```suggestion
}
// Callbacks may close writers and reenter removeWriter(), changing
totalAllocations.
// Recalculate before validating and publishing the scale.
newScale = totalAllocations <= totalMemoryPool
? 1.0
: (double) totalMemoryPool / totalAllocations;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]