wangyum commented on code in PR #3927:
URL: https://github.com/apache/parquet-java/pull/3927#discussion_r4570850268
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/MemoryManager.java:
##########
@@ -93,44 +114,65 @@ synchronized void addWriter(InternalParquetRecordWriter<?>
writer, Long allocati
* @param writer the writer that has been closed
*/
synchronized void removeWriter(InternalParquetRecordWriter<?> writer) {
- writerList.remove(writer);
+ Long allocation = writerList.remove(writer);
+ if (allocation == null) {
+ return;
+ }
+ totalAllocations -= allocation;
+ allocationCounts.compute(allocation, (key, count) -> (count == null ||
count <= 1) ? null : count - 1);
if (!writerList.isEmpty()) {
- updateAllocation();
+ recomputeScale();
+ } else {
+ // No live writers are left: reset so a stale shrunken scale cannot be
observed or used
+ // while the manager sits idle (writers read scale lazily, so it is the
sole live state).
+ scale = 1.0;
}
}
/**
- * Update the allocated size of each writer based on the current allocations
and pool size.
+ * Recompute the scale factor applied to every writer's row group size from
the current total
+ * allocation and pool size. Writers do not store a pushed threshold; each
one reads {@link
+ * #getScale()} lazily, so this only has to update the handful of fields
below instead of
+ * visiting every writer in {@code writerList}.
*/
- private void updateAllocation() {
- long totalAllocations = 0;
- for (Long allocation : writerList.values()) {
- totalAllocations += allocation;
- }
+ private void recomputeScale() {
+ double newScale;
if (totalAllocations <= totalMemoryPool) {
- scale = 1.0;
+ newScale = 1.0;
} else {
- scale = (double) totalMemoryPool / totalAllocations;
+ newScale = (double) totalMemoryPool / totalAllocations;
LOG.warn(String.format(
"Total allocation exceeds %.2f%% (%,d bytes) of heap memory\n"
+ "Scaling row group sizes to %.2f%% for %d writers",
- 100 * memoryPoolRatio, totalMemoryPool, 100 * scale,
writerList.size()));
+ 100 * memoryPoolRatio, totalMemoryPool, 100 * newScale,
writerList.size()));
for (Runnable callBack : callBacks.values()) {
// we do not really want to start a new thread here.
callBack.run();
}
Review Comment:
Thanks for catching this and for the thorough repro — really appreciate the
detailed write-up! Committed your suggestion directly (7f232eb6f), and I've
also added a regression test (`testScaleReflectsStateAfterCallbackReentrancy`)
that reproduces the exact scenario: two writers whose combined allocation
exceeds the pool, with a scale-exceeded callback that reentrantly closes one
via `removeWriter()`. Verified it fails without the fix (0.8333 instead of 1.0,
matching your 100/60/60 example) and passes with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]