dossett commented on code in PR #3927:
URL: https://github.com/apache/parquet-java/pull/3927#discussion_r4566160701


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/MemoryManager.java:
##########
@@ -93,44 +114,65 @@ synchronized void addWriter(InternalParquetRecordWriter<?> 
writer, Long allocati
    * @param writer the writer that has been closed
    */
   synchronized void removeWriter(InternalParquetRecordWriter<?> writer) {
-    writerList.remove(writer);
+    Long allocation = writerList.remove(writer);
+    if (allocation == null) {
+      return;
+    }
+    totalAllocations -= allocation;
+    allocationCounts.compute(allocation, (key, count) -> (count == null || 
count <= 1) ? null : count - 1);
     if (!writerList.isEmpty()) {
-      updateAllocation();
+      recomputeScale();
+    } else {
+      // No live writers are left: reset so a stale shrunken scale cannot be 
observed or used
+      // while the manager sits idle (writers read scale lazily, so it is the 
sole live state).
+      scale = 1.0;
     }
   }
 
   /**
-   * Update the allocated size of each writer based on the current allocations 
and pool size.
+   * Recompute the scale factor applied to every writer's row group size from 
the current total
+   * allocation and pool size. Writers do not store a pushed threshold; each 
one reads {@link
+   * #getScale()} lazily, so this only has to update the handful of fields 
below instead of
+   * visiting every writer in {@code writerList}.
    */
-  private void updateAllocation() {
-    long totalAllocations = 0;
-    for (Long allocation : writerList.values()) {
-      totalAllocations += allocation;
-    }
+  private void recomputeScale() {
+    double newScale;
     if (totalAllocations <= totalMemoryPool) {
-      scale = 1.0;
+      newScale = 1.0;
     } else {
-      scale = (double) totalMemoryPool / totalAllocations;
+      newScale = (double) totalMemoryPool / totalAllocations;
       LOG.warn(String.format(
           "Total allocation exceeds %.2f%% (%,d bytes) of heap memory\n"
               + "Scaling row group sizes to %.2f%% for %d writers",
-          100 * memoryPoolRatio, totalMemoryPool, 100 * scale, 
writerList.size()));
+          100 * memoryPoolRatio, totalMemoryPool, 100 * newScale, 
writerList.size()));
       for (Runnable callBack : callBacks.values()) {
         // we do not really want to start a new thread here.
         callBack.run();
       }

Review Comment:
   [Approved CODEX generated comment]
   
   Could we recalculate `newScale` after the callbacks? A callback can close a 
writer and reenter `removeWriter()` (the monitor is reentrant), changing 
`totalAllocations`. The outer call then uses its stale local `newScale` for the 
minimum-allocation check and overwrites the scale calculated by the nested 
removal.
   
   For a pool of 100 and two allocations of 60, a callback removing the first 
writer leaves total allocation at 60. The scale should return to 1.0, but this 
code publishes 0.8333; with a minimum allocation of 55 it incorrectly rejects 
the remaining writer. An isolated Docker probe using the exact before/after 
`MemoryManager` sources and stub writer registrations reproduced both cases; 
this recalculation fixes them.
   
   ```suggestion
         }
         // Callbacks may close writers and reenter removeWriter(), changing 
totalAllocations.
         // Recalculate before validating and publishing the scale.
         newScale = totalAllocations <= totalMemoryPool
             ? 1.0
             : (double) totalMemoryPool / totalAllocations;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to