Re: [PR] OAK-10541 - Pipelined strategy: improve memory management of transform stage [jackrabbit-oak]

[via GitHub]

thomasmueller commented on code in PR #1202:
URL: https://github.com/apache/jackrabbit-oak/pull/1202#discussion_r1395995398


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatch.java:
##########
@@ -51,49 +49,46 @@ public static NodeStateEntryBatch 
createNodeStateEntryBatch(int bufferSizeBytes,
     }
 
     private final ByteBuffer buffer;
-    private final ArrayList<SortKey> sortBuffer;
     private final int maxEntries;
+    private int numberOfEntries = 0;

Review Comment:
   0 is the default
   ```suggestion
       private int numberOfEntries;
   ```



##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedTransformTask.java:
##########
@@ -88,6 +89,7 @@ public int getThreadId() {
     private final TransformStageStatistics statistics;
     private final int threadId = threadIdGenerator.getAndIncrement();
     private long totalEnqueueDelayMillis = 0;
+    private long totalEmptyBatchQueueWaitTimeMillis = 0;

Review Comment:
   ```suggestion
       private long totalEnqueueDelayMillis;
       private long totalEmptyBatchQueueWaitTimeMillis;
   ```



##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/NodeStateEntryBatch.java:
##########
@@ -51,49 +49,46 @@ public static NodeStateEntryBatch 
createNodeStateEntryBatch(int bufferSizeBytes,
     }
 
     private final ByteBuffer buffer;
-    private final ArrayList<SortKey> sortBuffer;
     private final int maxEntries;
+    private int numberOfEntries = 0;
+    private int sizeOfEntriesBytes = 0;

Review Comment:
   ```suggestion
       private int sizeOfEntriesBytes;
   ```



##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedSortBatchTask.java:
##########
@@ -66,9 +68,12 @@ public long getTotalEntries() {
     private final BlockingQueue<NodeStateEntryBatch> nonEmptyBuffersQueue;
     private final BlockingQueue<Path> sortedFilesQueue;
     private final Path sortWorkDir;
-    private final byte[] copyBuffer = new byte[4096];
+    private final ArrayList<SortKey> sortBuffer = new ArrayList<>(32 * 1024);
     private long entriesProcessed = 0;
     private long batchesProcessed = 0;
+    private long timeCreatingSortArrayMillis = 0;
+    private long timeSortingMillis = 0;
+    private long timeWritingMillis = 0;

Review Comment:
   ```suggestion
       private long entriesProcessed;
       private long batchesProcessed;
       private long timeCreatingSortArrayMillis;
       private long timeSortingMillis;
       private long timeWritingMillis;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org