JingsongLi commented on code in PR #11256:
URL: https://github.com/apache/paimon/pull/11256#discussion_r4623618397


##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldListaggAgg.java:
##########
@@ -67,21 +76,44 @@ public Object agg(Object accumulator, Object inputField) {
             BinaryString[] accumulatorTokens =
                     splitByWholeSeparatorPreserveAllTokens(mergeFieldSD, 
delimiterBinaryString);
             Set<BinaryString> existingTokens = new 
HashSet<>(accumulatorTokens.length);
+            List<BinaryString> result = new ArrayList<>();
+            int accumulatorSize = 0;
             for (BinaryString token : accumulatorTokens) {
-                existingTokens.add(token);
+                if (!normalizeAccumulator) {
+                    existingTokens.add(token);
+                    continue;
+                }
+                if (StringUtils.isBlank(token.toString()) || 
!existingTokens.add(token)) {

Review Comment:
   [P2] Avoid rebuilding an already-normalized accumulator on repeated values
   
   Each merge now decodes every existing token and builds a second list of 
tokens and delimiters, even when the accumulator is already distinct and the 
input is a value it contains. That rebuilt list is discarded at line 115. For a 
group collecting tags, every repeated tag pays this cost during primary-key 
reads and compaction. I compared the master and PR aggregators against the same 
dependencies on JDK 11, using 512 distinct `category_N` tokens and 6,000 
updates with `category_0` per round. The outputs matched. In two JVMs per 
variant after warm-up, allocation rose from about 50.5 to 81.9 KB per merge and 
the warmed rounds took roughly 1.7–1.9× as long. Could we keep the original 
accumulator in the common case, materialize normalized output only after 
finding a token that must be removed, and avoid allocating a Java string for 
each existing token when checking whether it is blank?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to