zhangyue19921010 commented on code in PR #11256:
URL: https://github.com/apache/paimon/pull/11256#discussion_r4623733783


##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldListaggAgg.java:
##########
@@ -67,21 +76,44 @@ public Object agg(Object accumulator, Object inputField) {
             BinaryString[] accumulatorTokens =
                     splitByWholeSeparatorPreserveAllTokens(mergeFieldSD, 
delimiterBinaryString);
             Set<BinaryString> existingTokens = new 
HashSet<>(accumulatorTokens.length);
+            List<BinaryString> result = new ArrayList<>();
+            int accumulatorSize = 0;
             for (BinaryString token : accumulatorTokens) {
-                existingTokens.add(token);
+                if (!normalizeAccumulator) {
+                    existingTokens.add(token);
+                    continue;
+                }
+                if (StringUtils.isBlank(token.toString()) || 
!existingTokens.add(token)) {

Review Comment:
   Done in 0d4dff7: the original accumulator is now kept and only rebuilt after 
the first blank or duplicate token is found, and blank checks scan the UTF-8 
bytes instead of decoding (falling back only for non-ASCII). On your 512-token 
`category_0` case this is now 42.9 KB and ~21 µs per merge, vs. 92.0 KB / ~43 
µs before and 49.4 KB / ~23 µs on master.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to