JingsongLi commented on code in PR #11256:
URL: https://github.com/apache/paimon/pull/11256#discussion_r4623618397
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldListaggAgg.java:
##########
@@ -67,21 +76,44 @@ public Object agg(Object accumulator, Object inputField) {
BinaryString[] accumulatorTokens =
splitByWholeSeparatorPreserveAllTokens(mergeFieldSD,
delimiterBinaryString);
Set<BinaryString> existingTokens = new
HashSet<>(accumulatorTokens.length);
+ List<BinaryString> result = new ArrayList<>();
+ int accumulatorSize = 0;
for (BinaryString token : accumulatorTokens) {
- existingTokens.add(token);
+ if (!normalizeAccumulator) {
+ existingTokens.add(token);
+ continue;
+ }
+ if (StringUtils.isBlank(token.toString()) ||
!existingTokens.add(token)) {
Review Comment:
[P2] Avoid rebuilding an already-normalized accumulator on repeated values
Each merge now decodes every existing token and builds a second list of
tokens and delimiters, even when the accumulator is already distinct and the
input is a value it contains. That rebuilt list is discarded at line 115. For a
group collecting tags, every repeated tag pays this cost during primary-key
reads and compaction. I compared the master and PR aggregators against the same
dependencies on JDK 11, using 512 distinct `category_N` tokens and 6,000
updates with `category_0` per round. The outputs matched. In two JVMs per
variant after warm-up, allocation rose from about 50.5 to 81.9 KB per merge and
the warmed rounds took roughly 1.7–1.9× as long. Could we keep the original
accumulator in the common case, materialize normalized output only after
finding a token that must be removed, and avoid allocating a Java string for
each existing token when checking whether it is blank?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]