This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3f3b2a0c4c8 [fix](profile) add SafeStringBuilder to avoid OOM in
profile building (#56545)
3f3b2a0c4c8 is described below
commit 3f3b2a0c4c80cd05762d072c6ffdb01aedb0b082
Author: xuchenhao <[email protected]>
AuthorDate: Tue Sep 30 08:51:42 2025 +0800
[fix](profile) add SafeStringBuilder to avoid OOM in profile building
(#56545)
related issue #56087
### What problem does this PR solve?
This PR addresses a critical issue where Apache Doris may encounter a
fatal `OutOfMemoryError` during the construction of execution profiles.
The root cause is unbounded memory growth in the `StringBuilder` used
inside the recursive method `RuntimeProfile.prettyPrint` and
`RuntimeProfile.printChildCounters`. When the number of counters is
extremely large or the counter hierarchy is deeply nested (sometimes
even circular), `StringBuilder` attempts to expand its internal
character array beyond Java’s maximum allowed array size
(`Integer.MAX_VALUE`), leading to:
`java.lang.OutOfMemoryError: Required array length 2147483638+34 is too
large
`
This not only crashes the current thread but can also impact the
stability of the entire Frontend process.
To resolve this, this PR introduces a safe alternative:
`SafeStringBuilder`, which allows profile strings to be constructed up
to a safe upper limit. Once the limit is reached, the content is
truncated gracefully and marked with a `[TRUNCATED]` flag. Early-exit
checks are also added to avoid unnecessary computation once truncation
occurs.
## add SafeStringBuilder to avoid OOM in profile building
### 1. Background
During execution, Doris collects and prints execution profiles
(execution stats and counters). The printing is implemented recursively
via:
`RuntimeProfile.printChildCounters(String prefix, String counterName,
StringBuilder builder)`
This can trigger deep recursion, particularly if there are:
- A large number of counters
- Complex or circular child counter relationships
This unbounded recursion causes the `StringBuilder` to allocate more and
more memory, until Java refuses the allocation and throws
`OutOfMemoryError`. The error stack trace typically points to:
`java.lang.StringBuilder.append → newCapacity → hugeLength →
OutOfMemoryError`.
This PR addresses both the root cause (unsafe memory growth) and its
consequences (crash).
### 2. Key Code Changes
#### 2.1 Introduced Class: SafeStringBuilder.java
A new utility class `SafeStringBuilder` has been Introduced to replace
the original `StringBuilder` in profile building code. It provides the
following features:
- Enforces a maximum capacity limit (default: `Integer.MAX_VALUE - 16`)
- Automatically truncates appended content once the limit is reached
- Appends `[TRUNCATED]` to the final output if truncation occurs
#### 2.2 Refactored Methods
The following key methods have been updated to use `SafeStringBuilder`
instead of `StringBuilder`:
- `Profile.getProfileByLevel()`
- `Profile.getChangedSessionVars(SafeStringBuilder builder)`
- `Profile.getExecutionProfileContent(SafeStringBuilder builder)`
- `Profile.getOnStorageProfile(SafeStringBuilder builder)`
- `RuntimeProfile.prettyPrint(SafeStringBuilder builder, String prefix)`
- `RuntimeProfile.printChildCounters(String prefix, String counterName,
SafeStringBuilder builder)`
- `SummaryProfile.prettyPrint(SafeStringBuilder builder)`
#### 2.3 Early Exit After Truncation
To avoid unnecessary computation or memory usage after truncation
occurs, additional early-exit checks have been added to all major
profile building methods. Specifically:
- Before any recursive calls (e.g., in `RuntimeProfile.prettyPrint`),
`builder.isTruncated()` is checked, and further traversal is skipped if
true.
- Each stage in profile generation (session variables, execution
profile, etc.) now checks if truncation has already occurred, and exits
early if so.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../doris/common/profile/ExecutionProfile.java | 5 +-
.../org/apache/doris/common/profile/Profile.java | 36 +++--
.../doris/common/profile/RuntimeProfile.java | 29 +++-
.../doris/common/profile/SummaryProfile.java | 3 +-
.../doris/common/util/SafeStringBuilder.java | 79 ++++++++++
.../common/profile/ProfilePersistentTest.java | 15 +-
.../doris/common/profile/ProfileStructureTest.java | 3 +-
.../apache/doris/common/profile/ProfileTest.java | 3 +-
.../common/profile/RuntimeProfileMergeTest.java | 5 +-
.../doris/common/profile/RuntimeProfileTest.java | 5 +-
.../doris/common/util/SafeStringBuilderTest.java | 134 +++++++++++++++++
.../suites/query_profile/profile_size_limit.groovy | 164 +++++++++++++++++++++
12 files changed, 446 insertions(+), 35 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
index 91c0f510a84..2f6b0992ef6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
@@ -20,6 +20,7 @@ package org.apache.doris.common.profile;
import org.apache.doris.common.Pair;
import org.apache.doris.common.Status;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.planner.PlanFragmentId;
import org.apache.doris.thrift.TDetailedReportParams;
import org.apache.doris.thrift.TNetworkAddress;
@@ -320,12 +321,12 @@ public class ExecutionProfile {
}
public String toString() {
- StringBuilder sb = new StringBuilder();
+ SafeStringBuilder sb = new SafeStringBuilder();
root.prettyPrint(sb, "");
return sb.toString();
}
- public void prettyPrint(StringBuilder sb, String prefix) {
+ public void prettyPrint(SafeStringBuilder sb, String prefix) {
root.prettyPrint(sb, prefix);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
index 61c21ff61af..036a24e79d6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
@@ -20,7 +20,9 @@ package org.apache.doris.common.profile;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.Config;
import org.apache.doris.common.io.Text;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.nereids.NereidsPlanner;
import org.apache.doris.nereids.stats.HboPlanInfoProvider;
import org.apache.doris.nereids.stats.HboPlanStatisticsManager;
@@ -336,12 +338,24 @@ public class Profile {
}
public String getProfileByLevel() {
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
+ if (DebugPointUtil.isEnable("Profile.profileSizeLimit")) {
+ DebugPointUtil.DebugPoint debugPoint =
DebugPointUtil.getDebugPoint("Profile.profileSizeLimit");
+ int maxProfileSize = debugPoint.param("profileSizeLimit", 0);
+ builder = new SafeStringBuilder(maxProfileSize);
+ LOG.info("DebugPoint:Profile.profileSizeLimit, MAX_PROFILE_SIZE =
{}", maxProfileSize);
+ }
// add summary to builder
summaryProfile.prettyPrint(builder);
- getChangedSessionVars(builder);
- getExecutionProfileContent(builder);
- getOnStorageProfile(builder);
+ if (!builder.isTruncated()) {
+ getChangedSessionVars(builder);
+ }
+ if (!builder.isTruncated()) {
+ getExecutionProfileContent(builder);
+ }
+ if (!builder.isTruncated()) {
+ getOnStorageProfile(builder);
+ }
return builder.toString();
}
@@ -460,9 +474,9 @@ public class Profile {
}
// Return if profile has been stored to storage
- public void getExecutionProfileContent(StringBuilder builder) {
+ public void getExecutionProfileContent(SafeStringBuilder builder) {
if (builder == null) {
- builder = new StringBuilder();
+ builder = new SafeStringBuilder();
}
if (profileHasBeenStored()) {
@@ -653,7 +667,7 @@ public class Profile {
// Write summary profile and execution profile content to memory
this.summaryProfile.write(memoryDataStream);
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
getChangedSessionVars(builder);
getExecutionProfileContent(builder);
byte[] executionProfileBytes =
builder.toString().getBytes(StandardCharsets.UTF_8);
@@ -758,9 +772,9 @@ public class Profile {
this.changedSessionVarCache = changedSessionVar;
}
- private void getChangedSessionVars(StringBuilder builder) {
+ private void getChangedSessionVars(SafeStringBuilder builder) {
if (builder == null) {
- builder = new StringBuilder();
+ builder = new SafeStringBuilder();
}
if (profileHasBeenStored()) {
return;
@@ -792,7 +806,7 @@ public class Profile {
return durationMs > Config.qe_slow_log_ms;
}
- void getOnStorageProfile(StringBuilder builder) {
+ void getOnStorageProfile(SafeStringBuilder builder) {
if (!profileHasBeenStored()) {
return;
}
@@ -874,7 +888,7 @@ public class Profile {
}
public String toString() {
- StringBuilder stringBuilder = new StringBuilder();
+ SafeStringBuilder stringBuilder = new SafeStringBuilder();
getExecutionProfileContent(stringBuilder);
return stringBuilder.toString();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/RuntimeProfile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/RuntimeProfile.java
index f0cf81ed9fb..4067cd3e1fb 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/RuntimeProfile.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/RuntimeProfile.java
@@ -21,6 +21,7 @@ import org.apache.doris.common.Pair;
import org.apache.doris.common.Reference;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.thrift.TCounter;
import org.apache.doris.thrift.TPlanNodeRuntimeStatsItem;
@@ -207,7 +208,6 @@ public class RuntimeProfile {
}
-
public Counter addCounter(String name, TUnit type, String
parentCounterName) {
counterLock.writeLock().lock();
try {
@@ -405,14 +405,15 @@ public class RuntimeProfile {
// 2. Info Strings
// 3. Counters
// 4. Children
- public void prettyPrint(StringBuilder builder, String prefix) {
+ public void prettyPrint(SafeStringBuilder builder, String prefix) {
// 1. profile name
- builder.append(prefix).append(name).append(":");
-
- builder.append("\n");
+ builder.append(prefix).append(name).append(":").append("\n");
// plan node info
printPlanNodeInfo(prefix + " ", builder);
+ if (builder.isTruncated()) {
+ return;
+ }
// 2. info String
infoStringsLock.readLock().lock();
@@ -430,6 +431,9 @@ public class RuntimeProfile {
} finally {
infoStringsLock.readLock().unlock();
}
+ if (builder.isTruncated()) {
+ return;
+ }
// 3. counters
try {
@@ -437,12 +441,18 @@ public class RuntimeProfile {
} catch (Exception e) {
builder.append("print child counters error:
").append(e.getMessage());
}
+ if (builder.isTruncated()) {
+ return;
+ }
// 4. children
childLock.readLock().lock();
try {
for (int i = 0; i < childList.size(); i++) {
+ if (builder.isTruncated()) {
+ return;
+ }
Pair<RuntimeProfile, Boolean> pair = childList.get(i);
boolean indent = pair.second;
RuntimeProfile profile = pair.first;
@@ -453,7 +463,7 @@ public class RuntimeProfile {
}
}
- private void printPlanNodeInfo(String prefix, StringBuilder builder) {
+ private void printPlanNodeInfo(String prefix, SafeStringBuilder builder) {
if (planNodeInfos.isEmpty()) {
return;
}
@@ -480,7 +490,7 @@ public class RuntimeProfile {
}
public String toString() {
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
prettyPrint(builder, "");
return builder.toString();
}
@@ -580,7 +590,7 @@ public class RuntimeProfile {
}
}
- private void printChildCounters(String prefix, String counterName,
StringBuilder builder) {
+ private void printChildCounters(String prefix, String counterName,
SafeStringBuilder builder) {
Set<String> childCounterSet = childCounterMap.get(counterName);
if (childCounterSet == null) {
return;
@@ -589,6 +599,9 @@ public class RuntimeProfile {
counterLock.readLock().lock();
try {
for (String childCounterName : childCounterSet) {
+ if (builder.isTruncated()) {
+ return;
+ }
Counter counter = this.counterMap.get(childCounterName);
if (counter != null) {
builder.append(prefix).append(" -
").append(childCounterName).append(": ")
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
index 5454d1b5b7e..4e99ee14cf7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
@@ -19,6 +19,7 @@ package org.apache.doris.common.profile;
import org.apache.doris.common.Config;
import org.apache.doris.common.io.Text;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.qe.ConnectContext;
@@ -403,7 +404,7 @@ public class SummaryProfile {
return executionSummaryProfile;
}
- public void prettyPrint(StringBuilder builder) {
+ public void prettyPrint(SafeStringBuilder builder) {
summaryProfile.prettyPrint(builder, "");
executionSummaryProfile.prettyPrint(builder, "");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/SafeStringBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/SafeStringBuilder.java
new file mode 100644
index 00000000000..a492b34fd4a
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/SafeStringBuilder.java
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.profile.Profile;
+
+import lombok.Getter;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class SafeStringBuilder {
+ private StringBuilder builder = new StringBuilder();
+ @Getter
+ private long maxCapacity;
+ @Getter
+ private boolean truncated = false;
+ private Logger log = LogManager.getLogger(Profile.class);
+
+ public SafeStringBuilder() {
+ this(Integer.MAX_VALUE);
+ }
+
+ public SafeStringBuilder(int maxCapacity) {
+ if (maxCapacity < 16) {
+ log.warn("SafeStringBuilder max capacity {} must be greater than
16", maxCapacity);
+ maxCapacity = 16;
+ }
+ this.maxCapacity = maxCapacity - 16;
+ }
+
+ public SafeStringBuilder append(String str) {
+ if (!truncated) {
+ if (builder.length() + str.length() <= maxCapacity) {
+ builder.append(str);
+ } else {
+ log.warn("Append str truncated, builder length(): {}, str
length: {}, max capacity: {}",
+ builder.length(), str.length(), maxCapacity);
+ builder.append(str, 0, (int) (maxCapacity - builder.length()));
+ markTruncated();
+ }
+ }
+ return this;
+ }
+
+ public SafeStringBuilder append(Object obj) {
+ return append(String.valueOf(obj));
+ }
+
+ public int length() {
+ return builder.length();
+ }
+
+ public String toString() {
+ if (truncated) {
+ return builder.toString() + "\n...[TRUNCATED]";
+ }
+ return builder.toString();
+ }
+
+ private void markTruncated() {
+ truncated = true;
+ log.warn("SafeStringBuilder exceeded max capacity {}", maxCapacity);
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePersistentTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePersistentTest.java
index 7f216682b78..3e289e2d61b 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePersistentTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfilePersistentTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.common.profile;
import org.apache.doris.common.profile.SummaryProfile.SummaryBuilder;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.thrift.QueryState;
import org.apache.doris.thrift.TUniqueId;
@@ -135,9 +136,9 @@ public class ProfilePersistentTest {
}
Assert.assertFalse(readFailed);
- StringBuilder builder1 = new StringBuilder();
+ SafeStringBuilder builder1 = new SafeStringBuilder();
summaryProfile.prettyPrint(builder1);
- StringBuilder builder2 = new StringBuilder();
+ SafeStringBuilder builder2 = new SafeStringBuilder();
deserializedSummaryProfile.prettyPrint(builder2);
Assert.assertNotEquals("", builder1.toString());
@@ -201,7 +202,7 @@ public class ProfilePersistentTest {
Assert.assertEquals(profile.getQueryFinishTimestamp(),
readProfile.getQueryFinishTimestamp());
// Verify content is readable
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
readProfile.getOnStorageProfile(builder);
Assert.assertFalse(Strings.isNullOrEmpty(builder.toString()));
@@ -268,7 +269,7 @@ public class ProfilePersistentTest {
// Test with corrupted file
File profileFile = new File(profile.getProfileStoragePath());
FileUtils.writeStringToFile(profileFile, "corrupted content",
StandardCharsets.UTF_8);
- StringBuilder corruptedContent = new StringBuilder();
+ SafeStringBuilder corruptedContent = new SafeStringBuilder();
profile.getOnStorageProfile(corruptedContent);
Assert.assertTrue(corruptedContent.toString().contains("Failed to
read profile"));
@@ -381,7 +382,7 @@ public class ProfilePersistentTest {
Path tempDir = Files.createTempDirectory("profile-persistent-test");
try {
// First get profile content before storage
- StringBuilder beforeStorage = new StringBuilder();
+ SafeStringBuilder beforeStorage = new SafeStringBuilder();
profile.getExecutionProfileContent(beforeStorage);
// Write to storage
@@ -389,7 +390,7 @@ public class ProfilePersistentTest {
// Test with non-stored profile
Profile nonStoredProfile = constructRandomProfile(1);
- StringBuilder nonStoredBuilder = new StringBuilder();
+ SafeStringBuilder nonStoredBuilder = new SafeStringBuilder();
nonStoredProfile.getOnStorageProfile(nonStoredBuilder);
Assert.assertEquals("", nonStoredBuilder.toString());
@@ -403,7 +404,7 @@ public class ProfilePersistentTest {
zos.close();
fos.close();
- StringBuilder invalidBuilder = new StringBuilder();
+ SafeStringBuilder invalidBuilder = new SafeStringBuilder();
profile.getOnStorageProfile(invalidBuilder);
Assert.assertTrue(invalidBuilder.toString().contains("Failed to
read profile"));
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileStructureTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileStructureTest.java
index b9e52119c1f..0ec0dc1e3f0 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileStructureTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileStructureTest.java
@@ -18,6 +18,7 @@
package org.apache.doris.common.profile;
import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.thrift.TNetworkAddress;
import org.apache.doris.thrift.TUniqueId;
@@ -54,7 +55,7 @@ public class ProfileStructureTest {
TUniqueId queryId = new TUniqueId(1L, 2L);
ExecutionProfile profile = new ExecutionProfile(queryId,
Lists.newArrayList(0, 1));
- StringBuilder sb = new StringBuilder();
+ SafeStringBuilder sb = new SafeStringBuilder();
profile.prettyPrint(sb, " ");
String result = sb.toString();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileTest.java
index 53f8ebe548a..b23af1f4980 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.common.profile;
import org.apache.doris.common.Config;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.thrift.TUniqueId;
import mockit.Expectations;
@@ -244,7 +245,7 @@ public class ProfileTest {
profile.setQueryFinishTimestamp(System.currentTimeMillis());
profile.writeToStorage(testProfileStoragePath);
profile.releaseMemory();
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
profile.getOnStorageProfile(builder);
// Verify we got content
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileMergeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileMergeTest.java
index 9d2b6919337..d6171318191 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileMergeTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileMergeTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.common.profile;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.thrift.TCounter;
import org.apache.doris.thrift.TRuntimeProfileNode;
import org.apache.doris.thrift.TRuntimeProfileTree;
@@ -164,7 +165,7 @@ public class RuntimeProfileMergeTest {
RuntimeProfile mergedProfile = new RuntimeProfile("mergedProfile");
RuntimeProfile.mergeProfiles(Lists.newArrayList(profile1, profile2,
profile3), mergedProfile, null);
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
mergedProfile.prettyPrint(builder, "\t");
LOG.info("Merged profile:\n{}", builder.toString());
@@ -275,7 +276,7 @@ public class RuntimeProfileMergeTest {
RuntimeProfile mergedProfile = new RuntimeProfile("mergedProfile");
RuntimeProfile.mergeProfiles(Lists.newArrayList(profile1, profile2),
mergedProfile, null);
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
mergedProfile.prettyPrint(builder, "\t");
LOG.info("Merged profile:\n{}", builder.toString());
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileTest.java
index 977a54db6d7..7e40c7f146f 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/RuntimeProfileTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.common.profile;
+import org.apache.doris.common.util.SafeStringBuilder;
import org.apache.doris.thrift.TCounter;
import org.apache.doris.thrift.TRuntimeProfileNode;
import org.apache.doris.thrift.TRuntimeProfileTree;
@@ -93,7 +94,7 @@ public class RuntimeProfileTest {
profile.update(tprofileTree);
Assert.assertEquals(profile.getInfoString("key"), "value4");
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
profile.prettyPrint(builder, "");
Assert.assertEquals(builder.toString(),
"profileName:\n - key: value4\n - key3: value3\n");
@@ -178,7 +179,7 @@ public class RuntimeProfileTest {
tnodeASon.name = "ASON";
profile.update(tprofileTree);
- StringBuilder builder = new StringBuilder();
+ SafeStringBuilder builder = new SafeStringBuilder();
profile.computeTimeInProfile();
profile.prettyPrint(builder, "");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/util/SafeStringBuilderTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/util/SafeStringBuilderTest.java
new file mode 100644
index 00000000000..cfd4b67dff7
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/util/SafeStringBuilderTest.java
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+
+public class SafeStringBuilderTest {
+ private SafeStringBuilder builder;
+ private final int testMaxCapacity = 100;
+
+ @Before
+ public void setUp() {
+ builder = new SafeStringBuilder(testMaxCapacity);
+ }
+
+ @Test
+ public void testDefaultConstructor() {
+ SafeStringBuilder defaultBuilder = new SafeStringBuilder();
+ Assert.assertEquals(Integer.MAX_VALUE - 16,
defaultBuilder.getMaxCapacity());
+ }
+
+ @Test
+ public void testConstructorWithSmallCapacity() {
+ SafeStringBuilder smallBuilder = new SafeStringBuilder(10);
+ Assert.assertEquals(0, smallBuilder.getMaxCapacity());
+ }
+
+ @Test
+ public void testAppendStringWithinCapacity() {
+ String testString = "Hello";
+ builder.append(testString);
+ Assert.assertEquals(testString, builder.toString());
+ Assert.assertFalse(builder.isTruncated());
+ }
+
+ @Test
+ public void testMultipleAppendsWithinCapacity() {
+ builder.append("Hello").append(" ").append("World");
+ Assert.assertEquals("Hello World", builder.toString());
+ Assert.assertFalse(builder.isTruncated());
+ }
+
+ @Test
+ public void testAppendStringExceedingCapacity() {
+ String fillString = repeat('X', testMaxCapacity - 5);
+ builder.append(fillString);
+
+ String exceedString = "123456";
+ builder.append(exceedString);
+
+ // Should be truncated to exactly max capacity
+ Assert.assertEquals(testMaxCapacity - 16, builder.length());
+ Assert.assertTrue(builder.isTruncated());
+ Assert.assertTrue(builder.toString().endsWith("...[TRUNCATED]"));
+ }
+
+ @Test
+ public void testAppendObject() {
+ Object testObj = new Object() {
+ @Override
+ public String toString() {
+ return "TestObject";
+ }
+ };
+ builder.append(testObj);
+ Assert.assertEquals("TestObject", builder.toString());
+ }
+
+ @Test
+ public void testLength() {
+ Assert.assertEquals(0, builder.length());
+ builder.append("123");
+ Assert.assertEquals(3, builder.length());
+ }
+
+ @Test
+ public void testToStringNotTruncated() {
+ builder.append("Normal string");
+ Assert.assertEquals("Normal string", builder.toString());
+ }
+
+ @Test
+ public void testToStringTruncated() {
+ // Force truncation
+ builder.append(repeat('X', testMaxCapacity - 5));
+ Assert.assertTrue(builder.toString().endsWith("...[TRUNCATED]"));
+ }
+
+ @Test
+ public void testAppendAfterTruncation() {
+ // First append that causes truncation
+ builder.append(repeat('X', testMaxCapacity + 1));
+ Assert.assertTrue(builder.isTruncated());
+
+ // Subsequent append should be ignored
+ builder.append("This should not appear");
+ Assert.assertTrue(builder.toString().endsWith("...[TRUNCATED]"));
+ Assert.assertFalse(builder.toString().contains("This should not
appear"));
+ }
+
+ @Test
+ public void testExactCapacity() {
+ String exactString = repeat('X', testMaxCapacity - 16);
+ builder.append(exactString);
+ Assert.assertEquals(exactString, builder.toString());
+ Assert.assertFalse(builder.isTruncated());
+ }
+
+ private String repeat(char c, int count) {
+ char[] chars = new char[count];
+ for (int i = 0; i < count; i++) {
+ chars[i] = c;
+ }
+ return new String(chars);
+ }
+}
diff --git a/regression-test/suites/query_profile/profile_size_limit.groovy
b/regression-test/suites/query_profile/profile_size_limit.groovy
new file mode 100644
index 00000000000..f743c278616
--- /dev/null
+++ b/regression-test/suites/query_profile/profile_size_limit.groovy
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+import groovy.json.JsonSlurper
+import org.apache.doris.regression.util.DebugPoint
+import org.apache.doris.regression.util.NodeType
+
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import groovy.json.StringEscapeUtils
+
+final String PROFILE_SIZE_NOT_GREATER_THAN_ZERO_MSG = "Profile size is not
greater than 0"
+final String PROFILE_SIZE_GREATER_THAN_LIMIT_MSG = "Profile size is greater
than limit"
+
+def getProfileList = {
+ def dst = 'http://' + context.config.feHttpAddress
+ def conn = new URL(dst + "/rest/v1/query_profile").openConnection()
+ conn.setRequestMethod("GET")
+ def encoding =
Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
+ (context.config.feHttpPassword == null ? "" :
context.config.feHttpPassword)).getBytes("UTF-8"))
+ conn.setRequestProperty("Authorization", "Basic ${encoding}")
+ return conn.getInputStream().getText()
+}
+
+def getProfile = { id ->
+ def dst = 'http://' + context.config.feHttpAddress
+ def conn = new URL(dst +
"/api/profile/text/?query_id=$id").openConnection()
+ conn.setRequestMethod("GET")
+ def encoding =
Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
+ (context.config.feHttpPassword == null ? "" :
context.config.feHttpPassword)).getBytes("UTF-8"))
+ conn.setRequestProperty("Authorization", "Basic ${encoding}")
+ return conn.getInputStream().getText()
+}
+
+def getProfileWithToken = { token ->
+ def wholeString = getProfileList()
+ List profileData = new JsonSlurper().parseText(wholeString).data.rows
+ String profileId = "";
+ logger.info("{}", token)
+
+ for (def profileItem in profileData) {
+ if (profileItem["Sql Statement"].toString().contains(token)) {
+ profileId = profileItem["Profile ID"].toString()
+ logger.info("profileItem: {}", profileItem)
+ }
+ }
+
+ logger.info("$token: {}", profileId)
+ // Sleep 2 seconds to make sure profile collection is done
+ Thread.sleep(2000)
+
+ def String profile = getProfile(profileId).toString()
+ return profile;
+}
+
+suite('profile_size_limit', 'docker, nonConcurrent') {
+ def options = new ClusterOptions()
+ options.beNum = 1
+ options.enableDebugPoints()
+
+ docker(options) {
+ sql "set enable_profile=true;"
+ sql "set profile_level=2;"
+
+ sql """
+ DROP TABLE IF EXISTS profile_size_limit;
+ """
+ sql """
+ CREATE TABLE if not exists `profile_size_limit` (
+ `id` INT,
+ `name` varchar(64),
+ `age` INT,
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`id`) BUCKETS 10
+ PROPERTIES (
+ "replication_num"="1"
+ );
+ """
+
+ sql """
+ INSERT INTO profile_size_limit VALUES
+ (1, "Senior_Software_Engineer_Backend", 25),
+ (2, "Data_Scientist_Machine_Learning", 30),
+ (3, "Cloud_Architect_AWS_Certified", 28),
+ (4, "DevOps_Engineer_Kubernetes_Expert", 35),
+ (5, "Frontend_Developer_React_Specialist", 40),
+ (6, "Database_Administrator_MySQL", 32),
+ (7, "Security_Analyst_Penetration_Testing", 29),
+ (8, "Mobile_Developer_Flutter_Expert", 27),
+ (9, "AI_Researcher_Computer_Vision", 33),
+ (10, "System_Administrator_Linux", 26),
+ (20, "Network_Engineer_Cisco_Certified", 31),
+ (30, "QA_Automation_Engineer_Selenium", 34),
+ (40, "Business_Intelligence_Analyst", 28),
+ (50, "Technical_Lead_Scrum_Master", 29),
+ (60, "Embedded_Systems_Engineer", 36),
+ (70, "Blockchain_Developer_Ethereum", 30),
+ (80, "Game_Developer_Unity_Expert", 32),
+ (90, "Data_Engineer_Apache_Spark", 38),
+ (101, "Machine_Learning_Engineer_NLP", 27),
+ (201, "Cloud_Native_Developer_Docker", 33),
+ (301, "Cybersecurity_Specialist_SOC", 35),
+ (401, "Full_Stack_Developer_NodeJS", 29),
+ (501, "IoT_Engineer_Embedded_Systems", 31),
+ (601, "Site_Reliability_Engineer_SRE", 34),
+ (701, "Product_Manager_Technical", 28),
+ (801, "UX_Designer_Interaction_Design", 36),
+ (901, "Research_Scientist_AI", 30),
+ (1010, "Infrastructure_Engineer_Terraform", 32),
+ (2010, "Big_Data_Engineer_Hadoop", 37),
+ (3010, "Software_Architect_Enterprise", 39),
+ (4010, "Database_Developer_PostgreSQL", 33),
+ (5010, "Mobile_Architect_iOS_Android", 35),
+ (6010, "Security_Engineer_Cryptography", 40),
+ (7010, "Data_Analyst_Business_Intelligence", 31),
+ (8010, "Platform_Engineer_Cloud", 29),
+ (9010, "CTO_Technical_Strategy_Lead", 42);
+ """
+
+ def feHttpAddress = context.config.feHttpAddress.split(":")
+ def feHost = feHttpAddress[0]
+ def fePort = feHttpAddress[1] as int
+
+ sql """
+ select "${token}", * from profile_size_limit;
+ """
+ def String profile = getProfileWithToken(token)
+ logger.info("Profile of ${token} size: ${profile.size()}")
+ assertTrue(profile.size() > 0, PROFILE_SIZE_NOT_GREATER_THAN_ZERO_MSG)
+
+ int maxProfileSize = profile.size()
+
+ while (maxProfileSize >= 64) {
+ maxProfileSize /= 2;
+
+ DebugPoint.enableDebugPoint(feHost, fePort, NodeType.FE,
"Profile.profileSizeLimit",
+ ["profileSizeLimit": maxProfileSize.toString()])
+ token = UUID.randomUUID().toString()
+ sql """
+ select "${token}", * from profile_size_limit;
+ """
+ profile = getProfileWithToken(token)
+ logger.info("Profile of ${token} size: ${profile.size()}, limit:
${maxProfileSize}")
+ assertTrue(profile.size() <= maxProfileSize,
PROFILE_SIZE_GREATER_THAN_LIMIT_MSG)
+ }
+
+ DebugPoint.disableDebugPoint(feHost, fePort, NodeType.FE,
"Profile.profileSizeLimit")
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
