This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6225732fb2b [opt](cache) enhance cache key computation by removing
comments and trimming SQL input (#46099)
6225732fb2b is described below
commit 6225732fb2b834c2584a3400389ca8bc47b2af9e
Author: York Cao <[email protected]>
AuthorDate: Mon Jan 6 18:38:01 2025 +0800
[opt](cache) enhance cache key computation by removing comments and
trimming SQL input (#46099)
- Currently, the SQL cache system in Doris may miss cache hits due to
semantically identical queries being treated as different because of:
- Extra whitespace characters in the SQL query
- SQL comments that don't affect the query execution
- For example, these queries are semantically identical but would
generate different cache keys:
```sql
SELECT * FROM table;
-- Same query with comments and extra spaces
/* Comment */ SELECT * FROM table ;
```
- This PR improves the SQL cache hit rate by:
- Trimming whitespace from SQL queries
- Removing SQL comments before calculating the cache key MD5
- This ensures that queries that are semantically identical but differ
only in whitespace or comments will now hit the same cache entry,
improving cache efficiency and reducing unnecessary query executions
---
fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java
index 29be4af41a7..20f06e59ce9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java
@@ -26,6 +26,7 @@ import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.mysql.FieldInfo;
import org.apache.doris.mysql.privilege.DataMaskPolicy;
import org.apache.doris.mysql.privilege.RowFilterPolicy;
+import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Variable;
import org.apache.doris.nereids.util.Utils;
@@ -349,7 +350,7 @@ public class SqlCacheContext {
/** doComputeCacheKeyMd5 */
public synchronized PUniqueId doComputeCacheKeyMd5(Set<Variable>
usedVariables) {
- StringBuilder cacheKey = new StringBuilder(originSql);
+ StringBuilder cacheKey = new
StringBuilder(NereidsParser.removeCommentAndTrimBlank(originSql.trim()));
for (Entry<FullTableName, String> entry : usedViews.entrySet()) {
cacheKey.append("|")
.append(entry.getKey())
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
