This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 7a808893ea [branch1.2][fix](catalog) disable hdfs fs cache to avoid to
many fs cache in memory (#21276)
7a808893ea is described below
commit 7a808893eadf0e0725bf545af07a7d0e5b2f9953
Author: Mingyu Chen <[email protected]>
AuthorDate: Wed Jun 28 15:03:21 2023 +0800
[branch1.2][fix](catalog) disable hdfs fs cache to avoid to many fs cache
in memory (#21276)
---
.../main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
index 8c266168a0..8b7d7f83c6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
@@ -256,6 +256,9 @@ public class HiveMetaStoreCache {
// Otherwise, getSplits() may throw exception: "Not a file xxx"
//
https://blog.actorsfit.com/a?ID=00550-ce56ec63-1bff-4b0c-a6f7-447b93efaa31
jobConf.set("mapreduce.input.fileinputformat.input.dir.recursive",
"true");
+ // FileInputFormat.setInputPaths() will call FileSystem.get(),
which will create new FileSystem
+ // and save it in FileSystem.Cache. We don't need this cache.
+ jobConf.set("fs.hdfs.impl.disable.cache", "true");
FileInputFormat.setInputPaths(jobConf, finalLocation);
try {
InputFormat<?, ?> inputFormat =
HiveUtil.getInputFormat(jobConf, key.inputFormat, false);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
