yuqi1129 commented on code in PR #6689:
URL: https://github.com/apache/gravitino/pull/6689#discussion_r1998326489
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -73,18 +77,27 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.awaitility.Awaitility;
-import org.awaitility.core.ConditionTimeoutException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class HadoopCatalogOperations extends ManagedSchemaOperations
implements CatalogOperations, FilesetCatalog {
private static final String SCHEMA_DOES_NOT_EXIST_MSG = "Schema %s does not
exist";
private static final String FILESET_DOES_NOT_EXIST_MSG = "Fileset %s does
not exist";
+ private static final ThreadPoolExecutor FILE_SYSTEM_EXECUTOR =
+ new ThreadPoolExecutor(
+ 2,
+ Math.max(2, Runtime.getRuntime().availableProcessors() / 2),
Review Comment:
Please make the thread number alias with the default value as the jetty
server, see
https://github.com/apache/gravitino/blob/261562538c008b2395079789a6bfbfc625bb39c6/server-common/src/main/java/org/apache/gravitino/server/web/JettyServerConfig.java#L63-L78
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -73,18 +77,27 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.awaitility.Awaitility;
-import org.awaitility.core.ConditionTimeoutException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class HadoopCatalogOperations extends ManagedSchemaOperations
implements CatalogOperations, FilesetCatalog {
private static final String SCHEMA_DOES_NOT_EXIST_MSG = "Schema %s does not
exist";
private static final String FILESET_DOES_NOT_EXIST_MSG = "Fileset %s does
not exist";
+ private static final ThreadPoolExecutor FILE_SYSTEM_EXECUTOR =
+ new ThreadPoolExecutor(
+ 2,
+ Math.max(2, Runtime.getRuntime().availableProcessors() / 2),
+ 60L,
+ TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(500),
+ r -> {
+ Thread thread = new Thread(r, "FileSystem-Get-Thread");
+ thread.setDaemon(true);
+ return thread;
Review Comment:
I think the `callerRun` policy may be better in this case.
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -101,6 +114,10 @@ public class HadoopCatalogOperations extends
ManagedSchemaOperations
private FileSystemProvider defaultFileSystemProvider;
+ static {
Review Comment:
You can replace with this static code block with
xxxx = new ThreadPoolExecutor() {
{
allowCoreThreadTimeOut(true);
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
