morningman commented on a change in pull request #7371:
URL: https://github.com/apache/incubator-doris/pull/7371#discussion_r769236029
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -582,14 +580,19 @@ private Catalog(boolean isCheckpointCatalog) {
this.tabletScheduler = new TabletScheduler(this, systemInfo,
tabletInvertedIndex, stat, Config.tablet_rebalancer_type);
this.tabletChecker = new TabletChecker(this, systemInfo,
tabletScheduler, stat);
+ // The pendingLoadTaskScheduler's queue size should not less than
Config.desired_max_waiting_jobs.
+ // So that we can guarantee that all submitted load jobs can be
scheduled without being starved.
this.pendingLoadTaskScheduler = new
MasterTaskExecutor("pending_load_task_scheduler",
Config.async_pending_load_task_pool_size,
- Config.async_pending_load_task_pool_size,
!isCheckpointCatalog);
+ Config.desired_max_waiting_jobs, !isCheckpointCatalog);
+ // The loadingLoadTaskScheduler's queue size is unlimited, so that it
can receive all loading tasks
+ // created after pending tasks finish. And don't worry about the high
concurrency, because the
+ // concurrency is limited by Config.desired_max_waiting_jobs and
Config.async_loading_load_task_pool_size.
this.loadingLoadTaskScheduler = new
MasterTaskExecutor("loading_load_task_scheduler",
Config.async_loading_load_task_pool_size,
- Config.async_loading_load_task_pool_size / 5,
!isCheckpointCatalog);
+ Integer.MAX_VALUE, !isCheckpointCatalog);
+
this.loadJobScheduler = new LoadJobScheduler();
this.loadManager = new LoadManager(loadJobScheduler);
this.streamLoadRecordMgr = new
StreamLoadRecordMgr("stream_load_record_manager",
Config.fetch_stream_load_record_interval_second * 1000);
- this.loadTimeoutChecker = new LoadTimeoutChecker(loadManager);
Review comment:
This checker is no logger needed. Because now the timeout is only
calculated when the task is actually executed.
So for now, if a job is in pending state, it will never be cleared.
And the length limit of the pending queue will ensure that there will not be
too many pending jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
