morningman commented on issue #2065: Routine Load may takes too many FE memory URL: https://github.com/apache/incubator-doris/issues/2065#issuecomment-547306633 The reason is that `beIdToBatchTask` in RoutineLoadTaskScheduler is not cleaned. And more and more `TRoutineLoadTask` objects are added to this map. It is not cleaned because the `submitBatchTasksIfNotEmpty()` is not called because the following conditions are not met: ``` if (needScheduleTasksQueue.size() == 0 || idleSlotNum == 0) { submitBatchTasksIfNotEmpty(); } ``` This can happen when we set FE config `max_running_txn_num_per_db` smaller than (num of BE * `max_concurrent_task_num_per_be`). Routine load tasks always failed to begin transaction due to exceed `max_running_txn_num_per_db` limit and be pushed back to the `needScheduleTasksQueue`. So `needScheduleTasksQueue` is always non-empty, but there are idle slots available.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
