dudufan opened a new issue, #24889: URL: https://github.com/apache/doris/issues/24889
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris 2.0.1 no_avx2 ### What's Wrong? After creating ROUTINE LOAD, it occurs TOO_MANY_TASKS error. I tought this error maybe resolved by others in [4713](https://github.com/apache/doris/issues/4713) [6342](https://github.com/apache/doris/pull/6342) , but I don't know how to resolve it. ```sql CREATE ROUTINE LOAD demo.load_test PROPERTIES( "desired_concurrent_number"="1", "max_batch_interval"="20", "max_batch_rows"="300000", "max_batch_size"="209715200", "strict_mode" = "false", "format" = "json" ) FROM KAFKA( "kafka_broker_list"="10.0.143.4:9092", "kafka_topic"="doris1", "property.group.id"="df-consumer", "property.client.id"="1234511", "property.kafka_default_offsets"="OFFSET_BEGINNING"); ``` then show the load task, msg: 2023-09-26 08:48:15:errCode = 2, detailMessage = failed to send task: errCode = 2, detailMessage = failed to submit task. error code: TOO_MANY_TASKS, msg: (10.0.209.74)[TOO_MANY_TASKS]839b8a214383416e-9302587c0fc837b7_10.0.209.74 related log below: fe.log ```shell 2023-09-25 16:37:38,623 INFO (thrift-server-pool-8|372) [ReportHandler.handleReport():196] receive report from be 14262. type: TASK, current queue size: 1 2023-09-25 16:37:38,623 INFO (Thread-49|99) [ReportHandler.taskReport():549] finished to handle task report from backend 14262, diff task num: 0. cost: 0 ms 2023-09-25 16:37:39,897 INFO (colocate group clone checker|89) [ColocateTableCheckerAndBalancer.matchGroup():309] finished to check tablets. unhealth/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 0 ms 2023-09-25 16:37:41,333 INFO (txnCleaner|83) [DatabaseTransactionMgr.abortTransaction():1325] abort transaction: TransactionState. transaction id: 4446, label: a239dddb3fa04a86-9d42ef8956a200ac, db id: 14266, table id list: 0, callback id: 33793, coordinator: FE: 10.0.209.74, transaction status: ABORTED, error replicas num: 0, replica ids: , prepare time: 1695630996560, commit time: -1, finish time: 1695631061332, reason: timeout by txn manager successfully 2023-09-25 16:37:41,333 INFO (txnCleaner|83) [DatabaseTransactionMgr.removeExpiredAndTimeoutTxns():1839] transaction [4446] is timeout, abort it by transaction manager 2023-09-25 16:37:41,334 INFO (txnCleaner|83) [DatabaseTransactionMgr.abortTransaction():1325] abort transaction: TransactionState. transaction id: 4447, label: 9094910533c546e1-81d0d06a15cc6c4a, db id: 14266, table id list: 0, callback id: 33790, coordinator: FE: 10.0.209.74, transaction status: ABORTED, error replicas num: 0, replica ids: , prepare time: 1695630996562, commit time: -1, finish time: 1695631061333, reason: timeout by txn manager successfully 2023-09-25 16:37:41,334 INFO (txnCleaner|83) [DatabaseTransactionMgr.removeExpiredAndTimeoutTxns():1839] transaction [4447] is timeout, abort it by transaction manager ``` be.INFO ```shell I0925 16:46:36.934396 1010 routine_load_task_executor.cpp:182] too many tasks in thread pool. reject task: 6617fd924e7b484b-9a5084014949055c, job id: 33790, queue size: 0, current tasks num: 10 W0925 16:46:36.934440 1010 backend_service.cpp:253] failed to submit routine load task. job id: 33790 task id: TUniqueId(hi=7356627321156749387, lo=-7327211452672440996) I0925 16:46:41.368535 1010 task_worker_pool.cpp:267] successfully submit task|type=CLEAR_TRANSACTION_TASK|signature=4477|queue_size=1 I0925 16:46:41.368573 1010 task_worker_pool.cpp:267] successfully submit task|type=CLEAR_TRANSACTION_TASK|signature=4478|queue_size=2 I0925 16:46:41.368590 1010 task_worker_pool.cpp:267] successfully submit task|type=CLEAR_TRANSACTION_TASK|signature=4479|queue_size=3 ``` be.waring.log... ```shell W0925 17:38:09.983999 21052 local_file_system.cpp:299] [IO_ERROR]failed to list /proc/21007/fd/: (2), 没有那个文件或目录 W0925 17:38:09.984050 21052 doris_metrics.cpp:379] failed to count fd: [IO_ERROR]failed to list /proc/21007/fd/: (2), 没有那个文件或目录 W0925 17:38:39.179450 17220 backend_service.cpp:253] failed to submit routine load task. job id: 33790 task id: TUniqueId(hi=5910686550301887220, lo=-7511476450436058681) ``` ### What You Expected? ROUTINE LOAD execute normally. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
