acheng-floyd opened a new issue, #32573: URL: https://github.com/apache/doris/issues/32573
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.1.0 ### What's Wrong? Upgrade to doris 2.1.0 from 2.0.5, Memory using just keep going until all memory has been used I create 3 routine load to load nginx log from kafka, it works fine on 2.0.5 version `nginx_log` this table has about 10G data (displayed in partion) `bfq_nginx_log` this table has about 70G data (displayed in partion) in 2.0.5 everything goes well, but in 2.1.0 the memory just keep growing,, after i reboot be node <img width="858" alt="image" src="https://github.com/apache/doris/assets/62228888/bd7265be-668a-4551-86de-fa7bb4ba933f";> <img width="1696" alt="image" src="https://github.com/apache/doris/assets/62228888/77e1393b-e0c2-4d30-a6da-9ffef0ec88c1";> here is the mem_tracker info of 4 nodes: <img width="1674" alt="image" src="https://github.com/apache/doris/assets/62228888/1c52d672-922b-45da-b3f8-a067c13934f7";> <img width="1686" alt="image" src="https://github.com/apache/doris/assets/62228888/c740ce1e-ce31-4ce2-875c-bea2d2dbcf42";> <img width="1685" alt="image" src="https://github.com/apache/doris/assets/62228888/27e34c9d-5f5b-407f-b0f1-f3a6d57c2250";> <img width="1672" alt="image" src="https://github.com/apache/doris/assets/62228888/2aa88e29-ac77-496e-aa92-a764b710f57b";> ``` CREATE ROUTINE LOAD logdb.nginx_log_load_job ON nginx_log COLUMNS TERMINATED BY ",", WHERE timestamp IS NOT NULL PROPERTIES ( "desired_concurrent_number" = "3", /* 根据实际情况调整并发数 */ "max_batch_interval" = "15", /* 最大批处理时间，单位秒 */ "max_batch_rows" = "1000000", /* 单批最大行数 */ "max_batch_size" = "1000000000", /* 单批最大数据量，单位字节 */ "strict_mode" = "false", /* 严格模式，如果为true，则数据格式不正确的行会导致任务失败 */ "format" = "json" /* 假设Kafka中的消息是JSON格式，如果不是，需要相应地调整 */ ) FROM KAFKA ( "kafka_broker_list" = "10.4.99.103:9092,10.4.99.104:9092,10.4.99.105:9092", "kafka_topic" = "nginxlogs", "property.group.id" = "doris_nginx_logs_group", /* Kafka消费者组ID，自定义以避免冲突 */ "property.client.id" = "doris_nginx_logs_client", /* Kafka客户端ID，自定义 */ "property.kafka_default_offsets" = "OFFSET_END" /* 从头开始消费Kafka消息 */ ); CREATE ROUTINE LOAD logdb.bfq_nginx_log_load_job ON bfq_nginx_log COLUMNS TERMINATED BY ",", WHERE timestamp IS NOT NULL PROPERTIES ( "desired_concurrent_number" = "3", /* 根据实际情况调整并发数 */ "max_batch_interval" = "15", /* 最大批处理时间，单位秒 */ "max_batch_rows" = "1000000", /* 单批最大行数 */ "max_batch_size" = "1000000000", /* 单批最大数据量，单位字节 */ "strict_mode" = "false", /* 严格模式，如果为true，则数据格式不正确的行会导致任务失败 */ "format" = "json" /* 假设Kafka中的消息是JSON格式，如果不是，需要相应地调整 */ ) FROM KAFKA ( "kafka_broker_list" = "10.4.99.103:9092,10.4.99.104:9092,10.4.99.105:9092", "kafka_topic" = "bfq_nginxlogs", "property.group.id" = "doris_bfq_nginx_logs_group", /* Kafka消费者组ID，自定义以避免冲突 */ "property.client.id" = "doris_bfq_nginx_logs_client", /* Kafka客户端ID，自定义 */ "property.kafka_default_offsets" = "OFFSET_END" /* 从当前开始消费Kafka消息 */ ); CREATE ROUTINE LOAD logdb.bfq_aws_nginx_log_load_job ON bfq_nginx_log COLUMNS TERMINATED BY ",", WHERE timestamp IS NOT NULL PROPERTIES ( "desired_concurrent_number" = "3", /* 根据实际情况调整并发数 */ "max_batch_interval" = "15", /* 最大批处理时间，单位秒 */ "max_batch_rows" = "1000000", /* 单批最大行数 */ "max_batch_size" = "1000000000", /* 单批最大数据量，单位字节 */ "strict_mode" = "false", /* 严格模式，如果为true，则数据格式不正确的行会导致任务失败 */ "format" = "json" /* 假设Kafka中的消息是JSON格式，如果不是，需要相应地调整 */ ) FROM KAFKA ( "kafka_broker_list" = "10.4.99.103:9092,10.4.99.104:9092,10.4.99.105:9092", "kafka_topic" = "bfq_aws_nginxlogs", "property.group.id" = "doris_bfq_aws_nginx_logs_group", /* Kafka消费者组ID，自定义以避免冲突 */ "property.client.id" = "doris_bfq_aws_nginx_logs_client", /* Kafka客户端ID，自定义 */ "property.kafka_default_offsets" = "OFFSET_BEGINNING" /* 从当前开始消费Kafka消息 */ ); ``` ### What You Expected? just as 2.0.5 everything goes well it works fine with 2.0.5 over 1 week ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
