caoyang10 opened a new issue #5231:
URL: https://github.com/apache/incubator-doris/issues/5231
BE crash with follow conditions:
1, high QPS over 1000
2, kafka routine load
3, cpu usage is above 40%
log:
...
rd_kafka_broker_t 0x75a025b00: 10.62.175.45:9092/75 NodeId 75 in state DOWN
(for 0.000s)
refcnt 0
outbuf_cnt: 0 waitresp_cnt: 0
2 messages sent, 76 bytes, 0 errors, 0 timeouts
2 messages received, 6767 bytes, 0 errors
0 messageset transmissions were retried
0 toppars:
cgrp:
AD_DORIS_ANALYSIS_CLUSTER_doris_ad_dsp_report_20201217173749 in state
term, flags 0x1
coord_id 35, managing broker (none)
toppars:
topics:
simplified_ad_log_report with 0 partitions, state exists, refcnt 1
desired partitions:
Metadata cache with 2586 entries:
cci_kinder_consume_action_log (inserted 385937ms ago, expires in 514062ms,
4 partition(s), valid)
....
*** Aborted at 1609803699 (unix time) try "date -d @1609803699" if you are
using GNU date ***
PC: @ 0x7ff7d63d3387 __GI_raise
*** SIGABRT (@0x45f000131f9) received by PID 78329 (TID 0x7ff45bfc2700) from
PID 78329; stack trace: ***
@ 0x7ff7d63d3400 (unknown)
@ 0x7ff7d63d3387 __GI_raise
@ 0x7ff7d63d4a78 __GI_abort
@ 0x18f63c3 rd_kafka_crash
@ 0x190bece rd_kafka_broker_destroy_final
@ 0x1964bf6 rd_kafka_metadata_refresh_topics
@ 0x1964f2b rd_kafka_metadata_refresh_known_topics
@ 0x19085da rd_kafka_broker_fail
@ 0x19088e7 rd_kafka_broker_conn_closed
@ 0x1908eb1 rd_kafka_recv
@ 0x1921878 rd_kafka_transport_io_event
@ 0x190f7a1 rd_kafka_broker_serve
@ 0x19109bd rd_kafka_broker_thread_main
@ 0x195f337 _thrd_wrapper_function
@ 0x7ff7d6188ea5 start_thread
@ 0x7ff7d649b8dd __clone
I upgrade the version of librdkafka to latest release version:
LIBRDKAFKA_DOWNLOAD="https://github.com/edenhill/librdkafka/archive/v1.5.3.tar.gz";
LIBRDKAFKA_NAME=librdkafka-1.5.3.tar.gz
LIBRDKAFKA_SOURCE=librdkafka-1.5.3
LIBRDKAFKA_MD5SUM="17017d9bdaf1398087d1f0dcad2e5cc7"
It doesn't happen during 5 days
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
