sollhui opened a new pull request, #57565:
URL: https://github.com/apache/doris/pull/57565
### What problem does this PR solve?
Can not use routine load to consumer data from Kafka 4.1.0 with SASL_SSL :
```
[20251029 15:55:11.956516 3882216 data_consumer.h:102]kafka log-7-BRKMAIN,
event: [thrd:sasl_ssl://10.16.107:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Enter main broker thread
[20251029 15:55:11.956537 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: sasl_ssl://100.16.10.7:19093/bootstrap: Selected for cluster
connectcion: leader query (broker has 0 connection attempt(s))
[20251029 15:55:11.956550 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selectingany broker for cluster connection: still
suppressed ffor 49ms: application metadata request
[20251029 15:55:11.956552 3882216 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:sasl_ssl://10.16.100.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootsstrap: Received CONNECT op
I20251029 15:55:11.956569 3882216 data_consumer.h:102] kafka log-7-STATE,
event: [thrd:sasl_ssl://10.16.10.7:19093/pootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Brokerchanged state INIT -> TRY_CONNECT
[20251029 15:55:11.956579 3882216 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: broker in state TRY_CONNECT connecting
[20251029 15:55:11.956581 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selectingany broker for cluster connection: still
suppressed ffor 49ms: application metadata request
|20251029 15:55:11.956588
3882216 data_consumer.h:102] kafka log-7-STATE, event:
[thrrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.169.10.7:19093/bootstrap: Broker changed state TRY_CONNECT
->CONNECT
[20251029 15:55:11.956596 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selectingany broker for cluster connection: still
suppressed ffor 49ms: application metadata request
I20251029 15:55:11.956745 3882216 data_consumer.h:102] kaflka log-7-CONNECT,
event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Connecting to ipv4#10.16.10.7:19093
I20251029 15:55:11.957020 3882216 data_consumer.h:102] katka log-7-CONNECT,
event: [thrd:sasl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Connected to ipv4#10.16.10.7:19093
(sasl_ssl) with socket 582
[20251029 15:55:11.957029 3882216 data_consumer.h:102]kafka log-7-STATE,
event: [thrd:sasl_ssl://10.16.10.7):19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Broker changed state CONNECT ->
SSL_HANDSHAKE
I20251029 15:55:11.957038 3882212 data_consumer.h:102] kafka log-7-CONNECT,
event: [thrd:app]: Not selecting any brokeer for cluster connection: still
suppressed for 49ms: appliccation metadata request
[20251029 15:55:11.982955 3882216 data_consumer.h:102]kafka log-7-CONNECTED,
event:【thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bodtstrap: Connected (#1)
I20251029 15:55:11.982990 3882216 data_consumer.h:102] kaflka log-7-FEATURE,
event:【thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Updatedenabled protocol features
+ApiVersion to ApiVersion
[20251029 15:55:11.982996 3882216 data_consumer.h:102]kafka log-7-STATE,
event: [thrd:sasl_ssl://10.16.10.7):19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Broker changed state SSL_HANDSHAKE ->
APIVERSION_QUERY
[20251029 15:55:11.983019 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selectingany broker for cluster connection: still
suppressed ffor 23ms: application metadata request
[20251029 15:55:11.984109 3882216 data_consumer.h:102]kafka log-7-FEATURE,
event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Updated enabled protocol features to
ApiVersion, BrokerBalancedConsumer,
sl,SaslHandshake,BrokerGroupCoordinator,LZ4,
OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReg
I20251029 15:55:11.984126 3882216 data_consumer.h:102] kaflka log-7-AUTH,
event: [thrd:sasl_ssl://10.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Auth in state APIVERSION_QUERY
(handshake supported)
I20251029 15:55:11.984167 3882216 data_consumer.h:102] katfka log-7-STATE,
event:
[thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:_sasl_ssl://10.16.10.7:19093/bootstrap:
Brokerchanged state APIVERSION_QUERY -> AUTH_HANDSHAKE
[20251029 15:55:11.984181 3882212 data consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selecting aany broker for cluster connection: still
suppressed for22ms: application metadata request
I20251029 15:55:11.984493 3882216 data_consumer.h:102] kafkka
log-7-SASLMECHS, event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Brokerr supported SASL mechanisms:
SCRAM-SHA-512
I20251029 15:55:11.984505 3882216 data_consumer.h:102] kafka log-7-AUTH,
event: [thrd:sasl_ssl:7/10.16.10.7:19093/boootstrap]:
sasl_ssl:7/10.16.10.7:19093/bootstrap: Auth in stcate AUTH_HANDSHAKE (handshake
supported)
[20251029 15:55:11.984512 3882216 data_consumer.h:102]kafka log-7-STATE,
event:
[thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:_sasl_ssl://10.16.10.7:19093/bootstrap:
Broker changed state AUTH_HANDSHAKE -> AUTH_REQ
I20251029 15:55:11.984524 3882216 data_consumer.h:102] kafka log-7-SASL,
event: [thrd:sasl ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:119093/bootstrap: Initializing SASL client: servicename
kafka, hostname 10.16.10.7, mechanis
s SCRAM-SHA-512, provider SCRAM (builtin)
[20251029 15:55:11.984524 3882212 data_consumer.h:102]kafka log-7-CONNECT,
event: [thrd:app]: Not selectingany broker for cluster connection: still
suppressed ffor 22ms: application metadata request
I20251029 15:55:11.984537 3882216 data_consumer.h:102] kafka
log-7-SASLSCRAM, event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: SASL SCRAM client in state
client-first-message
[20251029 15:55:11.984549 3882216 data_consumer.h:102]kafka log-7-SASL,
event: [thrd:sasl ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl:7/10.16.10.7:19093/bootstraap: Send SASL Kafka frame to broker (45
bytes)
I20251029 15:55:11.986088 3882216 data_consumer.h:102]kafka log-7-SASL,
event: [thrd:sasl ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Received SASL frame from broker (105
bytes)
[20251029 15:55:11.986101 3882216 data consumer.h:102]kafka log-7-SASLSCRAM,
event: [thrd:sasl_ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bodtstrap: SASL SCRAM client in state
server-first-messaggen
I20251029 15:55:11.993464 3882216 data_consumer.h:102]kafka log-7-SASL,
event: [thrd:sasl ssl://10.16.10.7:19093/bootstrap]:
sasl_ssl://10.16.10.7:19093/bootstrap: Send SASL Kafka frame to broker (189
bytes)
I20251029 15:55:12.288036 3249486 tablett manager.cpp:1065] find expired
transactions for 0 tablets
```
Core reason is librdkafka version is low, and this pr bump librdkafka from
1.9.2 to 2.11.0 for Kafka higher version compatibility.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
