lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3336966568
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K>
keySerializer, Seriali
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = apiVersions;
+ List<InetSocketAddress> addresses =
ClientUtils.parseAndValidateAddresses(config);
Review Comment:
uh we're changing this in another PR btw, for KIP-909..this one will go
first I imagine, so we need to align on the other one. cc @frankvicky
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler
addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler
txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition,
OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry :
offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new
CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(),
offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
Review Comment:
Thinking about the case of Unknown_topic_ID made me notice, we have the
metadata check here only (when creating the `txnOffsetCommitHandler`) , but all
the retry logic relies on re-enqueueing the same handler, so this won't be
triggered on retries I expect (would re-send the same topic-id if we get
unknown_topic_id).
Is the intention to handle that when processing the Unknown_topic_id?
(separate PR?)
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler
addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler
txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition,
OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry :
offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new
CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(),
offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
+ var requestTopicsByName = new HashMap<String,
TxnOffsetCommitRequestTopic>();
+ var topicNamesByIds = new HashMap<Uuid, String>();
+ var topics = new ArrayList<TxnOffsetCommitRequestTopic>();
+ var allHaveTopicIds = true;
+ for (var entry : offsets.entrySet()) {
+ var tp = entry.getKey();
+ var offsetAndMetadata = entry.getValue();
+ pendingTxnOffsetCommits.put(
+ tp,
+ new CommittedOffset(
+ offsetAndMetadata.offset(),
+ offsetAndMetadata.metadata(),
+ offsetAndMetadata.leaderEpoch()
+ )
+ );
+ var topicId = topicIds.getOrDefault(tp.topic(), Uuid.ZERO_UUID);
+ allHaveTopicIds &= !topicId.equals(Uuid.ZERO_UUID);
+ var topic = requestTopicsByName.computeIfAbsent(tp.topic(), name
-> {
+ var t = new
TxnOffsetCommitRequestTopic().setName(name).setTopicId(topicId);
+ topics.add(t);
+ if (!topicId.equals(Uuid.ZERO_UUID)) {
+ topicNamesByIds.put(topicId, name);
+ }
+ return t;
+ });
+ topic.partitions().add(new
TxnOffsetCommitRequestData.TxnOffsetCommitRequestPartition()
+ .setPartitionIndex(tp.partition())
+ .setCommittedOffset(offsetAndMetadata.offset())
+
.setCommittedLeaderEpoch(offsetAndMetadata.leaderEpoch().orElse(RecordBatch.NO_PARTITION_LEADER_EPOCH))
+ .setCommittedMetadata(offsetAndMetadata.metadata()));
}
- final TxnOffsetCommitRequestData data = new
TxnOffsetCommitRequestData()
+ var data = new TxnOffsetCommitRequestData()
.setTransactionalId(transactionalId)
.setGroupId(groupMetadata.groupId())
.setProducerId(producerIdAndEpoch.producerId)
.setProducerEpoch(producerIdAndEpoch.epoch)
.setMemberId(groupMetadata.memberId())
.setGenerationIdOrMemberEpoch(groupMetadata.generationId())
.setGroupInstanceId(groupMetadata.groupInstanceId().orElse(null))
-
.setTopics(TxnOffsetCommitRequest.getTopics(pendingTxnOffsetCommits));
- final TxnOffsetCommitRequest.Builder builder =
- TxnOffsetCommitRequest.Builder.forTopicNames(data,
isTransactionV2Enabled());
+ .setTopics(topics);
+ var builder = allHaveTopicIds
+ ? TxnOffsetCommitRequest.Builder.forTopicIdsOrNames(data,
isTransactionV2Enabled(), true)
Review Comment:
this opens the door for the Unknown_topic_id in the response. Is that coming
in a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
