[GitHub] [kafka] fvaleri opened a new pull request, #12656: Update offset.storage.topic description
fvaleri opened a new pull request, #12656: URL: https://github.com/apache/kafka/pull/12656 This is a minor change just to clarify that this topic only stores offsets of source connector tasks. Instead, offsets for sink connector tasks are stored in __consumer_offsets, as for consumer groups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (KAFKA-14238) KRaft replicas can delete segments not included in a snapshot
[
https://issues.apache.org/jira/browse/KAFKA-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-14238.
-
Resolution: Fixed
> KRaft replicas can delete segments not included in a snapshot
> -
>
> Key: KAFKA-14238
> URL: https://issues.apache.org/jira/browse/KAFKA-14238
> Project: Kafka
> Issue Type: Bug
> Components: core, kraft
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Blocker
> Fix For: 3.3.0
>
>
> We see this in the log
> {code:java}
> Deleting segment LogSegment(baseOffset=243864, size=9269150,
> lastModifiedTime=1662486784182, largestRecordTimestamp=Some(1662486784160))
> due to retention time 60480ms breach based on the largest record
> timestamp in the segment {code}
> This then cause {{KafkaRaftClient}} to throw an exception when sending
> batches to the listener:
> {code:java}
> java.lang.IllegalStateException: Snapshot expected since next offset of
> org.apache.kafka.controller.QuorumController$QuorumMetaLogListener@195461949
> is 0, log start offset is 369668 and high-watermark is 547379
> at
> org.apache.kafka.raft.KafkaRaftClient.lambda$updateListenersProgress$4(KafkaRaftClient.java:312)
> at java.base/java.util.Optional.orElseThrow(Optional.java:403)
> at
> org.apache.kafka.raft.KafkaRaftClient.lambda$updateListenersProgress$5(KafkaRaftClient.java:311)
> at java.base/java.util.OptionalLong.ifPresent(OptionalLong.java:165)
> at
> org.apache.kafka.raft.KafkaRaftClient.updateListenersProgress(KafkaRaftClient.java:309){code}
> The on disk state for the cluster metadata partition confirms this:
> {code:java}
> ls __cluster_metadata-0/
> 00369668.index
> 00369668.log
> 00369668.timeindex
> 00503411.index
> 00503411.log
> 00503411.snapshot
> 00503411.timeindex
> 00548746.snapshot
> leader-epoch-checkpoint
> partition.metadata
> quorum-state{code}
> Noticed that there are no {{checkpoint}} files and the log doesn't have a
> segment at base offset 0.
> This is happening because the {{LogConfig}} used for KRaft sets the retention
> policy to {{delete}} which causes the method {{deleteOldSegments}} to delete
> old segments even if there are no snaspshot for it. For KRaft, Kafka should
> only delete segment that breach the log start offset.
> Log configuration for KRaft:
> {code:java}
> val props = new Properties()
> props.put(LogConfig.MaxMessageBytesProp,
> config.maxBatchSizeInBytes.toString)
> props.put(LogConfig.SegmentBytesProp, Int.box(config.logSegmentBytes))
> props.put(LogConfig.SegmentMsProp, Long.box(config.logSegmentMillis))
> props.put(LogConfig.FileDeleteDelayMsProp,
> Int.box(Defaults.FileDeleteDelayMs))
> LogConfig.validateValues(props)
> val defaultLogConfig = LogConfig(props){code}
> Segment deletion code:
> {code:java}
> def deleteOldSegments(): Int = {
> if (config.delete) {
> deleteLogStartOffsetBreachedSegments() +
> deleteRetentionSizeBreachedSegments() +
> deleteRetentionMsBreachedSegments()
> } else {
> deleteLogStartOffsetBreachedSegments()
> }
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] hachikuji commented on pull request #12653: KAFKA-14240; Validate kraft snapshot state on startup
hachikuji commented on PR #12653: URL: https://github.com/apache/kafka/pull/12653#issuecomment-1250074589 Test failures look unrelated. I've triggered one more build to be on the safe side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hachikuji merged pull request #12655: KAFKA-14238; KRaft metadata log should not delete segment past the latest snapshot
hachikuji merged PR #12655: URL: https://github.com/apache/kafka/pull/12655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] mdedetrich commented on pull request #12284: KAFKA-13980: Upgrade from Scala 2.12.15 to 2.12.16
mdedetrich commented on PR #12284: URL: https://github.com/apache/kafka/pull/12284#issuecomment-1250062919 Scala 2.12.17 just came out so I have rebased the PR with the new Scala version. This also means that all of the qualms mentioned in https://github.com/apache/kafka/pull/12284#issuecomment-1158283064 aren't relevant anymore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] agriffaut commented on pull request #12652: KAFKA-14236: ListGroups produce too much Denied logs in authorizer
agriffaut commented on PR #12652: URL: https://github.com/apache/kafka/pull/12652#issuecomment-1250047875 Also, we may use `authHelper.filterByAuthorized` instead of `groups.filter(group => authHelper.authorize( ...` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] vamossagar12 commented on pull request #12561: KAFKA-12495: Exponential backoff retry to prevent rebalance storms when worker joins after revoking rebalance
vamossagar12 commented on PR #12561: URL: https://github.com/apache/kafka/pull/12561#issuecomment-1250045922 > Details I checked in the changes by fixing the 2 unit tests that were failing. This run, there didn't seem to be connect specific failures. My bad on the oversight the last time around. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] agriffaut commented on pull request #12652: KAFKA-14236: ListGroups produce too much Denied logs in authorizer
agriffaut commented on PR #12652: URL: https://github.com/apache/kafka/pull/12652#issuecomment-1250030904 @hachikuji can you review that little change please ? It will save tons of unjustified logs entries -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
