Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1232

2022-09-17 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 505846 lines...]
[2022-09-17T16:34:40.559Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:34:40.559Z] 
[2022-09-17T16:34:40.559Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2022-09-17T16:34:52.825Z] 
[2022-09-17T16:34:52.825Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2022-09-17T16:34:52.825Z] 
[2022-09-17T16:34:52.825Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2022-09-17T16:34:52.825Z] 
[2022-09-17T16:34:52.825Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:34:52.825Z] 
[2022-09-17T16:34:52.825Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2022-09-17T16:34:54.555Z] 
[2022-09-17T16:34:54.555Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:34:54.555Z] 
[2022-09-17T16:34:54.555Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-17T16:35:43.726Z] 
[2022-09-17T16:35:43.726Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-17T16:35:43.726Z] 
[2022-09-17T16:35:43.726Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-17T16:36:34.320Z] 
[2022-09-17T16:36:34.320Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-17T16:36:34.320Z] 
[2022-09-17T16:36:34.320Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-17T16:36:41.724Z] 
[2022-09-17T16:36:41.724Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-17T16:36:41.724Z] 
[2022-09-17T16:36:41.724Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-17T16:37:33.706Z] 
[2022-09-17T16:37:33.706Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-09-17T16:37:33.706Z] 
[2022-09-17T16:37:33.706Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-09-17T16:37:33.706Z] 
[2022-09-17T16:37:33.706Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-09-17T16:37:33.706Z] 
[2022-09-17T16:37:33.706Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-09-17T16:37:33.706Z] 
[2022-09-17T16:37:33.706Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-09-17T16:37:34.828Z] 
[2022-09-17T16:37:34.828Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-09-17T16:37:37.488Z] 
[2022-09-17T16:37:37.488Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-09-17T16:37:37.488Z] 
[2022-09-17T16:37:37.488Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-09-17T16:37:44.136Z] 
[2022-09-17T16:37:44.136Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-09-17T16:37:44.136Z] 
[2022-09-17T16:37:44.136Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-09-17T16:37:48.570Z] 
[2022-09-17T16:37:48.570Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-09-17T16:37:48.570Z] 
[2022-09-17T16:37:48.570Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-09-17T16:37:49.715Z] 
[2022-09-17T16:37:49.715Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-09-17T16:37:49.715Z] 
[2022-09-17T16:37:49.715Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-09-17T16:38:10.534Z] 
[2022-09-17T16:38:10.534Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-09-17T16:38:10.534Z] 
[2022-09-17T16:38:10.534Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-09-17T16:38:14.806Z] 
[2022-09-17T16:38:14.806Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-09-17T16:38:14.806Z] 
[2022-09-17T16:38:14.806Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-09-17T16:38:18.867Z] 
[2022-09-17T16:38:18.867Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-09-17T16:38:18.867Z] 
[2022-09-17T16:38:18.867Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-09-17T16:38:23.637Z] 
[2022-09-17T16:38:23.637Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-09-17T16:38:23.637Z] 
[2022-09-17T16:38:23.637Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut()

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #78

2022-09-17 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 572751 lines...]
[2022-09-17T16:09:03.067Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2022-09-17T16:09:03.067Z] 
[2022-09-17T16:09:03.067Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2022-09-17T16:09:03.067Z] 
[2022-09-17T16:09:03.067Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2022-09-17T16:09:03.067Z] 
[2022-09-17T16:09:03.067Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:09:03.067Z] 
[2022-09-17T16:09:03.067Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2022-09-17T16:09:07.927Z] 
[2022-09-17T16:09:07.927Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2022-09-17T16:09:07.927Z] 
[2022-09-17T16:09:07.927Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2022-09-17T16:09:08.882Z] 
[2022-09-17T16:09:08.883Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:09:08.883Z] 
[2022-09-17T16:09:08.883Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2022-09-17T16:09:08.883Z] 
[2022-09-17T16:09:08.883Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2022-09-17T16:09:08.883Z] 
[2022-09-17T16:09:08.883Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2022-09-17T16:09:12.418Z] 
[2022-09-17T16:09:12.418Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = true] PASSED
[2022-09-17T16:09:12.418Z] 
[2022-09-17T16:09:12.418Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = true] STARTED
[2022-09-17T16:09:26.650Z] 
[2022-09-17T16:09:26.650Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = true] PASSED
[2022-09-17T16:09:26.650Z] 
[2022-09-17T16:09:26.650Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = true] STARTED
[2022-09-17T16:09:36.346Z] 
[2022-09-17T16:09:36.346Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2022-09-17T16:09:36.346Z] 
[2022-09-17T16:09:36.346Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-09-17T16:09:43.067Z] 
[2022-09-17T16:09:43.067Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = true] PASSED
[2022-09-17T16:09:43.067Z] 
[2022-09-17T16:09:43.067Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = false] STARTED
[2022-09-17T16:09:57.280Z] 
[2022-09-17T16:09:57.280Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = false] PASSED
[2022-09-17T16:09:57.280Z] 
[2022-09-17T16:09:57.280Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] STARTED
[2022-09-17T16:10:02.622Z] 
[2022-09-17T16:10:02.622Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-09-17T16:10:02.622Z] 
[2022-09-17T16:10:02.622Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-09-17T16:10:09.125Z] 
[2022-09-17T16:10:09.126Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-09-17T16:10:09.126Z] 
[2022-09-17T16:10:09.126Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-09-17T16:10:11.450Z] 
[2022-09-17T16:10:11.450Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] PASSED
[2022-09-17T16:10:11.450Z] 
[2022-09-17T16:10:11.450Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = false] STARTED
[2

[jira] [Resolved] (KAFKA-14238) KRaft replicas can delete segments not included in a snapshot

2022-09-17 Thread Jason Gustafson (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-14238.
-
Resolution: Fixed

> KRaft replicas can delete segments not included in a snapshot
> -
>
> Key: KAFKA-14238
> URL: https://issues.apache.org/jira/browse/KAFKA-14238
> Project: Kafka
>  Issue Type: Bug
>  Components: core, kraft
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Blocker
> Fix For: 3.3.0
>
>
> We see this in the log
> {code:java}
> Deleting segment LogSegment(baseOffset=243864, size=9269150, 
> lastModifiedTime=1662486784182, largestRecordTimestamp=Some(1662486784160)) 
> due to retention time 60480ms breach based on the largest record 
> timestamp in the segment {code}
> This then cause {{KafkaRaftClient}} to throw an exception when sending 
> batches to the listener:
> {code:java}
>  java.lang.IllegalStateException: Snapshot expected since next offset of 
> org.apache.kafka.controller.QuorumController$QuorumMetaLogListener@195461949 
> is 0, log start offset is 369668 and high-watermark is 547379
>   at 
> org.apache.kafka.raft.KafkaRaftClient.lambda$updateListenersProgress$4(KafkaRaftClient.java:312)
>   at java.base/java.util.Optional.orElseThrow(Optional.java:403)
>   at 
> org.apache.kafka.raft.KafkaRaftClient.lambda$updateListenersProgress$5(KafkaRaftClient.java:311)
>   at java.base/java.util.OptionalLong.ifPresent(OptionalLong.java:165)
>   at 
> org.apache.kafka.raft.KafkaRaftClient.updateListenersProgress(KafkaRaftClient.java:309){code}
> The on disk state for the cluster metadata partition confirms this:
> {code:java}
>  ls __cluster_metadata-0/
> 00369668.index
> 00369668.log
> 00369668.timeindex
> 00503411.index
> 00503411.log
> 00503411.snapshot
> 00503411.timeindex
> 00548746.snapshot
> leader-epoch-checkpoint
> partition.metadata
> quorum-state{code}
> Noticed that there are no {{checkpoint}} files and the log doesn't have a 
> segment at base offset 0.
> This is happening because the {{LogConfig}} used for KRaft sets the retention 
> policy to {{delete}} which causes the method {{deleteOldSegments}} to delete 
> old segments even if there are no snaspshot for it. For KRaft, Kafka should 
> only delete segment that breach the log start offset.
> Log configuration for KRaft:
> {code:java}
>   val props = new Properties()
>   props.put(LogConfig.MaxMessageBytesProp, 
> config.maxBatchSizeInBytes.toString)
>   props.put(LogConfig.SegmentBytesProp, Int.box(config.logSegmentBytes))
>   props.put(LogConfig.SegmentMsProp, Long.box(config.logSegmentMillis))
>   props.put(LogConfig.FileDeleteDelayMsProp, 
> Int.box(Defaults.FileDeleteDelayMs))
>   LogConfig.validateValues(props)
>   val defaultLogConfig = LogConfig(props){code}
> Segment deletion code:
> {code:java}
>  def deleteOldSegments(): Int = {
>   if (config.delete) {
> deleteLogStartOffsetBreachedSegments() +
>   deleteRetentionSizeBreachedSegments() +
>   deleteRetentionMsBreachedSegments()
>   } else {
> deleteLogStartOffsetBreachedSegments()
>   }
> }{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)