[ https://issues.apache.org/jira/browse/KAFKA-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812485#comment-17812485 ]
Apoorv Mittal commented on KAFKA-15772: --------------------------------------- Failure of test: `testAbortTransactionTimeout` in `TransactionsWithTieredStoreTest` class https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-15251/7/tests {code:java} Errororg.apache.kafka.common.errors.TimeoutException: Timeout expired after 3000ms while awaiting InitProducerIdStacktraceorg.apache.kafka.common.errors.TimeoutException: Timeout expired after 3000ms while awaiting InitProducerIdStandard Output[2024-01-30 16:29:01,250] INFO [LocalTieredStorage Id=0] Creating directory: [/tmp/kafka-remote-tier-transactionswithtieredstoretest11967450412731752897/kafka-tiered-storage] (org.apache.kafka.server.log.remote.storage.LocalTieredStorage:289)[2024-01-30 16:29:01,250] INFO [LocalTieredStorage Id=0] Created local tiered storage manager [0]:[kafka-tiered-storage] (org.apache.kafka.server.log.remote.storage.LocalTieredStorage:301)[2024-01-30 16:29:01,251] INFO Started configuring topic-based RLMM with configs: {remote.log.metadata.topic.replication.factor=3, remote.log.metadata.topic.num.partitions=3, remote.log.metadata.common.client.bootstrap.servers=localhost:40061, broker.id=0, remote.log.metadata.initialization.retry.interval.ms=300, remote.log.metadata.common.client.security.protocol=PLAINTEXT, cluster.id=z_bOu1YoRbKNNIThjztsdA, log.dir=/tmp/kafka-6827936654389854503} (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:358)[2024-01-30 16:29:01,251] INFO Successfully configured topic-based RLMM with config: TopicBasedRemoteLogMetadataManagerConfig{clientIdPrefix='__remote_log_metadata_client_0', metadataTopicPartitionsCount=3, consumeWaitMs=120000, metadataTopicRetentionMs=-1, metadataTopicReplicationFactor=3, initializationRetryMaxTimeoutMs=120000, initializationRetryIntervalMs=300, commonProps={security.protocol=PLAINTEXT, bootstrap.servers=localhost:40061}, consumerProps={security.protocol=PLAINTEXT, key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, enable.auto.commit=false, bootstrap.servers=localhost:40061, exclude.internal.topics=false, auto.offset.reset=earliest, client.id=__remote_log_metadata_client_0_consumer}, producerProps={security.protocol=PLAINTEXT, enable.idempotence=true, value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer, acks=all, bootstrap.servers=localhost:40061, key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer, client.id=__remote_log_metadata_client_0_producer}} (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:364)[2024-01-30 16:29:01,252] INFO Initializing topic-based RLMM resources (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:377)[2024-01-30 16:29:01,363] INFO Topic __remote_log_metadata does not exist. Error: This server does not host this topic-partition. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:466)[2024-01-30 16:29:01,366] ERROR Encountered error while creating __remote_log_metadata topic. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:528)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.createTopic(TopicBasedRemoteLogMetadataManager.java:509) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:396) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.[2024-01-30 16:29:01,366] INFO Sleep for 300 ms before it is retried again. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:401)[2024-01-30 16:29:01,538] WARN [LocalTieredStorage Id=1] Remote storage with ID [/tmp/kafka-remote-tier-transactionswithtieredstoretest11967450412731752897] already exists on the file system. Any data already in the remote storage will not be deleted and may result in an inconsistent state and/or provide stale data. (org.apache.kafka.server.log.remote.storage.LocalTieredStorage:295)[2024-01-30 16:29:01,538] INFO [LocalTieredStorage Id=1] Created local tiered storage manager [1]:[kafka-tiered-storage] (org.apache.kafka.server.log.remote.storage.LocalTieredStorage:301)[2024-01-30 16:29:01,538] INFO Started configuring topic-based RLMM with configs: {remote.log.metadata.topic.replication.factor=3, remote.log.metadata.topic.num.partitions=3, remote.log.metadata.common.client.bootstrap.servers=localhost:37601, broker.id=1, remote.log.metadata.initialization.retry.interval.ms=300, remote.log.metadata.common.client.security.protocol=PLAINTEXT, cluster.id=z_bOu1YoRbKNNIThjztsdA, log.dir=/tmp/kafka-17844464900419347760} (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:358)[2024-01-30 16:29:01,539] INFO Successfully configured topic-based RLMM with config: TopicBasedRemoteLogMetadataManagerConfig{clientIdPrefix='__remote_log_metadata_client_1', metadataTopicPartitionsCount=3, consumeWaitMs=120000, metadataTopicRetentionMs=-1, metadataTopicReplicationFactor=3, initializationRetryMaxTimeoutMs=120000, initializationRetryIntervalMs=300, commonProps={security.protocol=PLAINTEXT, bootstrap.servers=localhost:37601}, consumerProps={security.protocol=PLAINTEXT, key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, enable.auto.commit=false, bootstrap.servers=localhost:37601, exclude.internal.topics=false, auto.offset.reset=earliest, client.id=__remote_log_metadata_client_1_consumer}, producerProps={security.protocol=PLAINTEXT, enable.idempotence=true, value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer, acks=all, bootstrap.servers=localhost:37601, key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer, client.id=__remote_log_metadata_client_1_producer}} (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:364)[2024-01-30 16:29:01,539] INFO Initializing topic-based RLMM resources (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:377)[2024-01-30 16:29:01,555] INFO Topic __remote_log_metadata does not exist. Error: This server does not host this topic-partition. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:466)[2024-01-30 16:29:01,557] ERROR Encountered error while creating __remote_log_metadata topic. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:528)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.createTopic(TopicBasedRemoteLogMetadataManager.java:509) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:396) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.[2024-01-30 16:29:01,557] INFO Sleep for 300 ms before it is retried again. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:401)[2024-01-30 16:29:01,668] INFO Topic __remote_log_metadata does not exist. Error: This server does not host this topic-partition. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:466)[2024-01-30 16:29:01,670] ERROR Encountered error while creating __remote_log_metadata topic. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:528)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.createTopic(TopicBasedRemoteLogMetadataManager.java:509) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:396) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.[2024-01-30 16:29:01,670] INFO Sleep for 300 ms before it is retried again. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:401)[2024-01-30 16:29:01,859] INFO Topic __remote_log_metadata does not exist. Error: This server does not host this topic-partition. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:466)[2024-01-30 16:29:01,861] ERROR Encountered error while creating __remote_log_metadata topic. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:528)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.createTopic(TopicBasedRemoteLogMetadataManager.java:509) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:396) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.[2024-01-30 16:29:01,861] INFO Sleep for 300 ms before it is retried again. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:401)[2024-01-30 16:29:01,972] INFO Topic __remote_log_metadata does not exist. Error: This server does not host this topic-partition. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:466)[2024-01-30 16:29:01,974] ERROR Encountered error while creating __remote_log_metadata topic. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:528)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.createTopic(TopicBasedRemoteLogMetadataManager.java:509) at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:396) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.[2024-01-30 16:29:01,974] INFO Sleep for 300 ms before it is retried again. (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager:401) {code} > Flaky test TransactionsWithTieredStoreTest > ------------------------------------------ > > Key: KAFKA-15772 > URL: https://issues.apache.org/jira/browse/KAFKA-15772 > Project: Kafka > Issue Type: Test > Components: Tiered-Storage > Affects Versions: 3.6.0 > Reporter: Divij Vaidya > Priority: Minor > Labels: flaky-test > > Multiple tests in this Test class have been flaky. See: > [https://ge.apache.org/scans/tests?search.rootProjectNames=kafka&search.startTimeMax=1698915424764&search.startTimeMin=1697320800000&search.tags=trunk&search.timeZoneId=Europe%2FBerlin&tests.container=org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest] > > Example failure: > [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14667/9/testReport/junit/org.apache.kafka.tiered.storage.integration/TransactionsWithTieredStoreTest/Build___JDK_11_and_Scala_2_13___testSendOffsetsToTransactionTimeout_String__quorum_kraft/] > > Other stack traces can be obtained from the ge.apache.org link above -- This message was sent by Atlassian Jira (v8.20.10#820010)