Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2832

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16591) RaftClusterInvocationContext ought to clear the admin queue when stopping

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16591.

Fix Version/s: 3.8.0
   Resolution: Fixed

> RaftClusterInvocationContext ought to clear the admin queue when stopping
> -
>
> Key: KAFKA-16591
> URL: https://issues.apache.org/jira/browse/KAFKA-16591
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
>  Labels: newbie
> Fix For: 3.8.0
>
>
> [https://github.com/apache/kafka/blob/0308543d596d564b68dcac371a8e3af69e06caea/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L265]
> Currently it does not cause issue since `Admin` can be closed repeatedly. 
> However, clearing all admin is always the right way after we close all of 
> them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2831

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2830

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16593) Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16593:
--

 Summary: Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions 
 Key: KAFKA-16593
 URL: https://issues.apache.org/jira/browse/KAFKA-16593
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


as title. the test is in tools module and it does not need to depend on test 
code of core module directly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16552) Create an internal config to control InitialTaskDelayMs in LogManager to speed up tests

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16552.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Create an internal config to control InitialTaskDelayMs in LogManager to 
> speed up tests
> ---
>
> Key: KAFKA-16552
> URL: https://issues.apache.org/jira/browse/KAFKA-16552
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> When startup LogManager, we'll create schedule tasks like: 
> kafka-log-retention, kafka-recovery-point-checkpoint threads...etc 
> ([here|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L629]).
>  All of them have public configs to configure the interval, like 
> `log.retention.check.interval.ms`. But in addition to the scheduler interval, 
> there's a hard coded InitialTaskDelayMs (30 seconds) for all of them. That 
> might not be a problem in production env, since it'll make the kafka server 
> start up faster. But in test env, the 30 secs delay means if there are tests 
> verifying the behaviors like log retention, it'll take 30 secs up to complete 
> the tests.
> To speed up tests, we should create an internal config (ex: 
> "log.initial.task.delay.ms") to control InitialTaskDelayMs in LogManager to 
> speed up tests. This is not intended to be used by normal users, just for 
> speeding up testing usage.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16592) ConfigKey constructor update can break clients using it

[Sagar Rao (Jira)]

Sagar Rao created KAFKA-16592:
-

 Summary: ConfigKey constructor update can break clients using it
 Key: KAFKA-16592
 URL: https://issues.apache.org/jira/browse/KAFKA-16592
 Project: Kafka
  Issue Type: Bug
Reporter: Sagar Rao
Assignee: Sagar Rao


In [KAFKA-14957|https://issues.apache.org/jira/browse/KAFKA-14957], the 
constructor of ConfigDef.ConfigKey was updated to add a new argument called 
{*}alternativeString{*}. As part of the PR, new *define* methods were also 
added which makes sense. However, since the constructor of 
*ConfigDef.ConfigKey* itself can be used directly by other clients which import 
the dependency, this can break all clients who were using the older constructor 
w/o the *alternativeString* argument. 

I bumped into this when I was testing 
the[kafka-connect-redis|[https://github.com/jcustenborder/kafka-connect-redis/tree/master]]
 connector. It starts up correctly against the official 3.7 release, but fails 
with the following error when run against a 3.8 snapshot

 

 
{code:java}
Caused by: java.lang.NoSuchMethodError: 
org.apache.kafka.common.config.ConfigDef$ConfigKey.(Ljava/lang/String;Lorg/apache/kafka/common/config/ConfigDef$Type;Ljava/lang/Object;Lorg/apache/kafka/common/config/ConfigDef$Validator;Lorg/apache/kafka/common/config/ConfigDef$Importance;Ljava/lang/String;Ljava/lang/String;ILorg/apache/kafka/common/config/ConfigDef$Width;Ljava/lang/String;Ljava/util/List;Lorg/apache/kafka/common/config/ConfigDef$Recommender;Z)V
 at 
com.github.jcustenborder.kafka.connect.utils.config.ConfigKeyBuilder.build(ConfigKeyBuilder.java:62)
 at 
com.github.jcustenborder.kafka.connect.redis.RedisConnectorConfig.config(RedisConnectorConfig.java:133)
 at 
com.github.jcustenborder.kafka.connect.redis.RedisSinkConnectorConfig.config(RedisSinkConnectorConfig.java:46)
 at 
com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector.config(RedisSinkConnector.java:73)
 at 
org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:538)
 at 
org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$3(AbstractHerder.java:412)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 ... 1 more
 
{code}
 

The reason for that is that the connector uses another library called 
connect-utils which invokes the old constructor 
[directly|https://github.com/jcustenborder/connect-utils/blob/master/connect-utils/src/main/java/com/github/jcustenborder/kafka/connect/utils/config/ConfigKeyBuilder.java#L62]

It is not expected for connector invocations to fail across versions so this 
would cause confusion.

We could argue that why is the constructor being invoked directly instead of 
using the *define* method, but there might be other clients doing the same. We 
should add the old constructor back which calls the new one by setting the 
*alternativeString* to null.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16591) RaftClusterInvocationContext ought to clear the admin queue when stopping

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16591:
--

 Summary: RaftClusterInvocationContext ought to clear the admin 
queue when stopping
 Key: KAFKA-16591
 URL: https://issues.apache.org/jira/browse/KAFKA-16591
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


[https://github.com/apache/kafka/blob/0308543d596d564b68dcac371a8e3af69e06caea/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L265]

Currently it does not cause issue since `Admin` can be closed repeatedly. 
However, clearing all admin is always the right way after we close all of them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

[Igor Soarez]

Hi Omnia,

Thanks for this KIP.

11. These seem to me to be small misspellings, please double-check:
s/MM2 main features/MM2's main features
s/syncing consumer group offset/syncing consumer group offsets
s/relays/relies
s/recored's offset/recorded offsets
s/clusters without need for/clusters without the need for
s/creating internal topic./creating an internal topic.
s/This KIP propose that/This KIP proposes that

12. The use of the word "customers" seems a bit odd to me in this context.
Did you perhaps mean one of "use-cases", "users" or "operators"?

13. "They still left with part#1 of this feature which add cost to
the progress of their replication."
I'm unsure what this means. Do you mean to say that
MirrorCheckpointConnector is disabled but MirrorSourceConnector is not?
Could you also clarify where the additional cost comes from?

14. This is probably more ignorance of mine: it doesn't seem obvious in
the KIP how increasing offset.lag.max  to INT_MAX helps reduce latency.
I'm guessing it's related to KAFKA-14610 but after having a look I
still couldn't understand why.


--
Igor

On Wed, Apr 17, 2024, at 3:22 PM, Omnia Ibrahim wrote:
> Thanks Chris and Mickael for the votes. 
> Can I please get one last +1 binding vote please?
> 
> Thanks
> Omnia
> 
> > On 12 Apr 2024, at 13:21, Chris Egerton  wrote:
> > 
> > +1 (binding), thanks Omnia!
> > 
> > On Fri, Apr 12, 2024, 03:46 Mickael Maison  wrote:
> > 
> >> Hi Omnia,
> >> 
> >> +1 (binding), thanks for the KIP!
> >> 
> >> Mickael
> >> 
> >> On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim 
> >> wrote:
> >>> 
> >>> Hi everyone, I would like to start a voting thread for KIP-1031: Control
> >> offset translation in MirrorSourceConnector
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector
> >>> 
> >>> For comments or feedback please check the discussion thread here
> >> https://lists.apache.org/thread/ym6zr0wrhglft5c000x9c8ych098s7h6
> >>> 
> >>> Thanks
> >>> Omnia
> >>> 
> >> 
> 
> 

[jira] [Created] (KAFKA-16590) Current offset is higher than last offset

[MAHDI JAFARPUR (Jira)]

MAHDI JAFARPUR created KAFKA-16590:
--

 Summary: Current offset is higher than last offset
 Key: KAFKA-16590
 URL: https://issues.apache.org/jira/browse/KAFKA-16590
 Project: Kafka
  Issue Type: Bug
Reporter: MAHDI JAFARPUR


After a network incident(without any PVC issue or data loss at disk level) in 
our data center, we see that the current offset(consumer group offset) of some 
partitions in some topics is higher than the last offset(last record offset in 
the partition) of that partition.
I read relevant Kafka documents about this issue, then I noticed that this 
issue never should happen because when you try to fetch or commit an offset 
that doesn't exist(higher or lower than Kafka offset range), Kafka returns an 
OffsetOutOfRangeException
also when there is no committed position in the consumer group Kafka will use 
{{auto.offset.reset }} configuration. I inquired about this issue with the 
Strimzi community, and they confirmed that it is not caused by Strimzi.
 
 
Questions:
 * Can this issue be from our consumer/producer library(Golang/Sarama)?
 * Can this issue be from the Kafka bug or probmel?

h3. Strimzi version

0.33.2
h3. Kubernetes version

K8s: v1.20.2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)