Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #25

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13812) Kraft TopicsDelta finishSnapshot maybe loss data

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky resolved KAFKA-13812.
---
Resolution: Not A Bug

> Kraft TopicsDelta finishSnapshot maybe loss data
> 
>
> Key: KAFKA-13812
> URL: https://issues.apache.org/jira/browse/KAFKA-13812
> Project: Kafka
>  Issue Type: Bug
>Reporter: jacky
>Priority: Major
>
> kafka.server.metadata.BrokerMetadataListener.HandleCommitsEvent will restore 
> the image from the snapshot and delete the topic not in the snapshot, but if 
> a topic has been created, and start producing data, when the broker restores 
> the snapshot which does not contain the new topic, it will delete the new 
> topic and the data will lost.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #107

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #849

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.0 #197

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13812) Kraft TopicsDelta deletedTopicIds maybe loss data

[jacky (Jira)]

jacky created KAFKA-13812:
-

 Summary: Kraft TopicsDelta deletedTopicIds maybe loss data
 Key: KAFKA-13812
 URL: https://issues.apache.org/jira/browse/KAFKA-13812
 Project: Kafka
  Issue Type: Bug
Reporter: jacky


kafka.server.metadata.BrokerMetadataListener.HandleCommitsEvent will restore 
the image from the snapshot and delete the topic not in the snapshot, but if a 
topic has been created, and start producing data, when the broker restores the 
snapshot which does not contain the new topic, it will delete the new topic and 
the data will lost.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #24

[Apache Jenkins Server]

See 

Re: Requesting permission for JIRA/Wiki

[Divij Vaidya]

Correction: I would like to open a PR for
https://issues.apache.org/jira/browse/KAFKA-12319 and not
https://issues.apache.org/jira/browse/KAFKA-13530 as stated earlier.

Divij Vaidya



On Sat, Apr 9, 2022 at 12:19 AM Divij Vaidya 
wrote:

> I would like to open a PR for
> https://issues.apache.org/jira/browse/KAFKA-13530
>
> Please give me permission to:
> 1. create and assign JIRA items
> 2. create new cwiki
>
> JIRA username: divijvaidya
> cwiki username: divijvaidya
>
> Thank you.
>
> Regards,
> Divij Vaidya
>
>

Requesting permission for JIRA/Wiki

[Divij Vaidya]

I would like to open a PR for
https://issues.apache.org/jira/browse/KAFKA-13530

Please give me permission to:
1. create and assign JIRA items
2. create new cwiki

JIRA username: divijvaidya
cwiki username: divijvaidya

Thank you.

Regards,
Divij Vaidya

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #848

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13811) Investigate sliding windows performance

[Leah Thomas (Jira)]

Leah Thomas created KAFKA-13811:
---

 Summary: Investigate sliding windows performance
 Key: KAFKA-13811
 URL: https://issues.apache.org/jira/browse/KAFKA-13811
 Project: Kafka
  Issue Type: Task
  Components: streams
Reporter: Leah Thomas


We recently fixed a bug in sliding windows so that a grace period of 0ms is 
properly calculated, see https://issues.apache.org/jira/browse/KAFKA-13739. 
Before this patch, sliding windows with a grace period of 0ms would just skip 
all records so nothing would get put into the store.

When we ran benchmarks for the 3.2 release we saw a significant drop in 
performance for sliding windows on both the 3.2 and trunk branches, see the 
`sliding windows` results 
[here|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/summaries/process-cumulative-rate/graph.html].]
 These benchmarks use a sliding window with a 0ms grace period, which means 
until now we weren't sending any values to the state store when running these 
benchmarks.

I ran benchmarks on the 
[commit|https://github.com/apache/kafka/commit/430f9c99012d1585aa544d4dadf449963296c1fd]
 before KAFKA-13739 and changed the grace period to 2 seconds to see if the bug 
fix changed anything. The performance was still low for [this 
run|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/experiments/sliding1min-3-5-3-3-0-430f9c9901-leah-20220408084241-streamsbench/].]

Given this, it seems like the performance for sliding windows has always been 
low but we didn't realize it because the bug fixed in KAFKA-13739 was present 
in the benchmarks we were running. We should investigate why the current 
algorithm is slow and see if improvements can be made



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #106

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #23

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #847

[Apache Jenkins Server]

See 

Re: Requesting permissions to contribute to Apache Kafka.

[Mickael Maison]

Hi,

Welcome to the Kafka community!
I've granted you permissions on the wiki and jira.

Thanks,
Mickael


On Fri, Apr 8, 2022 at 1:58 PM Evans Jahja  wrote:
>
> Hi, my name is Evans Jahja.
>
> I'd like to contribute to Kafka. Please kindly add me.
>
> wiki: evans.jahja
> jira: evans.jahja

[VOTE] 3.1.1 RC0

[Tom Bentley]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.1.1.

Apache Kafka 3.1.1 is a bugfix release and 29 issues have been fixed
since 3.1.0.

Release notes for the 3.1.1 release:
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday 15 April, 12:00 UTC

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/

* Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
https://github.com/apache/kafka/releases/tag/3.1.1-rc0

* Documentation:
https://kafka.apache.org/31/documentation.html

* Protocol:
https://kafka.apache.org/31/protocol.html

* Successful Jenkins builds for the 3.1 branch:
I will share a link one the build is complete.

/**

Thanks,

Tom

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Divij Vaidya]

Thanks for replying. I still have a few lingering questions/comments.

*Reg#1* Understood. I checked and the underlying system call is statvfs for
unix systems which should be ok to call here.
*Reg#2* Fair point. I checked again and yes, log.dir always means local
storage even when tiered storage is enabled.
*Reg#3* The rationale for adding these new (size) fields to the
`DescribeLogDirs` is to allow the administrator to monitor or perhaps take
automated action based on results. Doesn't monitoring the number of file
descriptors fall in the same category of use cases? I am assuming that we
want to add the size information in the API response because JVM makes it
possible to get this information in a platform agnostic manner which is not
true for open file descriptors, correct?
*Reg#4* Agree.
*New#5*: As an FYI, Java FileStore API breaks on large storage sizes. See:
https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been
hit by these limitations in the past. For JDK 11, you will probably have to
add defensive checks such as
https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148.
The documentation of the API mentioned in KIP will also be modified to
account for this edge case.
*New#6*: Can you please provide an example where the return for these APIs
would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will definitely
return results (else it throws an IOException)? I would propose that we get
rid of default since JVM guarantees that this would work on all platforms.
If it doesn't then it's a bug and should be uncovered via an exception.

Also, I would like to volunteer to code review (of course, it would be
non-binding) your implementation once this KIP is approved.

Regards,
Divij Vaidya

On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison 
wrote:

> Hi Divij,
>
> Thanks for taking a look!
>
> 1. In order to retrieve the sizes, the plan is to use getTotalSpace()
> and getUsableSpace() from java.nio.file.FileStore. The implementations
> may vary depending on the filesystem but these calls typically don't
> depend on the size of storage but instead just return metadata the
> filesystem maintains.
> 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as
> I understand brokers will still have local log dirs and remote volumes
> are not counted as log dirs. KIP-405 does not mention updating the
> DescribeLogDirs API. So I don't think this KIP needs to do anything
> special to be compatible with KIP-405. On the other hand, I wonder if
> KIP-405 should update DescribeLogDirs to provide details about the
> location of replicas.
> 3. Counting files can be a slow operation as it requires exploring all
> paths recursively to find all files. Administrators should
> definitively monitor file descriptors via host metrics but I'm not
> sure it's something we want to expose via the Kafka API. As mentioned
> it could be slow to compute and files are not really a Kafka concept.
> 4. DescribeLogDirs is usually a low volume API. This change should not
> significantly affect the latency of this API.
>
> Thanks,
> Mickael
>
>
>
>
> On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya 
> wrote:
> >
> > Hi Mickael
> >
> > Thanks for starting this. It is a very useful feature.
> >
> > Some initial thoughts (I am new to Kafka so please excuse if these are
> > naive suggestions):
> > 1. What is the impact on latency of the DescribeLogDirs API due to this
> > change? Would calculating the totalSpace from each logdir be a bottleneck
> > for the API? What if we are talking about a large storage size in the
> order
> > of hundred (or tens) of GBs?
> > 2. How does this fit in with RemoteStorage (KIP-405)? I think integration
> > with KIP-405 is worth discussing in the scope of this KIP. My
> > recommendation will be to add a new API in the RLMM
> > (RemoteLogMetadataManager) called GetLogSize() and leave it upto the
> remote
> > storage to perform a concrete implementation for this
> > interface. DescribeLogDirs could call this interface internally to
> provide
> > the relevant information.
> > 3. Do you think adding the number of files in the directory as part of
> the
> > API response will be useful as well? e.g. a use case where this
> information
> > will be useful is to monitor/alarm the situations when the number of
> files
> > are dangerously reaching the max value of file descriptors configured at
> > the OS.
> > 4. Please add an API latency perf test as part of the release criteria
> for
> > this change. We want to avoid regression.
> >
> > Regards,
> > Divij Vaidya
> >
> >
> >
> > On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison  >
> > wrote:
> >
> > > Hi,
> > >
> > > I wrote a small KIP to expose the total and usable space of logdirs
> > > via the DescribeLogDirs API:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >
> 

[jira] [Resolved] (KAFKA-13809) FileStreamSinkConnector and FileStreamSourceConnector should propagate full configuration to tasks

[Daniel Urban (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Urban resolved KAFKA-13809.
--
Resolution: Won't Fix

Connector configs are used for consumer/producer overrides, converter classes, 
etc.

Not necessary to propagate all configs to tasks.

> FileStreamSinkConnector and FileStreamSourceConnector should propagate full 
> configuration to tasks
> --
>
> Key: KAFKA-13809
> URL: https://issues.apache.org/jira/browse/KAFKA-13809
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Daniel Urban
>Priority: Major
>
> The 2 example connectors do not propagate the full connector configuration to 
> the tasks. This makes it impossible to override built-in configs, such as 
> producer/consumer overrides.
> This causes an issue even when used for testing purposes.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13810) Document behavior of KafkaProducer.flush() w.r.t callbacks

[Karsten Spang (Jira)]

Karsten Spang created KAFKA-13810:
-

 Summary: Document behavior of KafkaProducer.flush() w.r.t callbacks
 Key: KAFKA-13810
 URL: https://issues.apache.org/jira/browse/KAFKA-13810
 Project: Kafka
  Issue Type: Improvement
Reporter: Karsten Spang


The javadoc (3.1.0) for KafkaProducer says
{panel}
The post-condition of {{flush()}} is that any previously sent record will have 
completed (e.g. {{{}Future.isDone() == true{}}}). 
{panel}
It does not say anything about callbacks, though. It is not clear whether all 
callbacks have returned when {{flush()}} returns.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Get Kafka Error Codes on Producer send

[Sumit Chhonker]

Hi,

I am trying to implement Kafka producers using Apache Camel.

I want to know Kafka error codes (defined in Kafka protocol guide: (
https://kafka.apache.org/11/protocol.html#protocol_error_codes )

I also tried by writing Kafka producer in spring boot app,
1. used Producer Interceptor
2. Used Callbacks
3. Used Futures<> callback

But none of them worked for me.

Eg., If I send incorrect topic name, I only get timeout exception after 60s.

Q: How do I get to know the Error Code and Error Name for timeout or any
other error as given in link above?

Thanks in advance.

[jira] [Created] (KAFKA-13809) FileStreamSinkConnector and FileStreamSourceConnector should propagate full configuration to tasks

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-13809:


 Summary: FileStreamSinkConnector and FileStreamSourceConnector 
should propagate full configuration to tasks
 Key: KAFKA-13809
 URL: https://issues.apache.org/jira/browse/KAFKA-13809
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Daniel Urban


The 2 example connectors do not propagate the full connector configuration to 
the tasks. This makes it impossible to override built-in configs, such as 
key.converter.

This causes an issue even when used for testing purposes.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Requesting permissions to contribute to Apache Kafka.

[Evans Jahja]

Hi, my name is Evans Jahja.

I'd like to contribute to Kafka. Please kindly add me.

wiki: evans.jahja
jira: evans.jahja

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Mickael Maison]

Hi Divij,

Thanks for taking a look!

1. In order to retrieve the sizes, the plan is to use getTotalSpace()
and getUsableSpace() from java.nio.file.FileStore. The implementations
may vary depending on the filesystem but these calls typically don't
depend on the size of storage but instead just return metadata the
filesystem maintains.
2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as
I understand brokers will still have local log dirs and remote volumes
are not counted as log dirs. KIP-405 does not mention updating the
DescribeLogDirs API. So I don't think this KIP needs to do anything
special to be compatible with KIP-405. On the other hand, I wonder if
KIP-405 should update DescribeLogDirs to provide details about the
location of replicas.
3. Counting files can be a slow operation as it requires exploring all
paths recursively to find all files. Administrators should
definitively monitor file descriptors via host metrics but I'm not
sure it's something we want to expose via the Kafka API. As mentioned
it could be slow to compute and files are not really a Kafka concept.
4. DescribeLogDirs is usually a low volume API. This change should not
significantly affect the latency of this API.

Thanks,
Mickael




On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya  wrote:
>
> Hi Mickael
>
> Thanks for starting this. It is a very useful feature.
>
> Some initial thoughts (I am new to Kafka so please excuse if these are
> naive suggestions):
> 1. What is the impact on latency of the DescribeLogDirs API due to this
> change? Would calculating the totalSpace from each logdir be a bottleneck
> for the API? What if we are talking about a large storage size in the order
> of hundred (or tens) of GBs?
> 2. How does this fit in with RemoteStorage (KIP-405)? I think integration
> with KIP-405 is worth discussing in the scope of this KIP. My
> recommendation will be to add a new API in the RLMM
> (RemoteLogMetadataManager) called GetLogSize() and leave it upto the remote
> storage to perform a concrete implementation for this
> interface. DescribeLogDirs could call this interface internally to provide
> the relevant information.
> 3. Do you think adding the number of files in the directory as part of the
> API response will be useful as well? e.g. a use case where this information
> will be useful is to monitor/alarm the situations when the number of files
> are dangerously reaching the max value of file descriptors configured at
> the OS.
> 4. Please add an API latency perf test as part of the release criteria for
> this change. We want to avoid regression.
>
> Regards,
> Divij Vaidya
>
>
>
> On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I wrote a small KIP to expose the total and usable space of logdirs
> > via the DescribeLogDirs API:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
> >

[jira] [Created] (KAFKA-13808) Mirrormaker2 stop sync data when modify topic partition in "Running a dedicated MirrorMaker cluster" mode

[YANGLiiN (Jira)]

YANGLiiN created KAFKA-13808:


 Summary: Mirrormaker2  stop sync data when modify topic partition 
in "Running a dedicated MirrorMaker cluster" mode
 Key: KAFKA-13808
 URL: https://issues.apache.org/jira/browse/KAFKA-13808
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect, mirrormaker
Affects Versions: 3.0.1, 3.0.0, 2.8.1, 2.7.2, 2.6.3, 3.1.0
Reporter: YANGLiiN


When use the mirrormaker2 with the "Running a dedicated MirrorMaker cluster" 
mode,  we modify the partition , then the mm2 stop sync data with the follow 
ERROR :

 

[2022-02-18 10:26:19,410] ERROR Error forwarding REST request 
(org.apache.kafka.connect.runtime.rest.RestClient)
java.lang.IllegalArgumentException: Invalid URI host: null (authority: null)
    at org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521)
    at org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506)
    at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464)
    at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453)
    at 
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107)
    at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607)
    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
[2022-02-18 10:26:19,416] ERROR [Worker clientId=connect-1, groupId=sync2022] 
Request to leader to reconfigure connector tasks failed 
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Error trying 
to forward REST request: Invalid URI host: null (authority: null)
    at 
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:147)
    at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607)
    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid URI host: null 
(authority: null)
    at org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521)
    at org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506)
    at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464)
    at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453)
    at 
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107)
    ... 6 more

 

the root cause is the connect doesn't process the mm2 add the 'NOTUSED' url

see 
[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L122]

 

[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1593-L1599]

 

we should add the "NOTUSED" logic  such as 

if (leaderUrl.startsWith("NOTUSED")) {
configBackingStore.putTaskConfigs(connName, rawTaskProps);
cb.onCompletion(null, null);
return;
}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] KIP-828: Add the corresponding validator to the configuration where the validator is missing

[Riven Sun]

Hi devs,  I've created a KIP that aims to add the corresponding validator
to the config where the validator is missing.
In order for the program to detect these incorrect configurations during
initialization:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-828%3A+Add+the+corresponding+validator+to+the+configuration+where+the+validator+is+missing
Please take a look and let me know if you have any feedback. Thanks. Riven
Sun