Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Colin, "Hmm... I'm not sure I follow. Users don't have to build their own tooling, right? They can use any of the shell scripts that we've shipped in the last few releases. For example, if any of your users run it, this shell script will delete all of the topics from your

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote: > Not sure how the AdminClient applies here, It is an external API and > is not part of KafkaProducer so any user who updates to latest version of > Kafka with this feature get to create the topics. > They have to build a tooling around AdminClient

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Justine, Thanks for the clarifications. I understand that auto-creation of topics will happen through CreateTopic request instead of metadata request. What I meant in earlier mail is producer client should not override broker config about auto-creation of topics. I agree with Harsha on other

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

Thank you for the suggestions Jason. And a side note for Guozhang, I updated the KIP to reflect the dependency on 447. On Tue, Aug 6, 2019 at 11:35 AM Jason Gustafson wrote: > Hi Boyang, thanks for the updates. I have a few more comments: > > 1. We are adding some new fields to TxnOffsetCommit

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

Hi Colin, Satish, Stanislav,  Did I answer all your comments/concerns for KIP-491 ?  Please let me know if you have more questions regarding this feature.  I would like to start coding soon. I hope this feature can get into the open source trunk so every time we upgrade Kafka in our

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi, I have so far received pretty much no comments on the technical details outlined in the KIP. While I am happy to continue with my own ideas of how to implement this, I would much prefer to at least get a very broad "looks good in principle, but still lots to flesh out" from a few people

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Ismael, Sure AdminClient can do that and we should've shipped a config or use the existing one to block that. Not all users are yet to upgrade to AdminClient and start using that to cause issues yet. In shared environment we should allow server to set sane defaults and not

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

Hi Alexandre. Thanks for this analysis. IMHO, there are 4 ways to ago about this: 1. We don't fix the bug directly but instead update the Kafka documentation telling clients to configure themselves correctly - Silly but easy to achieve. 2. We adopt Stanislav's solution that fixes the problem -

[jira] [Created] (KAFKA-8759) Message Order is reversed when client run behind a VPN

M. Manna created KAFKA-8759: --- Summary: Message Order is reversed when client run behind a VPN Key: KAFKA-8759 URL: https://issues.apache.org/jira/browse/KAFKA-8759 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Harsha, I mentioned policies and the authorizer. For example, with CreateTopicPolicy, you can implement the limits you describe. If you have ideas of how that should be improved, please submit a KIP. My point is that this KIP is not introducing any new functionality with regards to what rogue

Re: [VOTE] KIP-476: Add Java AdminClient interface

Hi all, Just a quick note to let you all know that the KIP ran into a slight hiccup along the way. The original change saw the return value of `KafkaClientSupplier.getAdminClient` changed from `AdminClient` to the new `Admin`, thereby allowing implementers to return a proxy is they so wanted.

Re: [ANNOUNCE] Apache Kafka 2.3.0

Thanks for pointing out! Should be fixed now. Colin did a corresponding PR against kafka-site repo. Feel free to do a PR directly for fixes like this. -Matthias On 7/31/19 2:53 AM, Mickael Maison wrote: > Hi, > > It looks like the protocol page was not updated. It still only lists 2.2 APIs. >

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

Hi Colin, Thank you for taking a look! I agree, being able to set consumer group offsets via the AdminClient would be really useful, hence I created this KIP. With the total absence of binding votes, I guessed I needed to make some changes. Do you mean you preferred the previous naming

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

Thanks for the KIP. This makes sense to me. Just a couple small comments: 1. Can the listOffsets API be used to get the start and end offsets? In the consumer, we use separate APIs for this: `beginningOffsets` and `endOffsets` to avoid the need for sentinels. An alternative would be to introduce

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi, Even with policies one needs to implement that, so for every user who doesn't want a producer to create topics or have limits around partitions & replication factor they have to implement a policy. The KIP is changing the behavior , it might not be introducing the new functionality

Re: [VOTE] KIP-492 Add java security providers in Kafka Security config

On 2019/07/29 19:22:02, Sandeep Mopuri wrote: > Hi all, after some good discussion > about the > KIP > , > I'm

RE: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

Hey Colin, Can the ListPartitionReassignmentsResult include the status of the current reassignment progress of each partition? A reassignment can be in progress for different reasons and the status can give the option to alter the current reassignment. Example - A leaderISRRequest of a new

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Harsha, After taking this all into consideration, I've updated the KIP to no longer allow client-side configuration of replication factor and partitions. Instead, the broker defaults will be used as long as the broker supports KIP 464. If the broker does not support this KIP, then the client

Re: [VOTE] KIP-492 Add java security providers in Kafka Security config

Hi Jeff, can you make this comment in the PR https://github.com/apache/kafka/pull/7090, instead of the vote thread. Let's move the discussion there. On Tue, Aug 6, 2019 at 10:26 AM Jeff Huang wrote: > > > On 2019/07/29 19:22:02, Sandeep Mopuri wrote: > > Hi all, after some good discussion > >

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

Hi Boyang, thanks for the updates. I have a few more comments: 1. We are adding some new fields to TxnOffsetCommit to support group-based fencing. Do we need these fields to be persisted in the offsets topic to ensure that the fencing still works after a coordinator failover? 2. Since you are

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Justine, Thanks for the KIP. This is a nice addition to the producer client without running admin-client’s create topic APIs. Does producer wait for the topic to be created successfully before it tries to publish messages to that topic? I assume that this will not throw an error that the topic

Build failed in Jenkins: kafka-trunk-jdk11 #735

See Changes: [wangguoz] Minor: Refactor methods to add metrics to sensor in `StreamsMetricsImpl` -- [...truncated 2.59 MB...] org.apache.kafka.streams.StreamsConfigTest

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Satish, Thanks for looking at the KIP. Yes, the producer will wait for the topic to be created before it can send any messages to it. I would like to clarify "overriding" broker behavior. If the client enables client-side autocreation, the only difference will be that the topic auto-creation

[DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hey all, I'd like to kick off discussion on KIP-441, aimed at the long restore times in Streams during which further active processing and IQ are blocked. Please give it a read and let us know your thoughts

[jira] [Created] (KAFKA-8760) KIP-504: Add new Java Authorizer API

Rajini Sivaram created KAFKA-8760: - Summary: KIP-504: Add new Java Authorizer API Key: KAFKA-8760 URL: https://issues.apache.org/jira/browse/KAFKA-8760 Project: Kafka Issue Type: New

Build failed in Jenkins: kafka-trunk-jdk8 #3832

See Changes: [wangguoz] Minor: Refactor methods to add metrics to sensor in `StreamsMetricsImpl` -- [...truncated 2.59 MB...]

[jira] [Created] (KAFKA-8761) Flaky Test AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels

Sophie Blee-Goldman created KAFKA-8761: -- Summary: Flaky Test AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels Key: KAFKA-8761 URL: https://issues.apache.org/jira/browse/KAFKA-8761

Re: [DISCUSS] KIP-481: SerDe Improvements for Connect Decimal type in JSON

Hello Everyone, Summarizing an in-person discussion with Randall (this is copied from the KIP): The original KIP suggested supporting an additional representation - base10 encoded text (e.g. `{"asText":"10.2345"}`). This causes issues because it is impossible to disambiguate between TEXT and

[jira] [Created] (KAFKA-8763) Flaky Test SaslSslAdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels

Sophie Blee-Goldman created KAFKA-8763: -- Summary: Flaky Test SaslSslAdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels Key: KAFKA-8763 URL:

Build failed in Jenkins: kafka-2.2-jdk8 #156

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than -- [...truncated 2.38 MB...] > Task :kafka-2.2-jdk8:streams:processResources

Can we get a 2.3.1 release?

Hi all, 2.3.0 was released June 25, 2019. There are 22 fixes scheduled for inclusion in 2.3.1 so far, 18 of which are resolved/closed. https://issues.apache.org/jira/browse/KAFKA-8736?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.3.1 Do

Build failed in Jenkins: kafka-1.0-jdk7 #275

See Changes: [matthias] KAFKA-8602: Backport bugfix for standby task creation (#7148) -- [...truncated 295.67 KB...] kafka.api.SslEndToEndAuthorizationTest >

[jira] [Created] (KAFKA-8762) Flaky Test AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevelsCanResetLoggerToCurrentRoot

Sophie Blee-Goldman created KAFKA-8762: -- Summary: Flaky Test AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevelsCanResetLoggerToCurrentRoot Key: KAFKA-8762 URL:

[VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

Hello Everyone, After discussions on https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E I've opened this KIP up for formal voting. KIP:

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hello Sophie, Thanks for the proposed KIP. I left some comments on the wiki itself, and I think I'm still not very clear on a couple or those: 1. With this proposal, does that mean with num.standby.replicas == 0, we may sometimes still have some standby tasks which may violate the config? 2. I

[DISCUSS] KIP-504 - Add new Java Authorizer Interface

Hi all, I have created a KIP to replace the Scala Authorizer API with a new Java API: - https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface This is replacement for KIP-50 which was accepted but never merged. Apart from moving to a Java API

Re: [ANNOUNCE] Apache Kafka 2.3.0

Thanks for the bug report, Mickael. As Matthias said, it should be fixed now. I also updated the CVEs page. cheers, Colin On Wed, Jul 31, 2019, at 02:53, Mickael Maison wrote: > Hi, > > It looks like the protocol page was not updated. It still only lists 2.2 APIs. >

Build failed in Jenkins: kafka-2.3-jdk8 #81

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than -- [...truncated 2.95 MB...] kafka.controller.ControllerChannelManagerTest >

Build failed in Jenkins: kafka-2.0-jdk8 #285

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than -- [...truncated 893.65 KB...] kafka.coordinator.group.GroupCoordinatorTest >

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Harsha, Thanks for taking a look. I'm not sure I follow the discussion about AdminClient. KafkaAdminClient has been around for about 2 years at this point as a public class. There are many programs that use it to automatically create topics. Kafka Streams does this, for example. If any

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

Hi Koushik, Thanks for the idea. This KIP is already pretty big, so I think we'll have to consider ideas like this in follow-on KIPs. In general, figuring out what's wrong with replication is a pretty tough problem. If we had an API for this, we'd probably want it to be unified, and not

Build failed in Jenkins: kafka-trunk-jdk8 #3833

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than [gwen] MINOR: some small style fixes to RoundRobinPartitioner --

Build failed in Jenkins: kafka-trunk-jdk11 #736

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than [gwen] MINOR: some small style fixes to RoundRobinPartitioner --

Build failed in Jenkins: kafka-2.1-jdk8 #220

See Changes: [matthias] KAFKA-8736: Streams performance improvement, use isEmpty() rather than -- [...truncated 234.20 KB...] kafka.api.AuthorizerIntegrationTest >

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Colin, There is no behavior in Kafka producer that allowed users to delete the topics or delete the records. So citing them as an example doesn't makes sense in this context. But there is a functionality which allowed creation of topics if they don't exist in the cluster and this

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi Justin, Thanks for making changes. I still have concern that we are prioritizing producer config over server side which is breaking the backward compatibility of broker's auto.topic.create.enable as far as producer is concerned. Also

[PLC4X] [Kafka Connect] Help with Kafka Connect configuration?

Hi, we are currently working on improving the PLC4X Kafka Connect plugin. While the current version offered a pretty simple configuration, this is not quite suitable for production scenarios. In contrast to normal Kafka Connect sources/sinks the PLC4X source and sink should distribute the load

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

Hi Viktor, Thanks for taking over this KP. Current proposed ACL changes allows users to create tokens for any user. Thinking again about this, admins may want to configure a user to impersonate limited number of other users. This allows us to configure fine-grained permissions. But this

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

Hello, I wrote a small change [1] to make clients validate the size of messages received from a broker at the protocol-level. However I don't like the change. It does not really solve the problem which originally motivated the KIP - that is, protocol mismatch (plaintext to SSL endpoint). More

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

This looks like a useful feature, the strategy makes sense, and the KIP is thorough and nicely written. Thanks! Cyrus On Thu, Aug 1, 2019, 12:40 PM Chris Egerton wrote: > Thanks Arjun! Looks good to me. > > On Thu, Aug 1, 2019 at 12:33 PM Arjun Satish > wrote: > > > Thanks for the feedback,

[jira] [Created] (KAFKA-8758) Know your KafkaPrincipal

Stéphane Derosiaux created KAFKA-8758: - Summary: Know your KafkaPrincipal Key: KAFKA-8758 URL: https://issues.apache.org/jira/browse/KAFKA-8758 Project: Kafka Issue Type: Wish

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Hi Guozhang, I left my comments inline. On Thu, Jul 18, 2019 at 8:28 PM Guozhang Wang wrote: > > Hello Bruno, > > Thanks for the feedbacks, replied inline. > > On Mon, Jul 1, 2019 at 7:08 AM Bruno Cadonna wrote: > > > Hi Guozhang, > > > > Thank you for the KIP. > > > > 1) As far as I

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi All, Bumping this, I'd be happy to get some additional feedback and/or votes. Thanks, Viktor On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi All, > > I'd like to start a vote on this KIP. > >

Re: [DISCUSS] KIP-503: deleted topics metric

Thanks for the KIP David, As you mentioned in the KIP - "when a large number of topics (partitions, really) are deleted at once, it can take significant time for the Controller to process everything. In that sense, does it make sense to have the metric expose the number of partitions that are