Re: Kafka SSH Tunnel Connection without editing hostfile

We cannot use external endpoints because of security reasons. Is there an option to tell zookeeper/broker not to send broker host detail metadata to its clients? On Thu, Sep 12, 2019 at 3:05 PM M. Manna wrote: > Have you tried using EXTERNAL endpoints for your Kafka broker to separate > TLS

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

This is exciting! +1 (non-binding) - Dhruvil On Thu, Sep 12, 2019 at 12:29 PM Bill Bejeck wrote: > Thanks for the KIP! > > +1 (binding) > > -Bill > > On Thu, Sep 12, 2019 at 3:26 PM Ismael Juma wrote: > > > Thanks for the KIP, +1 (binding). > > > > Ismael > > > > On Mon, Sep 9, 2019, 8:28 AM

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

Thanks for the KIP, Arjun. It's going to be really nice to be able to set the log levels dynamically, especially through the REST API. However, I think it's not clear what behavior the KIP is actually proposing with respect to which loggers are exposed (just those that are used, or common

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Thanks for updating the KIP. I agree with John that we (ie, you :)) can start a vote. -Matthias On 9/11/19 9:23 AM, John Roesler wrote: > Thanks for the update, Jukka! > > I'd be in favor of the current proposal. Not sure how the others feel. > If people generally feel positive, it might be

[DISCUSS] 2.2.2 Bug Fix Release

Hey everyone, I'd like to volunteer for the release manager of the 2.2.2 bug fix release. Kafka 2.2.1 was released on June 3 and so far 25 issues have been fixed since then. Here is a complete list:

[jira] [Resolved] (KAFKA-8899) Optimize Partition.maybeIncrementLeaderHW

[ https://issues.apache.org/jira/browse/KAFKA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Bradstreet resolved KAFKA-8899. - Resolution: Duplicate Duplicate of https://issues.apache.org/jira/browse/KAFKA-8841 >

[jira] [Created] (KAFKA-8904) Reduce metadata lookups when producting to a large number of topics

Brian Byrne created KAFKA-8904: -- Summary: Reduce metadata lookups when producting to a large number of topics Key: KAFKA-8904 URL: https://issues.apache.org/jira/browse/KAFKA-8904 Project: Kafka

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

Hi Radai, Thanks for the KIP. Sounds interesting. I assume that if an InterruptedException were caught, that would be rethrown, rather than returning false? It might be good to specify that. Can you give an example of how this would be used? best, Colin On Thu, Sep 12, 2019, at 15:26,

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

bump. if no one has any comments on this can we initiate a vote? On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote: > > Hi, > > I would like to start discussion on KIP-514 that proposes we add a > bounded flush() API to producer. > > Link to the KIP: >

Build failed in Jenkins: kafka-2.2-jdk8 #165

See Changes: [rhauch] HOTFIX: fix compile error in 2.2 branch (#7327) -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H31

[jira] [Created] (KAFKA-8903) allow the new replica (offset 0) to catch up with current leader using latest offset

GEORGE LI created KAFKA-8903: Summary: allow the new replica (offset 0) to catch up with current leader using latest offset Key: KAFKA-8903 URL: https://issues.apache.org/jira/browse/KAFKA-8903 Project:

Build failed in Jenkins: kafka-trunk-jdk8 #3902

See Changes: [wangguoz] KAFKA-8355: add static membership to range assignor (#7014) -- [...truncated 2.62 MB...] org.apache.kafka.trogdor.common.WorkerUtilsTest >

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Thanks for the KIP! +1 (binding) -Bill On Thu, Sep 12, 2019 at 3:26 PM Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > Ismael > > On Mon, Sep 9, 2019, 8:28 AM Colin McCabe wrote: > > > Hi all, > > > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > >

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Thanks for the KIP, +1 (binding). Ismael On Mon, Sep 9, 2019, 8:28 AM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > Self-Managed Metadata Quorum. > > The DISCUSS thread from the mailing list is here: > >

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

You are proposing a different design pattern for the SSL reconfigurable extension point than the one already implemented for MetricsReporter. You need a good reason to justify this decision. It is as if you consider SslFactory the extension point. This is indeed something I proposed multiple

Re: [ DISCUSS ] KIP-512:Adding headers to RecordMetaData

Bumping this in the hope I can get more feedback and/or votes. https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData Thanks, Renuka M On Mon, Sep 9, 2019 at 2:41 PM Renuka M wrote: > > Hi All, > > Could you please take a look and let us know what you

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

+1 (non-binding) Le jeu. 12 sept. 2019 à 20:35, Gwen Shapira a écrit : > +1 (binding) > > On Mon, Sep 9, 2019 at 8:28 AM Colin McCabe wrote: > > > > Hi all, > > > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > Self-Managed Metadata Quorum. > > > > The DISCUSS thread from

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

+1 (binding) On Mon, Sep 9, 2019 at 8:28 AM Colin McCabe wrote: > > Hi all, > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a Self-Managed > Metadata Quorum. > > The DISCUSS thread from the mailing list is here: >

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

+1 (non-binding) Ron On Thu, Sep 12, 2019 at 5:04 AM Tom Bentley wrote: > +1 (non-binding) > > On Mon, Sep 9, 2019 at 4:28 PM Colin McCabe wrote: > > > Hi all, > > > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > > Self-Managed Metadata Quorum. > > > > The DISCUSS thread

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Sorry about the confusion, my notes were conflicting! Let me give an example to clarify. Let's say KIP-519 adds a new interface SslEngineFactory and I have a custom implementation MySslEngineFactory that gets keystore material from a web service using a custom config ` my.ssl.keystore.url`.

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

Thanks Arjun. +1 On Thu, Sep 12, 2019 at 9:58 AM Gwen Shapira wrote: > The new REST API for logger management looks great to me. > > > On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish > wrote: > > > > Bumping this thread. > > > > If there are no further comments, please add your votes here: > >

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

Hi Guozhang, That's a fair point. I think originally `committed` just hit the local cache. For EOS, we changed it to send a request on every call since the updated value came from outside the consumer. Given that, I think it is reasonable to push users toward the batch API. +1 -Jason On Thu,

[jira] [Created] (KAFKA-8902) Benchmark cooperative vs eager rebalancing

Sophie Blee-Goldman created KAFKA-8902: -- Summary: Benchmark cooperative vs eager rebalancing Key: KAFKA-8902 URL: https://issues.apache.org/jira/browse/KAFKA-8902 Project: Kafka Issue

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

+1 (binding) On Thu, Sep 12, 2019 at 1:53 PM Sophie Blee-Goldman wrote: > +1 (non-binding) > > On Wed, Sep 11, 2019 at 11:38 AM Vinoth Chandar > wrote: > > > +1 (non-binding). > > > > On Fri, Sep 6, 2019 at 12:46 AM Bruno Cadonna > wrote: > > > > > +1 (non-binding) > > > > > > On Fri, Sep 6,

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

+1 (non-binding) On Wed, Sep 11, 2019 at 11:38 AM Vinoth Chandar wrote: > +1 (non-binding). > > On Fri, Sep 6, 2019 at 12:46 AM Bruno Cadonna wrote: > > > +1 (non-binding) > > > > On Fri, Sep 6, 2019 at 12:32 AM Guozhang Wang > wrote: > > > > > > +1 (binding). > > > > > > On Thu, Sep 5, 2019

PR Broken builds

Hi, I’ve been working on KAFKA-8326 PR (https://github.com/apache/kafka/pull/6592 ), but I can never pass the checks. The build keeps failing with unrelated errors from SpotBugs. Seems like other people’s PRs are also affected. What’s the current

Re: _consumer_offsets is becoming rather big. How to purge?

Thanks Colin, that is the reason. On 2019/09/11 17:16:46, "Colin McCabe" wrote: > Hi Ash, > > At first guess, you probably had a problem with your log cleaner thread, > which resulted in the offsets log not being cleaned. Check if that thread is > running. > > best, > Colin > > On Wed,

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

The new REST API for logger management looks great to me. On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish wrote: > > Bumping this thread. > > If there are no further comments, please add your votes here: > https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html > > Thanks in advance, >

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

+1 (binding) - Thank you. Love the new REST API for logs. On Tue, Aug 20, 2019 at 3:05 PM Konstantine Karantasis wrote: > > Thanks for the updates on the KIP Arjun! > > +1 by me (non-binding) > > Konstantine > > > On Wed, Aug 14, 2019 at 6:46 AM Konstantine Karantasis < >

[jira] [Resolved] (KAFKA-8355) Add static membership to Range assignor

[ https://issues.apache.org/jira/browse/KAFKA-8355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8355. -- Fix Version/s: 2.4.0 Resolution: Fixed > Add static membership to Range assignor >

Re: Daily build artifacts

There are no nightly builds. -Matthias On 9/12/19 8:44 AM, Sean Glover wrote: > Hi, > > I'm curious if there's a daily build of trunk that can be accessed > publicly? Specifically, I'm interested in referencing client libraries > from a public maven or ivy repository, if one exists. I scanned

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

I'm confused. Can you launch a reconfiguration without a config change or not? If I understand the test case correctly, the design pattern to implement a reconfigurable extension point in Kafka is to implement the Reconfigurable interface. This means SslEngineFactory would be created once and

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Correction on my previous email. Custom implementations can use AlterConfig request without a config change by including `ssl.keystore.location` or `ssl.truststore.location` in their `reconfigurabkeConfigs()`. This will trigger the same codepath as we use for keystore updates when the file has

Daily build artifacts

Hi, I'm curious if there's a daily build of trunk that can be accessed publicly? Specifically, I'm interested in referencing client libraries from a public maven or ivy repository, if one exists. I scanned the Apache Kafka builds and the Release Process wiki, but I didn't see any mention of a

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Hi Clement, Kafka does special handling for keystore/truststore file changes when an AlterConfig request is processed, but that is not easy to extend to custom configs. I was thinking we could just add a custom config to trigger reconfiguration. For example, a config like

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

Bumping this thread. If there are no further comments, please add your votes here: https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html Thanks in advance, Arjun On Fri, Sep 6, 2019 at 4:22 PM Arjun Satish wrote: > Thanks a lot, Jason! Answers inline. I'll also modify the kip to

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

For the push notification, Rajini prefers if the trigger to reconfigure comes from an admin client. He says the admin client can reconfigure even though none of the config values have changed. This allows your custom impl to discover other conditions that have changed, for example the contents

Build failed in Jenkins: kafka-2.3-jdk8 #104

See -- Started by user davidarthur [EnvInject] - Loading node environment variables. Building remotely on H44 (ubuntu bionic) in workspace

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

True, the number is duplicated. do you know how can we get the problem fix? I was not aware, I missed, sorry the need to add the KIP to the table. -- Pere Missatge de Jorge Esteban Quilcate Otoya del dia dt., 3 de set. 2019 a les 13:43: > Hi Pere, > > Have you add your KIP to the list here

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

Hi Ismael, fair point and totally reasonable point. I added the CLI part to bring compatibility, but I can agree with not doing it right now per compatibility reasons. However, I do have a question for you: You say most of the CLI tools, is there anyone in your mind that you recall that would

[jira] [Created] (KAFKA-8901) Extend consumer group command to use the new Admin API to delete consumer offsets

David Jacot created KAFKA-8901: -- Summary: Extend consumer group command to use the new Admin API to delete consumer offsets Key: KAFKA-8901 URL: https://issues.apache.org/jira/browse/KAFKA-8901 Project:

Build failed in Jenkins: kafka-trunk-jdk8 #3901

See Changes: [jason] KAFKA-8875; CreateTopic API should check topic existence before -- [...truncated 6.09 MB...] org.apache.kafka.connect.data.DateTest >

Re: Kafka SSH Tunnel Connection without editing hostfile

Hey, I have done this before with this proxy: https://github.com/grepplabs/kafka-proxy#connect-to-kafka-through-socks5-proxy-example You can spin up a socks proxy when you ssh to your jumphost (-D argument if not mistaken) and configure the proxy as described in the readme. It is good for dev

Re: Kafka SSH Tunnel Connection without editing hostfile

Have you tried using EXTERNAL endpoints for your Kafka broker to separate TLS from internal traffic? Also, have you checked zk admin whether the broker metadata is exposing your TLS endpoints to clients ? On Thu, 12 Sep 2019 at 10:23, Akshay Das wrote: > Hi Team, > > I'm trying to consume from

Kafka SSH Tunnel Connection without editing hostfile

Hi Team, I'm trying to consume from a kafka cluster using java client, but the kafka server can only be accessed via jumphost/ssh tunnel. But even after creating ssh tunnel we are not able to read because once conusmer fetches metadata it uses original hosts to connect to broker. Is it possible

[jira] [Resolved] (KAFKA-8838) Allow consumer group tool to work with non-existing consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrik Kleindl resolved KAFKA-8838. --- Resolution: Not A Problem > Allow consumer group tool to work with non-existing consumer

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

+1 (non-binding). Thanks! On Thu, Sep 12, 2019 at 9:58 AM M. Manna wrote: > Gushing, > > Good kip. +1 (binding) from me. > > Thanks, > MAnna > > On Thu, 12 Sep 2019 at 09:55, Bruno Cadonna wrote: > > > Guozhang, Thanks for the KIP. > > > > +1 (non-binding) > > > > Best, > > Bruno > > > > On

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

+1 (non-binding) On Mon, Sep 9, 2019 at 4:28 PM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > Self-Managed Metadata Quorum. > > The DISCUSS thread from the mailing list is here: > >

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

Gushing, Good kip. +1 (binding) from me. Thanks, MAnna On Thu, 12 Sep 2019 at 09:55, Bruno Cadonna wrote: > Guozhang, Thanks for the KIP. > > +1 (non-binding) > > Best, > Bruno > > On Wed, Sep 11, 2019 at 9:17 AM Kamal Chandraprakash > wrote: > > > > Thanks for the KIP! > > > > LGTM, +1

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

Guozhang, Thanks for the KIP. +1 (non-binding) Best, Bruno On Wed, Sep 11, 2019 at 9:17 AM Kamal Chandraprakash wrote: > > Thanks for the KIP! > > LGTM, +1 (non-binding). > > On Wed, Sep 11, 2019 at 3:23 AM Matthias J. Sax > wrote: > > > I don't have a strong preference. So I am also fine to

Jenkins build is back to normal : kafka-trunk-jdk11 #810

See

Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver

Hi John, Thanks for explanation. Yes, indeed, this is happening in my app. I also checked my state store behind the output topic and there is always the latest value. Is it possible that TestTopologyDriver can fully mimic Kafka Streams? Maybe somehow records in the output topic can be ordered

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Thanks Clement and Rajini. Let me digest what both of you said. Clearly I need to understand more about the configurations - dynamic, custom etc. On the 'push notification' question Clement asked, The way I see is simple - Kafka defines the interface for SslEngineFactory. Implementation of that