[jira] [Resolved] (KAFKA-8587) One producer per thread for Kafka Streams EOS

[ https://issues.apache.org/jira/browse/KAFKA-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-8587. Resolution: Fixed > One producer per thread for Kafka Streams EOS >

Build failed in Jenkins: kafka-trunk-jdk8 #4499

See Changes: [github] MINOR: Clean up some test dependencies on ConfigCommand and TopicCommand -- [...truncated 2.90 MB...]

[jira] [Created] (KAFKA-9955) Exceptions thrown from SinkTask::close shadow other exceptions

Greg Harris created KAFKA-9955: -- Summary: Exceptions thrown from SinkTask::close shadow other exceptions Key: KAFKA-9955 URL: https://issues.apache.org/jira/browse/KAFKA-9955 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk11 #1423

See Changes: [github] KAFKA-9918; SslEngineFactory is NOT closed when channel is closing -- [...truncated 3.08 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

[jira] [Created] (KAFKA-9954) Config command didn't validate the unsupported user config change

Cheng Tan created KAFKA-9954: Summary: Config command didn't validate the unsupported user config change Key: KAFKA-9954 URL: https://issues.apache.org/jira/browse/KAFKA-9954 Project: Kafka

Re: [DISCUSS] Kafka 3.0

+1 for removing MM 1.0 when we cut a breaking release. It is sad to see that we are still investing in it (I just saw a KIP toward improving its reset policy). My understanding was that KIP-590 is not breaking compatibility, I think Guozhang said that in response to my question on the discussion

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Hi Cheng, On the KIP page, it lists this KIP as "draft." It seems like "under discussion" is appropriate here, right? > Currently, the initial socket connection timeout is depending on Linux kernel > setting > tcp_syn_retries. The timeout value is 2 ^ (tcp_sync_retries + 1) - 1 seconds.  >

Re: [DISCUSS] Kafka 3.0

Hey Colin, I think we should wait until after KIP-500's "bridge release" so there is a clean break from Zookeeper after 3.0. The bridge release by definition is an attempt to not break anything, so it theoretically doesn't warrant a major release. If that's not the case (i.e. if a single "bridge

Build failed in Jenkins: kafka-trunk-jdk8 #4498

See Changes: [github] KAFKA-9918; SslEngineFactory is NOT closed when channel is closing -- [...truncated 2.90 MB...] org.apache.kafka.streams.MockTimeTest >

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

Cool, that makes sense. Guozhang On Mon, May 4, 2020 at 2:50 PM Colin McCabe wrote: > I think once something becomes more complex than just key = value it's > time to consider an official Kafka API, rather than trying to fit it into > AlterConfigs. For example, for client quotas, we have

[DISCUSS] Kafka 3.0

Hi all, We've had a few proposals recently for incompatible changes. One of them is my KIP-604: Remove ZooKeeper Flags from the Administrative Tools. The other is Boyang's KIP-590: Redirect ZK Mutation Protocols to the Controller. I think it's time to start thinking about Kafka 3.0.

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

I think once something becomes more complex than just key = value it's time to consider an official Kafka API, rather than trying to fit it into AlterConfigs. For example, for client quotas, we have KIP-546. There are just so many reasons. Real Kafka APIs have well-defined compatibility

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

Got it. Besides SCRAM, are there other scenarios that we may have such "hierarchical" (I know the term may not be very accurate here :P) configs such as "config1=[key1=value1, key2=value2]" compared with most common pattern of "config1=value1" or "config1=value1,config2=value2"? For example I

[jira] [Created] (KAFKA-9953) support multiple consumerGroupCoordinators in TransactionManager

Joost van de Wijgerd created KAFKA-9953: --- Summary: support multiple consumerGroupCoordinators in TransactionManager Key: KAFKA-9953 URL: https://issues.apache.org/jira/browse/KAFKA-9953

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

Hi Sönke, You're right on both counts. Thanks for the corrections. Thinking about this more, I think we should just remove kafka-preferred-replica-election.sh. It just duplicates kafka-leader-election.sh, and it has been deprecated for some time. I changed the KIP. cheers, Colin On Mon,

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

Hi Tom, As you said, the controller doesn't know about log directories, although individual brokers do. So the brokers do currently have to enumerate all the partitions that need to be removed to the controllers explicitly. So this KIP isn't changing anything in that regard. The current

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

If we use AlterConfigs then we end up parsing strings like 'SCRAM-SHA-256=[iterations=8192,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' on the broker into the same information that's currently in ScramUserAlteration. This doesn't change the complexity of the command-line

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Hmm. A big part of the reason behind the KIP is that the default connection timeout behavior of the OS doesn't work for Kafka, right? For example, on Linux, if we wait 127 seconds for a connection attempt to time out, we won't get a chance to make another attempt in most cases. So I think it

Re: [DISCUSS] KIP-605 Expand Connect Worker Internal Topic Settings

Hi Randall, Looks great! Definitely in favor of a WARN level message instead of failing on startup. +1 non-binding when the vote thread comes Cheers, Chris On Mon, May 4, 2020 at 12:51 PM Randall Hauch wrote: > Thanks, Chris. > > 1. Added a use case that describes why you might want to use

Re: [DISCUSS] KIP-605 Expand Connect Worker Internal Topic Settings

Thanks, Chris. 1. Added a use case that describes why you might want to use -1 for replication factor but want to set other properties. 2&3. Thanks for bringing up these special properties that the worker always sets. I've added an "Excluded properties" column to the table of new properties, and

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Thanks for the feedback, Konstantine. I do agree it's probably better to keep the KIP focused and avoid this kind of baggage, since connectors that *can* optionally use a reader if available can pretty easily deal with the checks. That's 3 binding +1 votes (Gwen, me, and Konstantine), and the

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Hi, Guozhang, Thanks for the reply. A few more replies inlined below. On Sun, May 3, 2020 at 6:33 PM Guozhang Wang wrote: > Hello Jun, > > Thanks for your comments! I'm replying inline below: > > On Fri, May 1, 2020 at 12:36 PM Jun Rao wrote: > > > 101. Bootstrapping related issues. > > 101.1

[jira] [Created] (KAFKA-9952) Reconsider immediate propagation of hw when replica selector is used

Jason Gustafson created KAFKA-9952: -- Summary: Reconsider immediate propagation of hw when replica selector is used Key: KAFKA-9952 URL: https://issues.apache.org/jira/browse/KAFKA-9952 Project:

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

Hi Guozhang, Thank you for the KIP! Exposing also the inner types of the wrapper serdes would be important. For debugging as Matthias has already mentioned and to see more easily changes that are applied to a topology. I am also +1 on the `toJson()` method to easily access the topology

[GitHub] [kafka-site] mjsax commented on pull request #264: MINOR: Reorder the documentation sub-title to sync with the TOC

mjsax commented on pull request #264: URL: https://github.com/apache/kafka-site/pull/264#issuecomment-623619312 Thanks for the PR @showuon! This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Created] (KAFKA-9951) More testing of ReplicaSelector

Jason Gustafson created KAFKA-9951: -- Summary: More testing of ReplicaSelector Key: KAFKA-9951 URL: https://issues.apache.org/jira/browse/KAFKA-9951 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-9918) SslEngineFactory is NOT closed when channel is closing

[ https://issues.apache.org/jira/browse/KAFKA-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9918. Fix Version/s: 2.6.0 Resolution: Fixed > SslEngineFactory is NOT closed when

[VOTE] KIP-592: Kafka MirrorMaker 1 should replicate topics from "earliest"

Hi all, I'd like to start a vote on KIP-592: https://cwiki.apache.org/confluence/display/KAFKA/KIP-592%3A+MirrorMaker+1.0+should+replicate+topics+from+earliest Thanks -- *Jeff Widman* jeffwidman.com | 740-WIDMAN-J (943-6265) <><

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

Thanks for the explanation John. Guozhang On Sun, May 3, 2020 at 7:10 PM John Roesler wrote: > Hi Guozhang, > > Ah, good question. Yes, the assignor will always now try to achieve a > perfect balance. This was also the proposed default in the KIP before. The > config would have allowed users

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Thanks for the KIP Cheng, > The default value will be 10 seconds. I think we should make the default the current behavior. Meaning the default should leverage the default connect timeout from the operating system. > Proposed Changes I don't fully understand this section. It seems like it is

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

+1, non-binding On Mon, May 4, 2020, 9:24 AM Christopher Egerton wrote: > +1 (non-binding) > > On Mon, May 4, 2020 at 5:02 AM Edoardo Comar wrote: > > > +1 (non-binding) > > Thanks Mickael > > > > -- > > > > Edoardo Comar > > > > Event Streams

Re: [DISCUSS] KIP-605 Expand Connect Worker Internal Topic Settings

Thanks for the KIP Randall! Automatic creation of internal Connect topics has been very useful since KIP-154 and adapting the Connect workers to allow users to transparently use broker defaults will be useful as well. The KIP looks good. Konstantine On Sun, May 3, 2020 at 10:55 AM Christopher

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

I agree with the motivation of this KIP. I think it will enrich the capabilities of source connectors in a quite meaningful way. I'm +1 as well (binding). P.S. Randall I like your idea of enabling connectors to run in old as well as new workers after this KIP. However I think that such

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

+1 (non-binding) On Mon, May 4, 2020 at 5:02 AM Edoardo Comar wrote: > +1 (non-binding) > Thanks Mickael > > -- > > Edoardo Comar > > Event Streams for IBM Cloud > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > From: Mickael Maison > To: dev

[DISCUSSING] KIP-575: build a Kafka-Exporter-in-Java for the ALL metrics

hi all, iposted a KIP-575 about building a Kafka-Exporter-in-Java for ALL metrics.Check it out here if you get a chance:https://cwiki.apache.org/confluence/display/KAFKA/KIP-575%3A+build+a+Kafka-Exporter+by+Java Have a nice dayFrancis Lee QQ : 1026203200

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

+1 (non-binding) Thanks Mickael -- Edoardo Comar Event Streams for IBM Cloud IBM UK Ltd, Hursley Park, SO21 2JN From: Mickael Maison To: dev Date: 04/05/2020 11:45 Subject:[EXTERNAL] [VOTE] KIP-597: MirrorMaker2 internal

Re: [DISCUSS] KIP-585: Conditional SMT

Hi Tom, Referring back to Chris's points. 1) Exception handling - yes, agreed. This KIP can just say the exception handling for predicates is the same as transformations. 2) Worker plugins - agree too. 3) Filter SMT - If the Filter SMT is just becoming a transformation that drops any records

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

Hi Mickael, Thanks for the KIP. +1 (non-binding) Thanks, Andrew Schofield On 04/05/2020, 11:45, "Mickael Maison" wrote: Hi all, I'd like to start a vote on KIP-597: https://cwiki.apache.org/confluence/display/KAFKA/KIP-597%3A+MirrorMaker2+internal+topics+Formatters Thanks

[VOTE] KIP-597: MirrorMaker2 internal topics Formatters

Hi all, I'd like to start a vote on KIP-597: https://cwiki.apache.org/confluence/display/KAFKA/KIP-597%3A+MirrorMaker2+internal+topics+Formatters Thanks

Re: [DISCUSS] KIP-585: Conditional SMT

Hi Chris, Thanks for the feedback. On points 1 and 2 I agree. 3. This is a great point. I'm happy to add this to the KIP too, but note it removes the need to add validate(Map) to both Transformation and Predicate. That doesn't mean we shouldn't still do it, just that the change is not motivated

Build failed in Jenkins: kafka-trunk-jdk11 #1422

See Changes: [github] KAFKA-9798: Send one round synchronously before starting the async -- [...truncated 2.72 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

[GitHub] [kafka-site] showuon opened a new pull request #264: MINOR: Reorder the documentation sub-title to sync with the TOC

showuon opened a new pull request #264: URL: https://github.com/apache/kafka-site/pull/264 current Table of Content(TOC) in navigation sidebar:  **TOC in the content:**

Build failed in Jenkins: kafka-trunk-jdk14 #53

See Changes: [github] KAFKA-9798: Send one round synchronously before starting the async -- [...truncated 3.08 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

Hi Colin, thanks for the kip, lgtm overall with two small comments: 1. you mention kafka-leader-election.sh in the list of tools to remove the ZooKeeper option from, but I think the command never had that option, as it was introduced in KIP-460 specifically to replace the deprecated command with

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Ah, I see, thanks for the clarification! Shouldn't be an issue I think. My understanding of KIPs was always that they are mostly intended as a place to discuss and agree changes up front, whereas tracking the actual releases that things go into should be handled in Jira. So maybe we just create

Build failed in Jenkins: kafka-trunk-jdk8 #4497

See Changes: [github] KAFKA-9798: Send one round synchronously before starting the async -- [...truncated 3.06 MB...] org.apache.kafka.streams.TopologyTestDriverTest >