Re: [DISCUSS] KIP-492 Add java security providers in Kafka Security config

Thanks for the KIP Sandeep. LGTM. Mani & Rajini, can you please look at the KIP as well. Thanks, Harsha On Tue, Jul 16, 2019, at 2:54 PM, Sandeep Mopuri wrote: > Thanks for the suggestions, made changes accordingly. > > On Tue, Jul 16, 2019 at 9:27 AM Satish Duggana > wrote: > > > Hi

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

You can look at the implementation here for an example https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java On Tue, Jul 16, 2019, at 9:00 PM, Harsha wrote: > Hi Maulin, >This is not

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

Hi Maulin, This is not required. We are already addressing this. One can write a KeyStoreProvider and TrustStoreProvider. Please look at this JIRA https://issues.apache.org/jira/browse/KAFKA-8191 . It allows users to add custom security provider in which you can write

Build failed in Jenkins: kafka-2.3-jdk8 #67

See Changes: [github] KAFKA-8602: Separate PR for 2.3 branch (#7092) -- [...truncated 2.91 MB...] kafka.utils.CoreUtilsTest > testCsvMap PASSED kafka.utils.CoreUtilsTest >

Build failed in Jenkins: kafka-trunk-jdk8 #3790

See Changes: [github] MINOR: Increase `awaitCommits` timeout in ExampleConnectIntegrationTest -- [...truncated 6.43 MB...]

[jira] [Resolved] (KAFKA-1923) Negative offsets in replication-offset-checkpoint file

[ https://issues.apache.org/jira/browse/KAFKA-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1923. Resolution: Duplicate > Negative offsets in replication-offset-checkpoint file >

Build failed in Jenkins: kafka-trunk-jdk11 #700

See Changes: [github] MINOR: Increase `awaitCommits` timeout in ExampleConnectIntegrationTest -- [...truncated 2.55 MB...]

[VOTE] KIP-467: Augment ProduceResponse error messaging

Hello folks, I'd like to start a voting thread on KIP-467 to improve error communication and handling for producer response: https://cwiki.apache.org/confluence/display/KAFKA/KIP-467 %3A+Augment+ProduceResponse+error+messaging+for+specific+culprit+records Thanks, -- Guozhang

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hello folks, I've updated the wiki a bit more and here are the changes: 1. *InvalidRecordException* would inherit from ApiException (which is already non-retriable), not KafkaException. 2. Explained how to set the error codes if there are multiple error scenarios on the broker-side. I'm going

Re: [DISCUSS] KIP-492 Add java security providers in Kafka Security config

Thanks for the suggestions, made changes accordingly. On Tue, Jul 16, 2019 at 9:27 AM Satish Duggana wrote: > Hi Sandeep, > Thanks for the KIP, I have few comments below. > > >>“To take advantage of these custom algorithms, we want to support java > security provider parameter in security

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

It's a fair point that Ryanne raises. However, "running sum" is the same as "moving sum" from my understanding. The issue is still, that `Sum` and `Count` which seem to be the cleanest names cannot be used. While I agree that `TotalSum` and `TotalCount` is somewhat redundant, I still think it the

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

I'm +1 on Windowed, was about to suggest that as I was catching up on the discussion but Bill beat me to it :) On Tue, Jul 16, 2019 at 2:23 PM Bill Bejeck wrote: > Hi John, > > Thanks for the updates. > > I like RunningCount and RunningSum. > > What about WindowedCount, WindowedSum instead of

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

Hi John, Thanks for the updates. I like RunningCount and RunningSum. What about WindowedCount, WindowedSum instead of Moving? I'm just throwing this out there as Windowed seems more intuitive to me, but I'm not married to the idea. -Bill On Tue, Jul 16, 2019 at 5:09 PM John Roesler wrote: >

Re: [VOTE] KIP-466: Add support for List serialization and deserialization

Thanks, Daniyar! I'm +1 (nonbinding) -John On Tue, Jul 16, 2019 at 3:14 PM Development wrote: > > Hi, > > I’d like to start a vote thread for KIP-466. > > This addition will introduce new serde type ListSerde. > > More info at: >

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

No worries! Choosing good public API names is a high-impact design activity. Matthias, Bruno, Bill, and Stanislav, You've all contributed to this discussion or the vote so far... How do you feel about the proposed name change: MovingCount, MovingSum (instead of Sampled) RunningCount,

Build failed in Jenkins: kafka-trunk-jdk8 #3789

See Changes: [github] KAFKA-8662; Fix producer metadata error handling and consumer manual [bbejeck] KAFKA-8450: Using KeyValueTimeStamp in MockProcessor (#6933) --

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

Bump! Can somebody please review this?

[VOTE] KIP-466: Add support for List serialization and deserialization

Hi, I’d like to start a vote thread for KIP-466. This addition will introduce new serde type ListSerde. More info at: https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization Best, Daniyar Yeralin

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi, Yes, totally forgot about the statement. KIP-466 is updated. Thank you so much John Roesler, Matthias J. Sax, Sophie Blee-Goldman for your valuable input! I hope I did not cause too much trouble :) I’ll start the vote now. Best, Daniyar Yeralin > On Jul 16, 2019, at 3:17 PM, John

Re: [VOTE] KIP-484: Expose metrics for group and transaction metadata loading duration

Hey everyone! Thanks for your input. This KIP will be marked as accepted with 3 binding votes from Gwen, Jason, and Colin. Anastasia On Mon, Jul 15, 2019 at 7:36 PM Colin McCabe wrote: > +1 (binding). Thanks for the KIP, Anastasia. > > best, > Colin > > > On Fri, Jul 12, 2019, at 10:09,

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

John, that makes sense to me. Sorry for the bikeshedding. Ryanne On Tue, Jul 16, 2019 at 12:49 PM John Roesler wrote: > Thanks for the explanation and the suggestion, Ryanne, > > I went with "sampled" just because these are instances of SampledStat, > which in the Kafka Metrics ecosystem are

Build failed in Jenkins: kafka-trunk-jdk11 #699

See Changes: [bbejeck] KAFKA-8450: Using KeyValueTimeStamp in MockProcessor (#6933) -- [...truncated 2.60 MB...]

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi Daniyar, Thanks for that update. I took a look, and I think this is in good shape. One note, the statement "New method public static Serde> ListSerde() in org.apache.kafka.common.serialization.Serdes class (infers list implementation and inner serde from config file)" is still present in the

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi, Pushed new changes under my PR: https://github.com/apache/kafka/pull/6592 Feel free to put any comments in there. Best, Daniyar Yeralin > On Jul 15, 2019, at 1:06 PM, Development wrote: > > Hi John, > > I knew I was missing something. Yes,

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

Thanks for the explanation and the suggestion, Ryanne, I went with "sampled" just because these are instances of SampledStat, which in the Kafka Metrics ecosystem are computed from a window of recent samples. Thinking more about it, the fact that they are sampled and the fact that they are

Build failed in Jenkins: kafka-trunk-jdk8 #3788

See Changes: [matthias] KAFKA-8620: fix NPE due to race condition during shutdown while [wangguoz] KAFKA-8637: WriteBatch objects leak off-heap memory (#7050) --

Build failed in Jenkins: kafka-2.3-jdk8 #66

See Changes: [rajinisivaram] KAFKA-8662; Fix producer metadata error handling and consumer manual [rajinisivaram] KAFKA-8530; Check for topic authorization errors in OffsetFetch response

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

> measurements, which decay/expire over time Thanks John for the clarification. This was my (re-)reading of the code, but this is not what I think of when I hear "sampled". In fact, you'll notice that the Wikipedia pages for "Sample (statistics)" and "Sample (signal processing)" do not contain

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

Thanks for raising this concern, Ryanne, "Sampled" indicates that the metrics is sampled, namely that we maintain a set of samples from recent value measurements, which decay/expire over time. So, the metric value is only representative of the recent past. "Total" indicates that the metric value

Re: [DISCUSS] KIP-492 Add java security providers in Kafka Security config

Hi Sandeep, Thanks for the KIP, I have few comments below. >>“To take advantage of these custom algorithms, we want to support java >>security provider parameter in security config. This param can be used by >>kafka brokers or kafka clients(when connecting to the kafka brokers). The >>security

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

John, I mentioned on the VOTE thread that the proposed names are a bit confusing, > given that "sum", "total", and "count" are roughly synonymous... In particular, TotalSum is, I think, a "running total", though the naming doesn't really capture that. I think to avoid confusion, we should

Build failed in Jenkins: kafka-trunk-jdk11 #698

See Changes: [github] KAFKA-8662; Fix producer metadata error handling and consumer manual -- [...truncated 2.61 MB...]

[jira] [Created] (KAFKA-8674) Records that exceed maximum size on the Java Producer don't cause transaction failure

William Parker created KAFKA-8674: - Summary: Records that exceed maximum size on the Java Producer don't cause transaction failure Key: KAFKA-8674 URL: https://issues.apache.org/jira/browse/KAFKA-8674

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Hi again, all, I have started the voting thread. Please cast your votes (or voice your objections)! The vote will remain open at least 72 hours. Once it closes, I can send the PR pretty quickly. Thanks for all you help ironing out the details on this feature. -John On Mon, Jul 15, 2019 at 5:09

[VOTE] KIP-478 Strongly Typed Streams Processor API

Hi Dev, After a good discussion, I'd like to start the vote for KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw). The proposal is to deprecate the existing interface org.apache.kafka.streams.processor.Processor in favor of a new one, org.apache.kafka.streams.processor.api.Processor that

Re: [DISCUSS] KIP-479: Add Materialized to Join

Hi Bill, Thanks for driving this KIP toward a conclusion. I'm on board with your decision. You didn't mention whether you're still proposing to add the "queriable" flag to the Materialized config object, or just document that a Stream join is never queriable. Both options have come up earlier in

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hi Patrik, Ah, I guess that was an important distinction! Thanks for clarifying. If there's a method to call on the RocksDB, then you probably want to consider just registering the metric as a Gauge that delegates to RocksDB to get the value, as opposed to the normal pattern of using a Sensor

Re: Behaviour of KStreamKTableJoinProcessor

Hi again (again), Ties, Sorry for the confusion, but I was talking to someone else about this, and I started to make a ticket to fix it, and realized once I started looking into it that there is actually no repartition topic for a stream-globalTable join. So, if you do something like: =

[jira] [Created] (KAFKA-8673) Kafka stream threads stuck while sending offsets to transaction preventing join group from completing

Varsha Abhinandan created KAFKA-8673: Summary: Kafka stream threads stuck while sending offsets to transaction preventing join group from completing Key: KAFKA-8673 URL:

[jira] [Resolved] (KAFKA-8662) Produce fails if a previous produce was to an unauthorized topic

[ https://issues.apache.org/jira/browse/KAFKA-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8662. --- Resolution: Fixed Reviewer: Jason Gustafson > Produce fails if a previous produce was

Build failed in Jenkins: kafka-2.3-jdk8 #65

See Changes: [wangguoz] KAFKA-8637: WriteBatch objects leak off-heap memory (#7050) -- [...truncated 2.72 MB...] kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo

Jenkins build is back to normal : kafka-trunk-jdk11 #697

See

Build failed in Jenkins: kafka-trunk-jdk8 #3787

See Changes: [bbejeck] KAFKA-8602: Fix bug in stand-by task creation (#7008) -- [...truncated 8.88 MB...] org.apache.kafka.streams.TopologyTestDriverTest >