[jira] [Created] (KAFKA-5808) The Preferred Replica should be global and dynamic

Canes Kelly created KAFKA-5808: -- Summary: The Preferred Replica should be global and dynamic Key: KAFKA-5808 URL: https://issues.apache.org/jira/browse/KAFKA-5808 Project: Kafka Issue Type: New

Jenkins build is back to normal : kafka-trunk-jdk8 #1955

See

Build failed in Jenkins: kafka-trunk-jdk8 #1954

See Changes: [wangguoz] KAFKA-5379: ProcessorContext.appConfigs() should return parsed values -- [...truncated 4.79 MB...] org.apache.kafka.streams.StreamsConfigTest >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi, Rajini, Thanks for the updated KIP. I agree that those additional metrics can be useful. I was thinking what would an admin do if the value of one of those metrics is abnormal. An admin probably want to determine which client causes the abnormaly. So, a couple of more comments below. 10.

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

Sounds good to me as well. On Tue, Aug 29, 2017 at 2:43 AM, Ismael Juma wrote: > Sounds good to me too. Since this is a non controversial change, I suggest > starting the vote in 1-2 days if no-one else comments. > > Ismael > > On Thu, Aug 24, 2017 at 7:32 PM, Jason Gustafson

[GitHub] kafka-site pull request #72: Jason and Becket are PMC members

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka-site/pull/72 Jason and Becket are PMC members You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka-site update-pmc-members Alternatively

Jenkins build is back to normal : kafka-0.10.2-jdk7 #194

See

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

I think we need a KIP for config deprecations. The case for this one seems clear-cut since we strongly depend on the log cleaner for the management of consumer offsets and the transaction log, but we can see what others think. -Jason On Fri, Aug 25, 2017 at 10:00 AM, Pranav Maniar

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Hi Rajini, One more thought. Maybe we should also add an error_message field in the response like we do with CreateTopics response so that the server can return an appropriate message that we can bubble up to the user. Examples would be "Invalid username or password", "SASL Impersonation not

Jenkins build is back to normal : kafka-trunk-jdk7 #2688

See

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Yeah, I think expiring a batch but still wait for the response is probably reasonable given the result is not guaranteed anyways. @Jun, I think the frequent PID reset may still be possible if we do not wait for the in-flight response to return. Consider two partitions p0 and p1, the deadline of

Re: Permission to create KIP for discussions and the max.in.flight.requests.per.connection config doc

Hi Bhaskar, Comment inline. On Sun, Aug 27, 2017 at 12:37 PM, Bhaskar Gollapudi < bhaskargollap...@gmail.com> wrote: > max.in.flight.requests.per.connection The maximum number of unacknowledged > requests the client will send on a single connection before blocking. Note > that if this setting

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Thanks, Jason. +1 (non-binding) Colin On Tue, Aug 29, 2017, at 09:12, Jason Gustafson wrote: > Hi Colin, > > Thanks for the comments. Seems reasonable to provide a safer `equals` for > extensions. I don't think this needs to be part of the KIP, but I can add > it to my patch. > > Moving

Re: [DISCUSS] KIP-171: Extend Consumer Group Reset Offset for Stream Application

Hi Jorge, Thanks for the KIP. It would be a great to add feature to the reset tools. I made a pass over it and it looks good to me overall. I have a few comments: 1. For all the scenarios, do we allow users to specify more than one parameters? If not could you make that clear in the wiki, e.g.

Re: Permission to create KIP for discussions and the max.in.flight.requests.per.connection config doc

Hi, Bhaskar, Thanks for your interest. Just granted you the wiki permission. Jun On Sun, Aug 27, 2017 at 4:37 AM, Bhaskar Gollapudi < bhaskargollap...@gmail.com> wrote: > Hi , > > I would like to have have some permissions to create a KIP and start a > discussion. > > I think I was able to

[GitHub] kafka pull request #3721: Upgrade to ducktape 0.7.1

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3721 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5768) Upgrade ducktape version to 0.7.1, and use new kill_java_processes

[ https://issues.apache.org/jira/browse/KAFKA-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-5768. Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3721

[DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

Hi, In the discussion of KIP-185 (enable idempotence by default), we discovered some shortcomings of the existing idempotent producer implementation. Fixing these issues requires changes to the ProduceRequest and ProduceResponse protocols as well as changes to the values of the

Jenkins build is back to normal : kafka-0.11.0-jdk7 #289

See

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Hmm, I thought delivery.timeout.ms bounds the time from a message is in the accumulator (i.e., when send() returns) to the time when the callback is called. If we wait for request.timeout.ms for an inflight request and the remaining delivery.timeout.ms is less than request.timeout.ms, the callback

Build failed in Jenkins: kafka-0.10.2-jdk7 #193

See Changes: [me] MINOR: Verify startup of zookeeper service in system tests -- [...truncated 323.77 KB...] kafka.server.KafkaConfigTest >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

I think the semantics of delivery.timeout.ms need to allow for the possibility that the record was actually written. Unless we can keep on retrying indefinitely, there's really no way to know for sure whether the record was written or not. A delivery timeout just means that we cannot guarantee

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi Jason, If we expire the batch from user's perspective but still waiting for the response, would that mean it is likely that the batch will be successfully appended but the users will receive a TimeoutException? That seems a little non-intuitive to the users. Arguably it maybe OK though because

[GitHub] kafka pull request #3757: KAFKA-5379 follow up: reduce redundant mock proces...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3757 KAFKA-5379 follow up: reduce redundant mock processor context You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka

[jira] [Resolved] (KAFKA-5779) Single message may exploit application based on KStream

[ https://issues.apache.org/jira/browse/KAFKA-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-5779. Resolution: Not A Problem > Single message may exploit application based on KStream >

Build failed in Jenkins: kafka-trunk-jdk7 #2687

See Changes: [wangguoz] KAFKA-5379: ProcessorContext.appConfigs() should return parsed values -- [...truncated 914.18 KB...] kafka.server.epoch.LeaderEpochFileCacheTest

[jira] [Resolved] (KAFKA-3570) It'd be nice to line up display output in columns in ConsumerGroupCommand

[ https://issues.apache.org/jira/browse/KAFKA-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3570. -- Resolution: Fixed > It'd be nice to line up display output in columns in ConsumerGroupCommand >

[jira] [Resolved] (KAFKA-1980) Console consumer throws OutOfMemoryError with large max-messages

[ https://issues.apache.org/jira/browse/KAFKA-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1980. -- Resolution: Fixed Closing as per comments. Pl reopen if you think the issue still exists > Console

[GitHub] kafka pull request #3756: KAFKA-5261 added cached authorizer

GitHub user simplesteph opened a pull request: https://github.com/apache/kafka/pull/3756 KAFKA-5261 added cached authorizer attempt to improve Kafka performance by caching the results of the authorisation calls. You can merge this pull request into a Git repository by running:

[jira] [Resolved] (KAFKA-1675) bootstrapping tidy-up

[ https://issues.apache.org/jira/browse/KAFKA-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1675. -- Resolution: Fixed gradlew, gradlew.bat scripts are removed from repo. Pl reopen if you think the

[GitHub] kafka pull request #3455: KAFKA-5379 - ProcessorContext.appConfigs() should ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3455 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5379) ProcessorContext.appConfigs() should return parsed/validated values

[ https://issues.apache.org/jira/browse/KAFKA-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5379. -- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3455

[jira] [Resolved] (KAFKA-2173) Kafka died after throw more error

[ https://issues.apache.org/jira/browse/KAFKA-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2173. -- Resolution: Cannot Reproduce Might have fixed in latest versions. Pl reopen if you think the issue

[jira] [Resolved] (KAFKA-1636) High CPU in very active environment

[ https://issues.apache.org/jira/browse/KAFKA-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1636. -- Resolution: Won't Fix ConsumerIterator waits for the data from the underlying stream. Pl reopen if you

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Hi Rajini, The metrics in KIP-188 will provide counts across all users but the log could potentially be used to audit individual authentication events. I think these would be useful at INFO level but if it's inconsistent with the rest of Kafka, DEBUG is ok too. The default log4j config for

[jira] [Created] (KAFKA-5807) NPE on Connector.validate

Jeremy Custenborder created KAFKA-5807: -- Summary: NPE on Connector.validate Key: KAFKA-5807 URL: https://issues.apache.org/jira/browse/KAFKA-5807 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

I think I'm with Becket. We should wait for request.timeout.ms for each produce request we send. We can still await the response internally for PID/sequence maintenance even if we expire the batch from the user's perspective. New sequence numbers would be assigned based on the current PID until

[GitHub] kafka pull request #3755: KAFKA-5806. Fix transient unit test failure in tro...

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/3755 KAFKA-5806. Fix transient unit test failure in trogdor coordinator shutdown In the coordinator, we should check that 'shutdown' is not true before going to sleep waiting for the condition. You can

[jira] [Created] (KAFKA-5806) Fix transient unit test failure in trogdor coordinator shutdown

Colin P. McCabe created KAFKA-5806: -- Summary: Fix transient unit test failure in trogdor coordinator shutdown Key: KAFKA-5806 URL: https://issues.apache.org/jira/browse/KAFKA-5806 Project: Kafka

[jira] [Resolved] (KAFKA-4634) Issue of one kafka brokers not listed in zookeeper

[ https://issues.apache.org/jira/browse/KAFKA-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-4634. -- Resolution: Fixed Similar issue fixed in KAFKA-1387/Newer versions. Pl reopen if you think the issue

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

I'm updating the kip-91 writeup. There seems to be some confusion about expiring an inflight request. An inflight request gets a full delivery.timeout.ms duration from creation, right? So it should be max(remaining delivery.timeout.ms, request.timeout.ms)? Jun, we do want to wait for an inflight

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Hi Roger, If we are changing logging level for successful SASL authentications in the broker, we should probably do the same for SSL too. Since KIP-188 proposes to add new metrics for successful and failed authentications which may be more useful for monitoring, do we really need info-level

[GitHub] kafka pull request #3713: MINOR: simplify state transition for Kafka Streams...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3713 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Great improvement! +1 On Tue, Aug 29, 2017 at 10:09 AM, Roger Hoover wrote: > Just re-read the KIP and was wondering if you think INFO would be ok for > logging successful authentications? They should be relatively infrequent. > > On Tue, Aug 29, 2017 at 9:54 AM, Roger

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Just re-read the KIP and was wondering if you think INFO would be ok for logging successful authentications? They should be relatively infrequent. On Tue, Aug 29, 2017 at 9:54 AM, Roger Hoover wrote: > +1 (non-binding). Thanks, Rajini > > On Tue, Aug 29, 2017 at 2:10

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Great suggestions, Ismael and thanks for incorporating them, Rajini. Tracking authentication success and failures (#3) across listeners seems very useful for cluster administrators to identify misconfigured client or bad actors, especially until all clients implement KIP-152 which will add an

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

+1 (non-binding). Thanks, Rajini On Tue, Aug 29, 2017 at 2:10 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Thu, Aug 24, 2017 at 6:29 PM, Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to start vote

[jira] [Created] (KAFKA-5805) The console consumer crushes broker on Windows

Aleksandar created KAFKA-5805: - Summary: The console consumer crushes broker on Windows Key: KAFKA-5805 URL: https://issues.apache.org/jira/browse/KAFKA-5805 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Thanks, +1 On Tue, Aug 29, 2017 at 8:37 AM, Gwen Shapira wrote: > +1 > Thank you for this improvement! > > On Tue, Aug 29, 2017 at 7:46 AM, Mickael Maison > wrote: > > > +1 non binding > > Thanks > > > > On Tue, Aug 29, 2017 at 2:17 AM, Manikumar

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Hi Colin, Thanks for the comments. Seems reasonable to provide a safer `equals` for extensions. I don't think this needs to be part of the KIP, but I can add it to my patch. Moving `fromString` makes sense also. This method is technically already part of the public API, which means we should

[VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi all, I would like to start the vote on KIP-183 which will provide an AdminClient interface for electing the preferred replica, and refactor the kafka-preferred-replica-election.sh tool to use this interface. More details here:

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

+1 Thank you for this improvement! On Tue, Aug 29, 2017 at 7:46 AM, Mickael Maison wrote: > +1 non binding > Thanks > > On Tue, Aug 29, 2017 at 2:17 AM, Manikumar > wrote: > > +1 (non-binding) > > > > On Tue, Aug 29, 2017 at 2:43 PM, Stevo

[GitHub] kafka pull request #3754: KAFKA-5804: retain duplicates in ChangeLoggingWind...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3754 KAFKA-5804: retain duplicates in ChangeLoggingWindowBytesStore `ChangeLoggingWindowBytesStore` needs to have the same `retainDuplicates` functionality as `RocksDBWindowStore` else data could be lost

[jira] [Created] (KAFKA-5804) ChangeLoggingWindowBytesStore needs to retain duplicates when writing to the log

Damian Guy created KAFKA-5804: - Summary: ChangeLoggingWindowBytesStore needs to retain duplicates when writing to the log Key: KAFKA-5804 URL: https://issues.apache.org/jira/browse/KAFKA-5804 Project:

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

+1 non binding Thanks On Tue, Aug 29, 2017 at 2:17 AM, Manikumar wrote: > +1 (non-binding) > > On Tue, Aug 29, 2017 at 2:43 PM, Stevo Slavić wrote: > >> +1 (non-binding) >> >> On Tue, Aug 29, 2017 at 11:09 AM, Ismael Juma wrote:

[GitHub] kafka pull request #3753: Allow timestamp parameter in `ProcessorTopologyTes...

GitHub user sebigavril opened a pull request: https://github.com/apache/kafka/pull/3753 Allow timestamp parameter in `ProcessorTopologyTestDriver.process` All current implementations process records using the same timestamp. This makes it difficult to test operations that require

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Ismael, Thank you for the suggestions. The additional metrics sound very useful and I have added them to the KIP. Regards, Rajini On Tue, Aug 29, 2017 at 5:34 AM, Ismael Juma wrote: > Hi Rajini, > > There are a few other metrics that could potentially be useful. I'd be

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for the proposals. I think they make sense and I also agree with Jason's suggestions. Also, it would be good to include the updated ProduceRequest/Response schema in the KIP. Ismael On Tue, Aug 22, 2017 at 11:42 PM, Jason Gustafson wrote: > Thanks Apurva, > > On

Build failed in Jenkins: kafka-0.11.0-jdk7 #288

See Changes: [damian.guy] KAFKA-5787; StoreChangelogReader needs to restore partitions that were -- [...truncated 2.44 MB...]

Re: [DISCUSS] 0.11.0.1 bug fix release

We still have 2 outstanding issues to close. They both have patches available and will hopefully be completed shortly. Thanks, Damian On Thu, 24 Aug 2017 at 10:05 Damian Guy wrote: > A quick update. There are 2 remaining issues, that both have patches > available.

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

Sounds good to me too. Since this is a non controversial change, I suggest starting the vote in 1-2 days if no-one else comments. Ismael On Thu, Aug 24, 2017 at 7:32 PM, Jason Gustafson wrote: > Seems reasonable. I don't recall any specific reason for not providing this >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Rajini, There are a few other metrics that could potentially be useful. I'd be interested in what you and the community thinks: 1. The KIP currently includes `FetchDownConversionsPerSec`, which is useful. In the common case, one would want to avoid down conversion by using the lower message

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

+1 (non-binding) On Tue, Aug 29, 2017 at 2:43 PM, Stevo Slavić wrote: > +1 (non-binding) > > On Tue, Aug 29, 2017 at 11:09 AM, Ismael Juma wrote: > > > Thanks for the KIP, +1 (binding) from me. > > > > Ismael > > > > On Thu, Aug 24, 2017 at 6:48 PM, Rajini

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

+1 (non-binding) On Tue, Aug 29, 2017 at 11:09 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Thu, Aug 24, 2017 at 6:48 PM, Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to start the vote on KIP-187

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Thanks for the KIP, +1 (binding) from me. Ismael On Thu, Aug 24, 2017 at 6:29 PM, Rajini Sivaram wrote: > Hi all, > > I would like to start vote on KIP-152 to improve diagnostics of > authentication failures and to update clients to treat authentication > failures as

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Thanks for the KIP, +1 (binding) from me. Ismael On Thu, Aug 24, 2017 at 6:48 PM, Rajini Sivaram wrote: > Hi all, > > I would like to start the vote on KIP-187 that adds a cumulative count > metric associated with each Kafka rate metric to improve downstream >

[GitHub] kafka pull request #3747: KAFKA-5787: StoreChangelogReader needs to restore ...

Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/3747 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is