Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Thanks Ismael, AFAICT it is already a separate thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3ccajiktexrb1hbojrm7tmy1hdm_+t+psrz_6pawhnlbokp46o...@mail.gmail.com%3e On Wed, 19 Jul 2017 at 23:24 Guozhang Wang wrote: > Per 1. I suggested exposing the

Re: Consumer blocked when auth failed.

What log level do you have configured? You might bump up to DEBUG or TRACE and see if anything stands out. -Jason On Tue, Jul 18, 2017 at 7:59 PM, 陈江枫 wrote: > Hi, > > I was integrating Kafka with Spark, using DirectStream, when my > authentication fail, the stream just

[jira] [Created] (KAFKA-5615) Consumer hang in poll method while rebalancing is in progress (Duplicate of)

Geoffrey Stewart created KAFKA-5615: --- Summary: Consumer hang in poll method while rebalancing is in progress (Duplicate of) Key: KAFKA-5615 URL: https://issues.apache.org/jira/browse/KAFKA-5615

[GitHub] kafka pull request #3551: MINOR: added logging feature for ConfigEntry

GitHub user ppatierno opened a pull request: https://github.com/apache/kafka/pull/3551 MINOR: added logging feature for ConfigEntry Added toString() ovveride to ConfigEntry for logging You can merge this pull request into a Git repository by running: $ git pull

Jenkins build is back to normal : kafka-0.10.2-jdk7 #183

See

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

OK, sounds good. Let's just make sure we note this in the upgrade notes. Ismael On Wed, Jul 19, 2017 at 11:57 AM, Jason Gustafson wrote: > Ismael, I debated that also, but the main point was to make users aware of > the rebalance latency (with KIP-134 in mind). I'm guessing

[jira] [Created] (KAFKA-5616) can not perform a rolling upgrade from a non-secure to a secure Kafka cluster

zhu fangbo created KAFKA-5616: - Summary: can not perform a rolling upgrade from a non-secure to a secure Kafka cluster Key: KAFKA-5616 URL: https://issues.apache.org/jira/browse/KAFKA-5616 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk8 #1825

See Changes: [ismael] MINOR: Clarify wording in server.properties comment -- [...truncated 3.80 MB...] org.apache.kafka.common.config.ConfigDefTest >

[GitHub] kafka pull request #3551: MINOR: added logging feature for ConfigEntry

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3551 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Ismael, I've been working on the progress reporting assuming that it would be acceptable for the ReassignPartitionsCommand to poll the AdminClient API (and in turn the AdminClient API to poll the broker) in order to report progress in an interactive way. The alternative would, of course, be

Jenkins build is back to normal : kafka-trunk-jdk8 #1826

See

Re: [DISCUSS] KIP-176: Remove deprecated new-consumer option from tools

Hi all, I have updated the KIP-176 in order to consider to remove the --new-consumer option not only for ConsoleConsumer but even for other tools which use it : ConsumerPerformance

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Yes, I know it is a separate thread in the archives, it's just that Gmail's algorithm decided to place it under the same thread. This is a recurring problem and it affects discoverability of vote threads for Gmail users (I assume there are many such users). Ismael On Thu, Jul 20, 2017 at 1:32

[GitHub] kafka pull request #3555: KAFKA-5619: Make --new-consumer option as deprecat...

GitHub user ppatierno opened a pull request: https://github.com/apache/kafka/pull/3555 KAFKA-5619: Make --new-consumer option as deprecated in all tools Added deprecation and warning message on --new-consumer option usage with ConsumerGroupCommand and ConsumerPerformance tools.

[GitHub] kafka pull request #3085: KAFKA-4830: Augment KStream.print() to allow users...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3085 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-5619) Make --new-consumer option as deprecated in all tools

Paolo Patierno created KAFKA-5619: - Summary: Make --new-consumer option as deprecated in all tools Key: KAFKA-5619 URL: https://issues.apache.org/jira/browse/KAFKA-5619 Project: Kafka Issue

[GitHub] kafka pull request #3552: MINOR: Simplify SensorAccess usage

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3552 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3554: KAFKA-5123 Refactor ZkUtils readData* methods

GitHub user baluchicken opened a pull request: https://github.com/apache/kafka/pull/3554 KAFKA-5123 Refactor ZkUtils readData* methods You can merge this pull request into a Git repository by running: $ git pull https://github.com/baluchicken/kafka-1 KAFKA-5123 Alternatively

Re: [VOTE] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

Thanks for the KIP, +1 (binding). On Thu, Jun 1, 2017 at 9:44 AM, Dong Lin wrote: > Hi all, > > Can you please vote for KIP-164? The KIP can be found at > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+ >

[jira] [Created] (KAFKA-5618) Kafka stream not receive any topic/partitions/records info

Yogesh BG created KAFKA-5618: Summary: Kafka stream not receive any topic/partitions/records info Key: KAFKA-5618 URL: https://issues.apache.org/jira/browse/KAFKA-5618 Project: Kafka Issue Type:

Re: [DISCUSS] 2017 October release planning and release version

+1 on 1.0! Are we also going to move to java 8? I also think we should drop the Unstable annotations completely. Cheers, Damian On Wed, 19 Jul 2017 at 21:36 Guozhang Wang wrote: > Hi Stevo, > > Just trying to add to what Ismael has already replied you: > > > >

Re: Reason why TopicFilter and related classes are deprecated

I found that the deprecation was added speaking about the KIP-109 which discuss the old consumer deprecation. So the TopicFilter is used in deprecated classes like OldConsumer, ConsumerConnector and so on

[GitHub] kafka pull request #3553: KAFKA-5608: Follow-up to fix potential NPE and cla...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3553 KAFKA-5608: Follow-up to fix potential NPE and clarify method name You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka

[GitHub] kafka pull request #3249: MINOR: Fix docs; client no longer uses yammer metr...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3249 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk7 #2536

See Changes: [ismael] MINOR: Clarify wording in server.properties comment -- [...truncated 971.21 KB...] kafka.server.SaslApiVersionsRequestTest >

[jira] [Created] (KAFKA-5617) Update to the list of third-party clients

Daniel Schierbeck created KAFKA-5617: Summary: Update to the list of third-party clients Key: KAFKA-5617 URL: https://issues.apache.org/jira/browse/KAFKA-5617 Project: Kafka Issue Type:

[GitHub] kafka pull request #3552: MINOR: Simplify SensorAccess usage

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3552 MINOR: Simplify SensorAccess usage I was investigating an exception in this code and found a few opportunities for making it clearer. I also added the `out` folder to `.gitignore` as

[GitHub] kafka pull request #3097: KAFKA-5134 Replace zkClient.getChildren method wit...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3097 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Jenkins build is back to normal : kafka-trunk-jdk7 #2537

See

Re: [VOTE] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

+1, Thanks for the KIP. On Thu, Jul 20, 2017 at 7:08 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > On Thu, Jun 1, 2017 at 9:44 AM, Dong Lin wrote: > > > Hi all, > > > > Can you please vote for KIP-164? The KIP can be found at > >

[GitHub] kafka pull request #3546: MINOR: Apply extra serialized rsync step to both p...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3546 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #2301: KAFKA-3856: Cleanup Kafka Stream builder API (KIP-...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2301 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Hi all, I posted "KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress" for discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-180%3A+Add+a+broker+metric+specifying+the+number+of+consumer+group+rebalances+in+progress Check it out. cheers,

Re: Reason why TopicFilter and related classes are deprecated

Hi Paolo, That's right, we deprecated those classes as they are mainly used by the old consumer. Sounds good to avoid using them in the TopicCommand written in Java. Ismael On Thu, Jul 20, 2017 at 3:33 AM, Paolo Patierno wrote: > I found that the deprecation was added

[GitHub] kafka pull request #3556: KAFKA-5620

GitHub user jcustenborder opened a pull request: https://github.com/apache/kafka/pull/3556 KAFKA-5620 Expose the ClassCastException as the cause for the SerializationException. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (KAFKA-5620) SerializationException in doSend() masks class cast exception

Jeremy Custenborder created KAFKA-5620: -- Summary: SerializationException in doSend() masks class cast exception Key: KAFKA-5620 URL: https://issues.apache.org/jira/browse/KAFKA-5620 Project:

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, Yes, a poll-based approach sounds good since reassignment can take a long time. I even think that it should be manual. That is, the user should run a command to ask for the status of the rebalance instead of the tool doing it automatically. Ismael On Thu, Jul 20, 2017 at 4:12 AM, Tom

[GitHub] kafka pull request #3489: KAFKA-5556 : KafkaConsumer.commitSync throws Illeg...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3489 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] 2017 October release planning and release version

That's a good question, from https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.0 it seems we do have one deprecation (KIP-109: Old Consumer Deprecation ). I'm fine with not removing the

[GitHub] kafka pull request #3460: KAFKA-5534: `offsetForTimes` result should include...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3460 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3367: KAFKA-5468 WorkerSourceTask offset commit loglevel...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3367 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] 2017 October release planning and release version

We deprecated the old Scala consumers and some Stream APIs that I can remember. I can't comment on the Streams APIs, but we don't intend to remove the old Scala consumers before the June 2018 release (we said we'd keep it for at least one year after deprecation). Ismael Ismael On 21 Jul 2017

[jira] [Resolved] (KAFKA-5623) ducktape kafka service: do not assume Service contains num_nodes

[ https://issues.apache.org/jira/browse/KAFKA-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5623. -- Resolution: Fixed Fix Version/s: 0.11.0.1 0.10.2.2

[GitHub] kafka pull request #3557: KAFKA-5623: ducktape kafka service: do not assume ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3557 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3269: KAFKA-5410: Fix taskClass() method name in Connect...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3269 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Kafka doc : official repo or web site repo ?

Hi Paolo, Docs are a little bit confusing. Some of the docs are version-specific and those live in the main repo and are managed by the same branches that releases are managed on. Then there is the https://github.com/apache/kafka-site which contains the actual contents of the site. During

Re: [DISCUSS] 2017 October release planning and release version

Ack on the deprecation, as long as we actually give people the window. I guess we're not doing a good job of communicating what that period is anyway, so we can play a bit fast and loose. -Ewen On Thu, Jul 20, 2017 at 4:59 PM, Guozhang Wang wrote: > We do have the KIP in

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Jason, Regarding your comment about the current limitation on the information returned for a consumer group, do you think it's worth expanding the API to return some additional info (e.g. generation id, group leader, ...)? Thanks. --Vahid From: Jason Gustafson To:

Re: [DISCUSS] 2017 October release planning and release version

Did we deprecate anything in 0.11.0? The one concern with bumping major versions in consecutive releases is that you may not give people the room for transition if you deprecate and then immediately remove in the next release. -Ewen On Thu, Jul 20, 2017 at 4:51 AM, Damian Guy

Re: [DISCUSS] 2017 October release planning and release version

We do have the KIP in the candidate list of the next release (see wiki https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2017.Oct) as KIP-118: Drop Support for Java 7 in Kafka 0.11 .

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Colin, Thanks for the writeup! This KIP lgtm. Guozhang On Thu, Jul 20, 2017 at 10:52 AM, Colin McCabe wrote: > Hi all, > > I posted "KIP-180: Add a broker metric specifying the number of consumer > group rebalances in progress" for discussion: > >

[GitHub] kafka pull request #2498: MINOR: Make Kafka Connect step in quickstart more ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2498 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Hi Matthias, Damian, all, Thanks for your comments and sorry for super-late update. Sure, the DSL refactoring is not blocking for this KIP. I made some changes to KIP document based on my prototype. Please feel free to comment. Cheers, Jeyhun On Fri, Jul 7, 2017 at 9:35 PM Matthias J. Sax

[GitHub] kafka pull request #3557: KAFKA-5623: ducktape kafka service: do not assume ...

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/3557 KAFKA-5623: ducktape kafka service: do not assume Service contains nu… …m_nodes You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (KAFKA-5623) ducktape kafka service: do not assume Service contains num_nodes

Colin P. McCabe created KAFKA-5623: -- Summary: ducktape kafka service: do not assume Service contains num_nodes Key: KAFKA-5623 URL: https://issues.apache.org/jira/browse/KAFKA-5623 Project: Kafka

Build failed in Jenkins: kafka-0.10.2-jdk7 #186

See Changes: [me] KAFKA-5623: ducktape kafka service: do not assume Service contains -- [...truncated 331.79 KB...] kafka.server.KafkaConfigTest >

[GitHub] kafka pull request #3525: KAFKA-5431: cleanSegments should not set length fo...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3525 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Consumer throughput drop

Hi Ovidu, The see-saw behavior is inevitable with linux when you have concurrent reads and writes. However, tuning the following two settings may help achieve more stable performance (from Jay's link): > *dirty_ratio*Defines a percentage value. Writeout of dirty data begins > (via *pdflush*)

Re: [DISCUSS] 2017 October release planning and release version

+1 on 1.0. It's about time :) On Thu, Jul 20, 2017 at 5:11 PM Ewen Cheslack-Postava wrote: > Ack on the deprecation, as long as we actually give people the window. I > guess we're not doing a good job of communicating what that period is > anyway, so we can play a bit fast and

[jira] [Resolved] (KAFKA-5431) LogCleaner stopped due to org.apache.kafka.common.errors.CorruptRecordException

[ https://issues.apache.org/jira/browse/KAFKA-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-5431. Resolution: Fixed Fix Version/s: 0.11.1.0 Issue resolved by pull request 3525

Jenkins build is back to normal : kafka-trunk-jdk7 #2545

See

[GitHub] kafka pull request #3487: MINOR: Fixed misleading reference to HTTPS instead...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3487 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3558: MINOR: Safer handling of requests prior to SASL au...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3558 MINOR: Safer handling of requests prior to SASL authentication This implements two improvements for request handling prior to SASL authentication: 1. Only parse request types that are

[GitHub] kafka pull request #3559: KAFKA-5535: Handle null values in ExtractField

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/3559 KAFKA-5535: Handle null values in ExtractField You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka kafka-5535-extract-field-null

Build failed in Jenkins: kafka-trunk-jdk7 #2547

See Changes: [me] KAFKA-5410; Fix taskClass() method name in Connector and flush() [me] MINOR: Fixed misleading reference to HTTPS instead of SSL support in the --

[jira] [Resolved] (KAFKA-5582) Log compaction with preallocation enabled does not trim segments

[ https://issues.apache.org/jira/browse/KAFKA-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxihx resolved KAFKA-5582. --- Resolution: Duplicate > Log compaction with preallocation enabled does not trim segments >

Re: Consumer throughput drop

I suspect this is on Linux right? The way Linux works is it uses a percent of memory to buffer new writes, at a certain point it thinks it has too much buffered data and it gives high priority to writing that out. The good news about this is that the writes are very linear, well layed out, and

Build failed in Jenkins: kafka-trunk-jdk8 #1829

See Changes: [jason] KAFKA-5556; Fix IllegalStateException in KafkaConsumer.commitSync due to -- [...truncated 4.74 MB...]

Re: Consumer throughput drop

Yes, I’m using Debian Jessie 2.6 installed on this hardware [1]. It is also my understanding that Kafka is based on system’s cache (Linux in this case) which is based on Clock-Pro for page replacement policy, doing complex things for general workloads. I will check the tuning parameters, but I

[jira] [Created] (KAFKA-5621) The producer should retry expired batches when retries are enabled

Apurva Mehta created KAFKA-5621: --- Summary: The producer should retry expired batches when retries are enabled Key: KAFKA-5621 URL: https://issues.apache.org/jira/browse/KAFKA-5621 Project: Kafka

Jenkins build is back to normal : kafka-0.10.0-jdk7 #211

See

Build failed in Jenkins: kafka-0.10.2-jdk7 #185

See -- [...truncated 165.91 KB...] kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable

Build failed in Jenkins: kafka-0.10.1-jdk7 #125

See -- [...truncated 589.03 KB...] org.apache.kafka.streams.integration.RegexSourceIntegrationTest > testNoMessagesSentExceptionFromOverlappingPatterns PASSED

Build failed in Jenkins: kafka-0.10.2-jdk7 #184

See Changes: [me] MINOR: Apply extra serialized rsync step to both parallel and serial -- [...truncated 327.69 KB...] kafka.server.KafkaConfigTest >

[GitHub] kafka pull request #3550: KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3550 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk7 #2544

See Changes: [jason] KAFKA-5610; WriteTxnMarker handler should return -- [...truncated 968.30 KB...] kafka.server.epoch.LeaderEpochFileCacheTest >

Jenkins build is back to normal : kafka-trunk-jdk8 #1830

See

[jira] [Created] (KAFKA-5622) Add support for JMXMP

Vijay Prakash created KAFKA-5622: Summary: Add support for JMXMP Key: KAFKA-5622 URL: https://issues.apache.org/jira/browse/KAFKA-5622 Project: Kafka Issue Type: New Feature