Re: [DISCUSS] KIP-82 - Add Record Headers

So on mutable, and headers just at message level seems we're all agreed then. On Radai's comments. 1) agreed - kip updated. 2) Now i totally get the nasty code having this would create as noted by your example. And obviously we want an API which means boiler plate code and most common

[jira] [Resolved] (KAFKA-1569) Tool for performance and correctness of transactions end-to-end

[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1569. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1604) System Test for Transaction Management

[ https://issues.apache.org/jira/browse/KAFKA-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1604. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1601) ConsoleConsumer/SimpleConsumerPerformance should be transaction-aware

[ https://issues.apache.org/jira/browse/KAFKA-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1601. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1541) Add transactional request definitions to clients package

[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1541. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1526) Producer performance tool should have an option to enable transactions

[ https://issues.apache.org/jira/browse/KAFKA-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1526. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1527) SimpleConsumer should be transaction-aware

[ https://issues.apache.org/jira/browse/KAFKA-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1527. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1525) DumpLogSegments should print transaction IDs

[ https://issues.apache.org/jira/browse/KAFKA-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1525. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1524) Implement transactional producer

[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1524. Resolution: Unresolved This work has been superseded by KIP-98:

[jira] [Resolved] (KAFKA-1523) Implement transaction manager module

[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-1523. Resolution: Unresolved This work has been superseded by KIP-98:

[GitHub] kafka pull request #2606: kafka4811: ReplicaFetchThread may fail to create d...

GitHub user amethystic opened a pull request: https://github.com/apache/kafka/pull/2606 kafka4811: ReplicaFetchThread may fail to create due to existing metric Have fetcherThreadMap keyed off brokerId + fetcherId instead of broker + fetcherId, but did not consider the case where

[jira] [Assigned] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[ https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxi reassigned KAFKA-4811: --- Assignee: huxi > ReplicaFetchThread may fail to create due to existing metric >

[jira] [Commented] (KAFKA-4738) Remove generic type of class ClientState

[ https://issues.apache.org/jira/browse/KAFKA-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887309#comment-15887309 ] Matthias J. Sax commented on KAFKA-4738: [~sharad.develop] Are you working on this? > Remove

[jira] [Commented] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[ https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887308#comment-15887308 ] huxi commented on KAFKA-4811: - [~junrao] what about the port change? Do we also need to consider this

[jira] [Commented] (KAFKA-4706) Unify StreamsKafkaClient instances

[ https://issues.apache.org/jira/browse/KAFKA-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887304#comment-15887304 ] Matthias J. Sax commented on KAFKA-4706: [~sharad.develop] You did close your PR. Are you still

[jira] [Commented] (KAFKA-4623) Change Default unclean.leader.election.enabled from True to False

[ https://issues.apache.org/jira/browse/KAFKA-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887302#comment-15887302 ] Matthias J. Sax commented on KAFKA-4623: [~sharad.develop]: [~ewencp] left already a comment on

[jira] [Commented] (KAFKA-4722) Add application.id to StreamThread name

[ https://issues.apache.org/jira/browse/KAFKA-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887301#comment-15887301 ] Matthias J. Sax commented on KAFKA-4722: [~sharad.develop] I left already a comment on your PR. It

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Hey Mickael, The suggestion to add something to Node makes sense. I could imagine for example adding a flag to indicate that the connection has a higher "priority," meaning that we can allocate outside of the memory pool if necessary. That would still be generic even if the only use case is the

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

+1 on 0.11.0.0. Can we also include KIP-54 to the list? The PR for this KIP is ready for review. Thanks. --Vahid From: Ismael Juma To: dev@kafka.apache.org Date: 02/27/2017 07:47 PM Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning Sent by:

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

+1 for 0.11.0.0 Guozhang On Mon, Feb 27, 2017 at 8:14 PM, Becket Qin wrote: > Hi Ismael, > > Thanks for volunteering on the new release. > > I think 0.11.0.0 makes a lot of sense given the new big features we are > intended to include. > > Thanks, > > Jiangjie (Becket)

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

Hi Jun and everyone, I would like to change the KIP in the following way. Currently, if any replica if offline, the purge result for a partition will be NotEnoughReplicasException and its low_watermark will be 0. The motivation for this approach is that we want to guarantee that the data before

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

+1 for major version bump. A good bit of deprecated code I would like to see removed especially on old consumer side plus a few other settings defaults changed such as the brief discussion on mirrormaker options a few months back. Just be good to continue to make the new user experience a lot

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

Hi Ismael, Thanks for volunteering on the new release. I think 0.11.0.0 makes a lot of sense given the new big features we are intended to include. Thanks, Jiangjie (Becket) Qin On Mon, Feb 27, 2017 at 7:47 PM, Ismael Juma wrote: > Hi all, > > With 0.10.2.0 out of the

[DISCUSS] 0.10.3.0/0.11.0.0 release planning

Hi all, With 0.10.2.0 out of the way, I would like to volunteer to be the release manager for our next time-based release. See https://cwiki.apache.org/c onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous communication on time-based releases or need a reminder. I put together

[jira] [Updated] (KAFKA-4586) Add purgeDataBefore() API in AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4586: --- Status: Patch Available (was: Open) > Add purgeDataBefore() API in AdminClient >

[jira] [Updated] (KAFKA-4586) Add purgeDataBefore() API in AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4586: --- Fix Version/s: 0.10.3.0 > Add purgeDataBefore() API in AdminClient >

[jira] [Updated] (KAFKA-3959) __consumer_offsets wrong number of replicas at startup

[ https://issues.apache.org/jira/browse/KAFKA-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3959: --- Resolution: Fixed Assignee: Onur Karaman (was: Ewen Cheslack-Postava) Status: Resolved

[jira] [Updated] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[ https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4811: --- Labels: newbie (was: ) > ReplicaFetchThread may fail to create due to existing metric >

[jira] [Updated] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[ https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4811: --- Component/s: replication > ReplicaFetchThread may fail to create due to existing metric >

[jira] [Commented] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[ https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887119#comment-15887119 ] Jun Rao commented on KAFKA-4811: In AbstractFetcherManager, we only create a new ReplicaFetcherThread when

Re: [VOTE] KIP-118: Drop Support for Java 7 in Kafka 0.11

Thanks to everyone who voted and provided feedback. +1 (binding) from me too. The vote has passed with 4 binding votes (Grant, Jason, Guozhang, Ismael) and 11 non-binding votes (Bill, Damian, Eno, Edoardo, Mickael, Bharat, Onur, Vahid, Colin, Apurva, Tom). There were no 0 or -1 votes. I have

[jira] [Created] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

Jun Rao created KAFKA-4811: -- Summary: ReplicaFetchThread may fail to create due to existing metric Key: KAFKA-4811 URL: https://issues.apache.org/jira/browse/KAFKA-4811 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi Jun, In addition to the Eno's reference of why rebuild time with RAID-5 is more expensive, another concern is that RAID-5 will fail if more than one disk fails. JBOD is still works with 1+ disk failure and has better performance with one disk failure. These seems like good argument for using

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

Yes, that was going to be my last resort attempt :) I'm going to give that a try today and see how it goes. Although it isn't that great of a solution, I don't mind doing it since the logic resides at one single place in our application. -Jaikiran On Tuesday, February 28, 2017, James Cheng

Re: [DISCUSS] KIP-82 - Add Record Headers

If I understand correctly, the suggestion is to let headers be mutable on the producer side basically until after they've passed through the interceptors. That sounds like a reasonable compromise to me. @Becket 3. It might be useful to have headers at MessageSet level as well so we can > avoid

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Breaking clients without a deprecation period is something we only do as a last resort. Is there strong justification for doing it here? Ismael On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Hi Ismael, > > Yeah. I agree that it might break the clients

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

> On Feb 26, 2017, at 10:36 PM, Jaikiran Pai wrote: > > An update on one of the workaround I tried - I added some logic in our > consumption part to wait for the KafkaConsumer.partitionsFor() to return the > topic name in the list, before actually considering the

[GitHub] kafka pull request #2605: Kafka 4738:Remove generic type of class ClientStat...

GitHub user sharad-develop opened a pull request: https://github.com/apache/kafka/pull/2605 Kafka 4738:Remove generic type of class ClientState Remove generic type of class ClientState. Also removed Generic type from TaskAssignor as it referred to same generic type of ClientState

[jira] [Created] (KAFKA-4810) SchemaBuilder should be more lax about checking that fields are unset if they are being set to the same value

Ewen Cheslack-Postava created KAFKA-4810: Summary: SchemaBuilder should be more lax about checking that fields are unset if they are being set to the same value Key: KAFKA-4810 URL:

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

> > Lets say we sent the batch over the wire and received a > RecordTooLargeException, how do we split it as once we add the message to > the batch we loose the message level granularity. We will have to > decompress, do deep iteration and split and again compress. right? This > looks like a

[jira] [Commented] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

[ https://issues.apache.org/jira/browse/KAFKA-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886778#comment-15886778 ] Guozhang Wang commented on KAFKA-4778: -- [~peoplemerge] What I was looking for is actually the

[GitHub] kafka-site pull request #48: Filled in og meta tags

GitHub user derrickdoo opened a pull request: https://github.com/apache/kafka-site/pull/48 Filled in og meta tags Set og meta tags so sharing links to places like LinkedIn, Facebook, Twitter gives people good defaults. You can merge this pull request into a Git repository by

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi Ismael, Yeah. I agree that it might break the clients if the user is using the kafkaPrincipal directly. But since KafkaPrincipal is also a Java Principal and I think, it would be a right thing to do replace the kafkaPrincipal with Java Principal at this stage than later. We can mention in the

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should set up /opt/kafka-dev to be the source directory

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Description: {{docker/run_tests.sh }} should be able to run

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should set up /opt/kafka-dev the same way Vagrant does

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Summary: docker/run_tests.sh should set up /opt/kafka-dev the same way Vagrant does (was:

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should set up /opt/kafka-dev to be the source directory

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Summary: docker/run_tests.sh should set up /opt/kafka-dev to be the source directory (was:

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should be able to run sanity_checks/test_verifiable_producer.py against version 0.8.2.2

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Summary: docker/run_tests.sh should be able to run sanity_checks/test_verifiable_producer.py

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should be able to run sanity_checks/test_verifiable_producer.py against version 0.8.2.2

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Description: {{docker/run_tests.sh }}should be able to run

[jira] [Commented] (KAFKA-4809) docker/run_tests.sh should be able to run sanity_checks/test_verifiable_producer.py against version 0.8.2.2

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886715#comment-15886715 ] Colin P. McCabe commented on KAFKA-4809: Thanks for the background, [~ewencp]. I guess the

[jira] [Updated] (KAFKA-4809) docker/run_tests.sh should be able to run sanity_checks/test_verifiable_producer.py against version 0.8.2.2

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Description: docker/run_tests.sh should be able to run

Jenkins build is back to normal : kafka-trunk-jdk8 #1306

See

[GitHub] kafka pull request #2604: KAFKA-4794: Add access to OffsetStorageReader from...

GitHub user fhussonnois opened a pull request: https://github.com/apache/kafka/pull/2604 KAFKA-4794: Add access to OffsetStorageReader from SourceConnector This a first attempt to implement Add access to OffsetStorageReader from Source Connector. I am not sure if I did it

[jira] [Commented] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

[ https://issues.apache.org/jira/browse/KAFKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886680#comment-15886680 ] ASF GitHub Bot commented on KAFKA-4794: --- GitHub user fhussonnois opened a pull request:

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

RAID-10's code is much simpler (just stripe plus mirror) and under failure the recovery is much faster since it just has to read from a mirror, not several disks to reconstruct the data. Of course, the price paid is that mirroring is more expensive in terms of storage space. E.g., see

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Eno, Thanks for the pointers. Doesn't RAID-10 have a similar issue during rebuild? In both cases, all data on existing disks have to be read during rebuild? RAID-10 seems to still be used widely. Jun On Mon, Feb 27, 2017 at 1:38 PM, Eno Thereska wrote: >

[jira] [Commented] (KAFKA-4809) sanity_checks/test_verifiable_producer.py should test 0.10.x and not 0.8

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886610#comment-15886610 ] Ewen Cheslack-Postava commented on KAFKA-4809: -- The reason we want to make sure it works

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Unfortunately RAID-5/6 is not typically advised anymore due to failure issues, as Dong mentions, e.g.: http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ Eno > On 27 Feb 2017, at 21:16, Jun Rao

Re: [DISCUSS] KIP-82 - Add Record Headers

a few comments on the KIP as it is now: 1. instead of add(Header) + add (Iterable) i suggest we use add + addAll. this is more in line with how java collections work and may therefor be more intuitive 2. common user code dealing with headers will want get("someKey") / set("someKey"), or

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Dong, For RAID5, I am not sure the rebuild cost is a big concern. If a disk fails, typically an admin has to bring down the broker, replace the failed disk with a new one, trigger the RAID rebuild, and bring up the broker. This way, there is no performance impact at runtime due to rebuild.

[jira] [Commented] (KAFKA-3990) Kafka New Producer may raise an OutOfMemoryError

[ https://issues.apache.org/jira/browse/KAFKA-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886466#comment-15886466 ] Adrian McCague commented on KAFKA-3990: --- For what it's worth witnessing this as well, the received

[GitHub] kafka pull request #2603: MINOR: Minor reduce unnecessary calls to time.mill...

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/2603 MINOR: Minor reduce unnecessary calls to time.millisecond (part 2) Avoid calling time.milliseconds more often than necessary. Cleaning and committing logic can use the timestamp at the start of

[jira] [Commented] (KAFKA-4809) sanity_checks/test_verifiable_producer.py should test 0.10.x and not 0.8

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886429#comment-15886429 ] ASF GitHub Bot commented on KAFKA-4809: --- GitHub user cmccabe opened a pull request:

[GitHub] kafka pull request #2602: KAFKA-4809: sanity_checks/test_verifiable_producer...

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/2602 KAFKA-4809: sanity_checks/test_verifiable_producer.py should test 0.1… …0.x and not 0.8 You can merge this pull request into a Git repository by running: $ git pull

[jira] [Updated] (KAFKA-4809) sanity_checks/test_verifiable_producer.py should test 0.10.x and not 0.8

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Summary: sanity_checks/test_verifiable_producer.py should test 0.10.x and not 0.8 (was:

[jira] [Created] (KAFKA-4809) sanity_checks/test_verifiable_producer.py should test 0.10.1 and 0.10.2 and not 0.8

Colin P. McCabe created KAFKA-4809: -- Summary: sanity_checks/test_verifiable_producer.py should test 0.10.1 and 0.10.2 and not 0.8 Key: KAFKA-4809 URL: https://issues.apache.org/jira/browse/KAFKA-4809

[jira] [Updated] (KAFKA-4809) sanity_checks/test_verifiable_producer.py should test 0.10.x and not 0.8

[ https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-4809: --- Description: {{sanity_checks/test_verifiable_producer.py}} should test the latest 0.10.0,

Kafka MirrorMaker issues

Hi Kafka Gurus :) I am facing issues with KafkaMirror, I am using Kafka 0.10.1.1 and trying to use mirroring to create backup of kafka logs or perhaps this might be a great idea to do it, please let me know if its not. my consumer.properties: bootstrap.servers=localhost:9092 group.id=mirror

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

Thanks Ewen, "use the corresponding serializer internally and just add in the extra conversion steps for the data API" sounds good to me. Guozhang On Mon, Feb 27, 2017 at 8:24 AM, Ewen Cheslack-Postava wrote: > It's a different interface that's being implemented. The

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886365#comment-15886365 ] Mayuresh Gharat commented on KAFKA-4808: [~ijuma] Sure we can provide a better error message if we

[jira] [Commented] (KAFKA-4744) Streams_bounce test failing occassionally

[ https://issues.apache.org/jira/browse/KAFKA-4744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886354#comment-15886354 ] ASF GitHub Bot commented on KAFKA-4744: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #2601: KAFKA-4744: Increased timeout for bounce test

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2601 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-4744) Streams_bounce test failing occassionally

[ https://issues.apache.org/jira/browse/KAFKA-4744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4744. -- Resolution: Fixed Issue resolved by pull request 2601

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hey Mayuresh, 1) The batch would be split when an RecordTooLargeException is received. 2) Not lower the actual compression ratio, but lower the estimated compression ratio "according to" the Actual Compression Ratio(ACR). An example, let's start with Estimated Compression Ratio (ECR) = 1.0. Say

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hi Becket, Thanks for the expatiation. Regarding : 1) The batch would be split when an RecordTooLargeException is received. Lets say we sent the batch over the wire and received a RecordTooLargeException, how do we split it as once we add the message to the batch we loose the message level

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi Mayuresh, Sorry for the delay. The updated KIP states that there is no compatibility impact, but that doesn't seem right. The fact that we changed the type of Session.principal to `Principal` means that any code that expects it to be `KafkaPrincipal` will break. Either because of declared

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886297#comment-15886297 ] Ismael Juma commented on KAFKA-4808: [~mgharat], I think a separate error code would be a better

[jira] [Assigned] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-4808: -- Assignee: Mayuresh Gharat > send of null key to a compacted topic should throw error

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886288#comment-15886288 ] Mayuresh Gharat commented on KAFKA-4808: [~ijuma] I was thinking if throwing "INVALID_REQUEST"

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi Ismael, Joel, Becket Would you mind taking a look at this. We require 2 more binding votes for the KIP to pass. Thanks, Mayuresh On Thu, Feb 23, 2017 at 10:57 AM, Dong Lin wrote: > +1 (non-binding) > > On Wed, Feb 22, 2017 at 10:52 PM, Manikumar

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hi Becket, Seems like an interesting idea. I had couple of questions : 1) How do we decide when the batch should be split? 2) What do you mean by slowly lowering the "actual" compression ratio? An example would really help here. Thanks, Mayuresh On Fri, Feb 24, 2017 at 3:17 PM, Becket Qin

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Apologies for the late response. Thanks Jason for the suggestion. Yes you are right, the Coordinator connection is "tagged" with a different id, so we could retrieve it in NetworkReceive to make the distinction. However, currently the coordinator connection are made different by using:

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

It's a different interface that's being implemented. The functionality is the same (since it's just a simple pass through), but we intentionally named Converters differently than Serializers since they do more work than Serializers (besides the normal serialization they also need to convert

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885759#comment-15885759 ] Ismael Juma commented on KAFKA-4808: cc [~norwood] > send of null key to a compacted topic should

[jira] [Created] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

Ismael Juma created KAFKA-4808: -- Summary: send of null key to a compacted topic should throw error back to user Key: KAFKA-4808 URL: https://issues.apache.org/jira/browse/KAFKA-4808 Project: Kafka

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885716#comment-15885716 ] Ismael Juma commented on KAFKA-4779: The test failed again, this time with a different message:

[jira] [Reopened] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4779: > Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py >

[GitHub] kafka pull request #2600: KAFKA-4806: Prevent double logging of ConsumerConf...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2600 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-4806) KafkaConsumer: ConsumerConfig gets logged twice.

[ https://issues.apache.org/jira/browse/KAFKA-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885638#comment-15885638 ] ASF GitHub Bot commented on KAFKA-4806: --- Github user asfgit closed the pull request at:

[jira] [Resolved] (KAFKA-4806) KafkaConsumer: ConsumerConfig gets logged twice.

[ https://issues.apache.org/jira/browse/KAFKA-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4806. Resolution: Fixed Fix Version/s: 0.10.3.0 Issue resolved by pull request 2600

[jira] [Comment Edited] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[ https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885611#comment-15885611 ] Ismael Juma edited comment on KAFKA-4569 at 2/27/17 11:27 AM: -- A few

[jira] [Comment Edited] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[ https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885611#comment-15885611 ] Ismael Juma edited comment on KAFKA-4569 at 2/27/17 11:28 AM: -- A few

[jira] [Commented] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[ https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885611#comment-15885611 ] Ismael Juma commented on KAFKA-4569: A few comments: 1. We are using a MockTime without auto-tick, so

[jira] [Updated] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[ https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4569: --- Fix Version/s: 0.10.2.1 > Transient failure in >

[jira] [Updated] (KAFKA-4806) KafkaConsumer: ConsumerConfig gets logged twice.

[ https://issues.apache.org/jira/browse/KAFKA-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Ebert updated KAFKA-4806: --- Fix Version/s: 0.10.2.1 > KafkaConsumer: ConsumerConfig gets logged twice. >

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

@Vahid: make sense to add "new lag" info IMO, I will update the KIP. @Becket: 1. About deleting, I think ConsumerGroupCommand already has an option to delete Group information by topic. From delete docs: "Pass in groups to delete topic partition offsets and ownership information over the entire