Re: Review Request 35791: Patch for KAFKA-2298

method? Dong Lin wrote: I have thought about it as well. But probabaly no. Because in Selector.send() we put failed destinationId is put in failedSends rather than disconnected. The reason we use failedSends is because send() and poll() in Selector will be called asynchronously

Re: Review Request 35791: Patch for KAFKA-2298

4aee214b24fd990be003adc36d675f015bf22fe6 Diff: https://reviews.apache.org/r/35791/diff/ Testing --- Thanks, Dong Lin

Re: Guozhang Wang elected to Kafka PMC

Congrats Guozhang! Dong On Tue, Jun 16, 2015 at 9:57 AM, Timothy Chen tnac...@gmail.com wrote: Congrats Guozhang!! Tim On Tue, Jun 16, 2015 at 8:19 AM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: Congrats Guozhang! From: Ashish

Review Request 35540: Patch for KAFKA-2278

/ Testing --- Thanks, Dong Lin

Re: Review Request 34170: Patch for KAFKA-2191

://reviews.apache.org/r/34170/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 33049: Patch for KAFKA-2084

HandlerProcessor in package handler cannot be accessed in package com.sun.xml.internal.ws.handler import com.sun.xml.internal.ws.handler.HandlerProcessor.RequestOrResponse - Dong Lin On May 26, 2015, 6:53 p.m., Aditya Auradkar wrote

Re: Review Request 34170: Patch for KAFKA-2191

://reviews.apache.org/r/34418/diff/. Thanks, Dong Lin

Re: Review Request 33049: Patch for KAFKA-2084

On May 28, 2015, 12:58 p.m., Manikumar Reddy O wrote: core/src/main/scala/kafka/server/ClientQuotaMetrics.scala, line 145 https://reviews.apache.org/r/33049/diff/10/?file=972066#file972066line145 Are we using clientID as uniqueKey?. But as of now, clientID is not mandatory and it

Re: Review Request 36034: Patch for KAFKA-2306

-mail. To reply, visit: https://reviews.apache.org/r/36034/#review90303 --- On June 30, 2015, 4:04 a.m., Dong Lin wrote: --- This is an automatically generated e-mail. To reply

Re: Review Request 36034: Patch for KAFKA-2306

5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 0baf16e55046a2f49f6431e01d52c323c95eddf0 Diff: https://reviews.apache.org/r/36034/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 36034: Patch for KAFKA-2306

-exhausted with a different name. I prefer the first option since I feel in practice people just want to distinguish between the case that messages failed to get into the producer from the case the messages gets failed to send to the broker. Dong Lin wrote: Thanks for your comments. If I

Re: Review Request 36034: Patch for KAFKA-2306

-exhausted with a different name. I prefer the first option since I feel in practice people just want to distinguish between the case that messages failed to get into the producer from the case the messages gets failed to send to the broker. Dong Lin wrote: Thanks for your comments. If I

Re: Review Request 34418: Patch for KAFKA-2191

with both approaches. - Dong Lin On May 19, 2015, 5:12 p.m., Jay Kreps wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34418

Re: Review Request 36034: Patch for KAFKA-2306

5671a3fbeea8cb9a9ffeeb41aa1b132b92c0cae8 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 87dbd64f30f35dbf31d3820f9819a63c6c0d1e58 Diff: https://reviews.apache.org/r/36034/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 36034: Patch for KAFKA-2306

. To reply, visit: https://reviews.apache.org/r/36034/#review90730 --- On July 7, 2015, 5:47 p.m., Dong Lin wrote: --- This is an automatically generated e-mail. To reply, visit: https

Review Request 36034: Patch for KAFKA-2306

/kafka/clients/producer/internals/Sender.java 0baf16e55046a2f49f6431e01d52c323c95eddf0 Diff: https://reviews.apache.org/r/36034/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 33620: Patch for KAFKA-1690

, should it be removed? core/src/test/scala/unit/kafka/utils/TestUtils.scala (line 919) https://reviews.apache.org/r/33620/#comment147739 Should clientCert be removed since it is not used in the function? - Dong Lin On July 25, 2015, 7:11 p.m., Sriharsha Chintalapani wrote

Re: Review Request 36722: Patch for KAFKA-2332

/SyncProducer.scala dcee50113b1b1e062a56ab0f63ac6bb175be6b75 Diff: https://reviews.apache.org/r/36722/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 36722: Patch for KAFKA-2332

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36722/#review93833 --- On Aug. 4, 2015, 1:23 a.m., Dong Lin wrote

Re: Review Request 36722: Patch for KAFKA-2332

dcee50113b1b1e062a56ab0f63ac6bb175be6b75 Diff: https://reviews.apache.org/r/36722/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 33620: Patch for KAFKA-1690

= clientkeystore.jks was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) Can we avoid this warning to avoid confusion to user? - Dong Lin On July 25, 2015, 7:11 p.m., Sriharsha Chintalapani wrote

Re: Review Request 33620: Patch for KAFKA-1690

On July 30, 2015, 10:37 p.m., Dong Lin wrote: clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java, line 416 https://reviews.apache.org/r/33620/diff/13/?file=1021979#file1021979line416 Do you mean unwrapResult.getHandshakeStatus

Re: Review Request 33620: Patch for KAFKA-1690

/SSLTransportLayer.java (line 481) https://reviews.apache.org/r/33620/#comment148100 Similarly, maybe break out of loop if read = 0. Otherwise we might have infinite loop here. - Dong Lin On July 25, 2015, 7:11 p.m., Sriharsha Chintalapani wrote

Re: Review Request 36722: Patch for KAFKA-2332

, producerRequestStats.getProducerRequestStats(config.host, config.port).requestSizeHist.update(requestSize), will be problematic. On July 23, 2015, 5:56 p.m., Dong Lin wrote: What if we simply duplicate the sensors we are adding in 2136 for the new producer and consumer? Hi Adi, thanks for your comments! The sensors added

Re: Review Request 33620: Patch for KAFKA-1690

/AbstractConfig.java (line 108) https://reviews.apache.org/r/33620/#comment147469 Do you mean return copy? - Dong Lin On July 25, 2015, 7:11 p.m., Sriharsha Chintalapani wrote: --- This is an automatically generated e-mail. To reply, visit

Re: [jira] [Created] (KAFKA-1690) new java producer needs ssl support as a client

Cool. Thank Sriharsha. On Saturday, July 25, 2015, Sriharsha Chintalapani (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641752#comment-14641752 ] Sriharsha

Review Request 36722: Patch for KAFKA-2332

/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 34965: Patch for KAFKA-2241

core/src/main/scala/kafka/server/AbstractFetcherThread.scala 83fc47417dd7fe4edf030217fa7fd69d99b170b0 Diff: https://reviews.apache.org/r/34965/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 34965: Patch for KAFKA-2241

/main/scala/kafka/server/AbstractFetcherThread.scala 83fc47417dd7fe4edf030217fa7fd69d99b170b0 Diff: https://reviews.apache.org/r/34965/diff/ Testing --- Thanks, Dong Lin

Re: [VOTE] KIP-38: ZooKeeper authentication

+1 On Wed, Oct 21, 2015 at 5:01 PM, Jay Kreps wrote: > +1 > > -Jay > > On Wed, Oct 21, 2015 at 8:17 AM, Flavio Junqueira wrote: > > > Thanks everyone for the feedback so far. At this point, I'd like to start > > a vote for KIP-38. > > > > Summary: Add

Re: Kafka Monitoring Framework

Hey Lin, Thanks for your interest. As Kartik mentioned in the blog post, LinkedIn Kafka team is developing a Kafka monitoring framework to help detect problem that may only be found, for example, when you run server/client with different versions for a long time using production traffic. We hope

Re: Kafka Monitoring Framework

Lin: Yes, I think so. On Wed, Oct 14, 2015 at 1:48 PM, Lin Ma <lin...@huawei.com> wrote: > Thanks Dong. Then I bet it will take a while before its release. > > Lin > > > -Original Message- > From: Dong Lin [mailto:lindon...@gmail.com] > Sent: Tuesday, Octob

Re: [DISCUSS] Email to dev list for GitHub PR comments

I agree with Guozhang. On Tue, Oct 6, 2015 at 5:49 PM, Gwen Shapira wrote: > Agree with Guozhang. > > On Tue, Oct 6, 2015 at 3:22 PM, Guozhang Wang wrote: > > > I think github cannot batch comments in emails (yet?), which is sad.. > > > > I would prefer

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I agree with Jason we should aim to have clients to properly detect and handle unknown-protocol error. The hack of empty response should be shorten solution. After reading the earlier discussion and wiki, I have two questions: - In the future, do we expect all clients to send

Re: Kafka Mirror to consume data from beginning of topics

Yes, you can have your mirror maker reading from earliest offset by specifying "auto.offset.reset=smallest" in your consumer configuration file which is provided to mirror maker. On Thu, Oct 8, 2015 at 5:49 PM, Clelio De Souza wrote: > Hi there, > > I am trying to setup a

Re: [VOTE] KIP-31 - Move to relative offsets in compressed message sets.

+1 Dong On Tue, Oct 6, 2015 at 2:58 PM, Jiangjie Qin wrote: > Hi folks, > > Sorry for this prolonged voting session and thanks for the votes. > > There is an additional broker configuration change added to the KIP after > the vote. We propose to add a

Re: Review Request 34965: Patch for KAFKA-2241

c16f7edd322709060e54c77eb505c44cbd77a4ec core/src/main/scala/kafka/server/AbstractFetcherThread.scala 83fc47417dd7fe4edf030217fa7fd69d99b170b0 Diff: https://reviews.apache.org/r/34965/diff/ Testing --- Thanks, Dong Lin

Re: Review Request 34965: Patch for KAFKA-2241

://reviews.apache.org/r/34965/#review91159 --- On July 9, 2015, 10:35 p.m., Dong Lin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34965

Re: Review Request 34965: Patch for KAFKA-2241

it would be mostly redundant). However this is still not ideal, since it is a caveat that the user of the (public) forceClose API needs to be aware of. Dong Lin wrote: I agree. I have updated the code and comments to hopefully avoid any confusion to user. Joel Koshy wrote: Would

Re: 0.9.0.0 remaining jiras

Hey Jun, Should we also include https://issues.apache.org/jira/browse/KAFKA-2390 in 0.9.0? Becket told me that this is one of those patches ( https://issues.apache.org/jira/browse/KAFKA-2387) needed for new consumer API. Thanks, Dong On Tue, Sep 15, 2015 at 11:01 PM, Jason Rosenberg

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

Congratulations Sriharsha! Dong On Tue, Sep 22, 2015 at 4:17 AM, Guozhang Wang wrote: > Congrats Sriharsha! > > Guozhang > > On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao wrote: > > > I am pleased to announce that the Apache Kafka PMC has voted to > > invite

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Anna, Have you talked with Jiangjie Qin before initiating the vote? Jiangjie Qin is on vocation now and he may not have time to answer question related to KIP-32 in detail. Dong On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > Hi, > > I am opening the voting

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

-date > with all the decisions. Is there a particular reason why the KIP process > should wait? > > > On Tue, Dec 22, 2015 at 2:15 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hi Anna, > > > > Have you talked with Jiangjie Qin before initiating the vote? Jiangjie

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) On Fri, Feb 19, 2016 at 12:52 PM, Ismael Juma wrote: > +1 (non-binding) > > Ismael > > On Fri, Feb 19, 2016 at 7:00 PM, Becket Qin wrote: > > > Hi All, > > > > We would like to start this voting thread on making next Kafka release > >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Colin, Thanks much for the comment. Please see my reply inline. On Wed, Feb 1, 2017 at 1:54 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Feb 1, 2017, at 11:35, Dong Lin wrote: > > Hey Grant, Colin, > > > > My bad, I misunderstood Grant's s

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Sorry for the typo. I mean that before the KIP meeting, please free feel to provide comment in this email thread so that discussion in the KIP meeting can be more efficient. On Wed, Feb 1, 2017 at 6:53 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Eno, Colin, > > Would you

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

seems > >> better to put this field in the Session class to avoid changing > interface > >> that needs to be implemented by custom principal. > >> -> Doing this might be backwards incompatible as we need to > >> preserve the existing behavior of kafka-

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

no concern the KIP after the KIP meeting. In the meeting time, please feel free to provide comment in the thread so that discussion in the KIP meeting can be more efficient. Thanks, Dong On Wed, Feb 1, 2017 at 5:43 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Colin, > > Thanks much

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

> > > On 2 Feb 2017, at 02:53, Dong Lin <lindon...@gmail.com> wrote: > > > > Hey Eno, Colin, > > > > Would you have time next Tuesday morning to discuss the KIP? How about > 10 - > > 11 am? > > > > To make best use of our time, can you ple

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

org> wrote: > On Thu, Feb 2, 2017, at 17:54, Dong Lin wrote: > > Hey Colin, > > > > Thanks for the KIP. I have a few comments below: > > > > - I share similar view with Ismael that a Future-based API is better. > > PurgeDataFrom() is an example API that u

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

mplexity of administration with more > running instances. > > is anyone running kafka with anywhere near 100GB heaps? i thought the point > was to rely on kernel page cache to do the disk buffering > > On Thu, Jan 26, 2017 at 11:00 AM, Dong Lin <lindon...@gmail.com> wrot

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

ion of > returning > > a future from purgeDataFrom(). We can keep it that way. > > > > Thanks, > > > > Jun > > > > On Mon, Jan 23, 2017 at 4:24 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hi all, > > > > &g

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

This thread was been closed on Jan 18. We had more discussion after Guozhang's feedback on Jan 21. But no major change was made to the KIP after the discussion. On Tue, Jan 31, 2017 at 5:47 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Apurva, > > I think the KIP

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

ing (as I understand it). > > Disks are not the only resource on a machine, there are several instances > where multiple NICs are used for example. Do we want fine grained > management of all these resources? I'd argue that opens us the system to a > lot of complexity. > > Th

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

ead of ZK. Should we have a separate KIP in the future to migrate all existing notification to using RPC? > > Jun > > > On Wed, Jan 25, 2017 at 1:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Colin, > > > > Good point! Yeah we have actually considere

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

inlined > below. > > On Mon, Feb 6, 2017 at 7:22 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks for the review! Please see reply inline. > > > > On Mon, Feb 6, 2017 at 6:21 PM, Jun Rao <j...@confluent.io> wrote: > >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

admin can pool a few > > disks together to create a volume/directory and give that to Kafka. > > > > > > The kernel of my question will be that the admin already has tools to 1) > > create volumes/directories from a JBOD and 2) start a broker on a desired > > machine and 3

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

another client. Also, the > NetworkClient is an internal class which is not really meant for users. Do > we really want to open that up? Is the only benefit saving the number of > connections? Seems not worth it in my opinion. > > -Jason > > On Wed, Feb 8, 2017 at 6:43 PM,

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

Hey Colin, Thanks for updating the KIP. I have two followup questions: - It seems that setCreationConfig(...) is a bit redundant given that most arguments (e.g. topic name, partition num) are already passed to TopicsContext.create(...) when user creates topic. Should we pass the creationConfig

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

BTW, the idea to share NetworkClient is suggested by Radai and I like this idea. On Wed, Feb 8, 2017 at 6:39 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Colin, > > Thanks for updating the KIP. I have two followup questions: > > - It seems that setCreationConfig(...) is

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

Hey Colin, Thanks for the KIP. I have a few comments below: - I share similar view with Ismael that a Future-based API is better. PurgeDataFrom() is an example API that user may want to do it asynchronously even though there is only one request in flight at a time. In the future we may also have

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

On Tue, Feb 7, 2017 at 5:23 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Eno, > > Thanks much for the comment! > > I still think the complexity added to Kafka is justified by its benefit. > Let me provide my reasons below. > > 1) The additional logic is easy t

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hey Jorge, Thanks for the KIP. I have some quick comments: - Should we allow user to use wildcard to reset offset of all groups for a given topic as well? - Should we allow user to specify timestamp per topic partition in the json file as well? - Should the script take some credential file to

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

her the complexity added to Kafka is justified. > Operationally it seems to me an admin will still have to do all the three > items above. > > Looking forward to the discussion > Thanks > Eno > > > > On 1 Feb 2017, at 17:21, Dong Lin <lindon...@gmail.com> wrote: >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

w people using min.isr to keep > their data safe and the cluster operators would see a shrink in many ISRs > and hopefully an obvious log message leading to a quick fix. I haven't > thought through this idea in depth though. So there could be some > shortfalls. > > Thanks, >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

e code against disk errors. Formerly it was OK to just crash on a > > disk error; now it is not. It would be nice to see more in the test > > plan about injecting IOExceptions into disk handling code and verifying > > that we can handle it correctly. > > > > regards, &

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

ir level. > > 4. When broker had one of the dir failed, it can modify its " > /brokers/ids/[brokerId]" registry and remove the dir id, controller already > listening on this path can then be notified and run the replica assignment > accordingly where replica id is computed as

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

t have a partition that > only have one of the watermarks in case of a failure in between writing two > files. > > Guozhang > > On Sun, Jan 22, 2017 at 12:03 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Guozhang, > > > > Thanks for the revi

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

onding before the vote is called. Just wanted to point out > for the record that this approach may have some operational scenarios where > one of the replication files is missing and we need to treat them > specifically. > > > Guozhang > > > On Sun, Jan 22, 2017 at

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks. Please see my comment inline. On Mon, Jan 23, 2017 at 6:45 AM, Alexey Ozeritsky <aozerit...@yandex.ru> wrote: > > > 13.01.2017, 22:29, "Dong Lin" <lindon...@gmail.com>: > > Hey Alexey, > > > > Thanks for your review and the alternative app

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

rg/confluence/pages/diffpagesbyversion.action?pageId=67636826=13=14>. Please let me know if you have any concern with this change. Thanks, Dong On Mon, Jan 23, 2017 at 11:20 AM, Dong Lin <lindon...@gmail.com> wrote: > Thanks for the comment Jun. > > Yeah, I think there is us

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

3:30 AM, Alexey Ozeritsky <aozerit...@yandex.ru> wrote: > > > 23.01.2017, 22:11, "Dong Lin" <lindon...@gmail.com>: > > Thanks. Please see my comment inline. > > > > On Mon, Jan 23, 2017 at 6:45 AM, Alexey Ozeritsky <aozerit...@yandex.ru> > > w

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Colin, Thanks much for the comment. Please see me comment inline. On Thu, Jan 26, 2017 at 10:15 AM, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Jan 25, 2017, at 13:50, Dong Lin wrote: > > Hey Colin, > > > > Good point! Yeah we have actually consider

Re: [VOTE] KIP-115: Enforce offsets.topic.replication.factor

+1 On Wed, Jan 25, 2017 at 4:37 PM, Ismael Juma wrote: > +1 (binding) > > Ismael > > On Thu, Jan 26, 2017 at 12:34 AM, Onur Karaman < > onurkaraman.apa...@gmail.com > > wrote: > > > I'd like to start the vote for KIP-115: Enforce > > offsets.topic.replication.factor > > > >

Re: [DISCUSS] KIP-115: Enforce offsets.topic.replication.factor

+1 On Wed, Jan 25, 2017 at 4:22 PM, Ismael Juma wrote: > An important question is if this needs to wait for a major release or not. > > Ismael > > On Thu, Jan 26, 2017 at 12:19 AM, Ismael Juma wrote: > > > +1 from me too. > > > > Ismael > > > > On Thu, Jan

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

ed in the "alternate designs" design, even if you end > up deciding it's not the way to go. > > best, > Colin > > > On Thu, Jan 12, 2017, at 10:46, Dong Lin wrote: > > Hi all, > > > > We created KIP-112: Handle disk failure for JBOD. Please find the KIP

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

will not be affected. > > > Guozhang > > > > On Wed, Jan 18, 2017 at 6:12 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Thanks to everyone who voted and provided feedback! > > > > This KIP is now adopted with 3 binding +1s (Jun, Joel, Becket) an

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hey Mayuresh, Thanks for the KIP. I actually like the suggestions by Ismael and Jun. Here are my comments: 1. I am not sure we need to add the method buildPrincipal(Map principalConfigs). It seems that user can simply do principalBuilder.configure(...).buildPrincipal(...) without

Re: [DISCUSS] KIP-124: Request rate quotas

> As of the request rate quota, while it seems easy to enforce and > intuitive, > > there are some caveats. > > 1. Users do not have direct control over the request rate, i.e. users do > > not known when a request will be sent by the clients. > > 2. Each request may requir

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, Could you please let me know if the solutions above could address your concern? I really want to move the discussion forward. Thanks, Dong On Tue, Feb 14, 2017 at 8:17 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > > Thanks for all your help and time to discuss

Re: [DISCUSS] KIP-124: Request rate quotas

I couldn't find information in the KIP on > where this window would be configured. > > On Fri, Feb 17, 2017 at 9:45 AM, Dong Lin <lindon...@gmail.com> wrote: > > > To correct the typo above: It seems to me that determination of request > > rate is not any more diffic

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Onur, Thanks for the well-written KIP! I have two questions below. 1) In the process of migrating from OZKCCs and MDZKCCs to MEZKCCs, we will may a mix of OZKCCs, MDZKCCs and MEZKCCs. OZKCC and MDZKCC will only commit to zookeeper and MDZKCC will use kafka-based offset storage. Would we lose

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Rajini, Thanks for the KIP. I have some questions: - I am wondering why throttling based on request rate is listed as a rejected alternative. Can you provide more specific reason why it is difficult for administrators to decide request rates to allocate? It seems to me that determination of

Re: [DISCUSS] KIP-124: Request rate quotas

To correct the typo above: It seems to me that determination of request rate is not any more difficult than determination of *byte* rate as both metrics are commonly used to measure performance and provide guarantee to user. On Fri, Feb 17, 2017 at 9:40 AM, Dong Lin <lindon...@gmail.com>

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

. Thanks, Dong On Thu, Feb 23, 2017 at 6:46 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Dong, > > My replies are inlined below. > > On Thu, Feb 23, 2017 at 4:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks for you reply! Le

Re: [DISCUSS] KIP-124: Request rate quotas

be an important change depending on the answer to 1) above. We probably need to document this more explicitly. Dong On Thu, Feb 23, 2017 at 10:56 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > > Yeah you are right. I thought it wasn't because at LinkedIn it will be too

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

ZKCC because MEZKCC has > > "dual.commit.enabled" set to true as well as "offsets.storage" set to > > kafka. The combination of these configs results in the consumer fetching > > offsets from both kafka and zookeeper and just picking the greater of the > > two. >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

But controller still needs to learn about offline replicas from LeaderAndIsrResponse. I think this is better than the current design. Do you have any concern with this design? Thanks, Dong On Thu, Feb 23, 2017 at 7:12 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > > Sure, h

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

nt. Well, if all log directories are available, the failed log > directory path will be cleared. In the rarer case that a log directory is > still offline and one of the replicas registered in the failed log > directory shows up in another available log directory, I am not quite sure. &g

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, I don't think we should allow failed replicas to be re-created on the good disks. Say there are 2 disks and each of them is 51% loaded. If any disk fail, and we allow replicas to be re-created on the other disks, both disks will fail. Alternatively we can disable replica creation if

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

caRequest > to any offline replica. > > Thanks, > > Jun > > > On Tue, Feb 21, 2017 at 2:37 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks much for your comments. > > > > I actually proposed the design to store

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

+1 (non-binding) On Wed, Feb 22, 2017 at 10:52 PM, Manikumar wrote: > +1 (non-binding) > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Jun, > > > > Thanks a lot for the comments and reviews. > > I agree we should

Re: [DISCUSS] KIP-124: Request rate quotas

> So there will be no reuse of existing metrics/sensors. The new ones > for > > > > request processing time based throttling will be completely > independent > > > of > > > > existing metrics/sensors, but will be consistent in format. > > > > > >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

the broker detects this during broker startup, it can probably > just log an error and exit. The admin can remove the redundant partitions > manually and then restart the broker. > > Thanks, > > Jun > > On Sat, Feb 18, 2017 at 9:31 PM, Dong Lin <lindon...@gmail.com> wrote: &g

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, Motivated by your suggestion, I think we can also store the information of created replicas in per-broker znode at /brokers/created_replicas/ids/[id]. Does this sound good? Regards, Dong On Tue, Feb 21, 2017 at 2:37 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > &

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

not sure that will be the case. This is at least a concern where MM is mirroring traffic for only a few partitions of high byte-in rate. Thus I am wondering if we should do the optimization proposed above. Thanks, Dong On Wed, Feb 22, 2017 at 6:39 PM, Dong Lin <lindon...@gmail.com> wrote: > H

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hey Becket, Thanks for the KIP. I have one question here. Suppose producer's batch.size=100 KB, max.in.flight.requests.per.connection=1. Since each ProduceRequest contains one batch per partition, it means that 100 KB compressed data will be produced per partition per round-trip time as of

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Rajini, I think it makes a lot of sense to use io_thread_units as metric to quota user's traffic here. LGTM overall. I have some questions regarding sensors. - Can you be more specific in the KIP what sensors will be added? For example, it will be useful to specify the name and attributes of

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

iles. Clearly, log dirs that are completely > inaccessible will still be considered bad after a broker process bounce. > > best, > Colin > > > > > +1 (non-binding) aside from that > > > > > > > > On Wed, Feb 8, 2017, at 00:47, Dong Lin wrote: >

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

. offsetsForTimes) which are typically used for operation on multiple partitions at a time. On Fri, Feb 10, 2017 at 5:05 PM, Dong Lin <lindon...@gmail.com> wrote: > Hi Jun, > > Currently KIP-107 uses this API: > > Future<Map<TopicPartition, PurgeDataResult>> > purgeD

<    1   2   3   4   5   6   7   8   9   10   >