Re: Review Request 33065: Patch for KAFKA-1928

src/main/scala/kafka/network/SocketServer.scala <https://reviews.apache.org/r/33065/#comment135876> We probably can use the e @ pattern matching to handle both types of exceptions. - Jun Rao On May 20, 2015, 10:42 a.m., Gwen Shapira wrote: > >

Re: Review Request 34450: Fix KAFKA-2017; rebased

/kafka/utils/ZkUtils.scala <https://reviews.apache.org/r/34450/#comment135967> This doesn't seem to be used. - Jun Rao On May 20, 2015, 4:13 p.m., Guozhang Wang wrote: > > --- > This is an aut

Re: [VOTE] KIP-21 Dynamic Configuration

; > > > > > The downside of this approach is that we will have 2 separate ways of > > > changing topic configs (AlterTopic and AlterConfig). While a general > > > AlterConfig only makes sense if we plan to have more than two types of > > > entity configs..

Re: [Vote] KIP-11 Authorization design for kafka security

Parth, Thanks for driving this. Could you update the status of the KIP in the wiki? Thanks, Jun On Wed, May 20, 2015 at 2:37 PM, Parth Brahmbhatt < pbrahmbh...@hortonworks.com> wrote: > This vote is now Closed with 4 binding +1s and 4 non binding +1s. > > Thanks > Parth > > On 5/20/15, 12:04 P

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

ows > attached. The patch can help improve the consume performance around > 25%~50%. > > Thanks, Honghai Chen > > -Original Message- > From: Jun Rao [mailto:j...@confluent.io] > Sent: Wednesday, May 6, 2015 5:39 AM > To: dev@kafka.apache.org > Subject

Re: Review Request 33065: Patch for KAFKA-1928

> On May 21, 2015, 2:11 a.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/network/RequestChannel.scala, line 110 > > > > > > This would result in the following log entry: > > > > TRACE Completed > > r

Re: [VOTE] KIP-21 Dynamic Configuration

ange "InvalidTopicConfiguration" error code to "InvalidEntityConfig" > as proposed in KIP-21. > > > Thanks, > Aditya > > > From: Jun Rao [j...@confluent.io] > Sent: Thursday, May 21, 2015 8:41 AM > To

Re: Review Request 33620: Patch for KAFKA-1690

nnect to sth like prepare()? The confusion is that the key is already connected and we are trying to connect again. clients/src/main/java/org/apache/kafka/common/network/Selector.java <https://reviews.apache.org/r/33620/#comment136233> Not sure why we need to change to a while loop

Re: Review Request 33620: Patch for KAFKA-1690

/SSLFactory.java <https://reviews.apache.org/r/33620/#comment136255> This doesn't seem to exist in java 6. - Jun Rao On May 21, 2015, 5:37 p.m., Sriharsha Chintalapani wrote: > > --- > This is an automatically generated e-m

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

ot allow it to create a new topic. We should have another > Request that takes in config parameters from the user regarding how he/she > wants the topic to be created. This request can be used if we get an empty > TopicMetadata Response. > > > Thanks, > > Mayuresh > > >

Re: Review Request 34415: Patch for KAFKA-2195

maintain the compatibility of the all constructors. The reason is that a client always constructs a response from a struct. We can freely change the signature of other constructors since only the server will use them. - Jun Rao On May 19,

Kafka KIP hangout May 26

Hi, Everyone, We will have a KIP hangout at 11 PST on May 26. The following is the agenda. If you want to attend and is not on the invite, please let me know. Agenda: KIP-12 (sasl/ssl authentication): status check KIP-21 (configuration management) KIP-19 (Add a request timeout to NetworkClient) K

Re: Review Request 33065: Patch for KAFKA-1928

ain/scala/kafka/network/SocketServer.scala <https://reviews.apache.org/r/33065/#comment136692> Perhaps we can just log the exception. - Jun Rao On May 24, 2015, 4:53 p.m., Gwen Shapira wrote: > > --- > This is an automatically g

Re: Review Request 34608: Patch for KAFKA-2217

we can just have completedSends(). - Jun Rao On May 26, 2015, 4:38 p.m., Jason Gustafson wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https

Re: Review Request 34415: Patch for KAFKA-2195

/main/java/org/apache/kafka/common/requests/MetadataResponse.java <https://reviews.apache.org/r/34415/#comment136737> Could we remove this constructor and just keep the one with errors? - Jun Rao On May 24, 2015, 7:49 p.m., Andrii Biletskyi

Re: Review Request 34450: Fix KAFKA-2017

> On May 21, 2015, 12:16 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines > > 102-106 > > <https://reviews.apache.org/r/34450/diff/2/?file=965426#file965426line102> > > > > Another way to do this is to o

Re: Review Request 33065: Patch for KAFKA-1928

ics related properties in KafkaConfig. - Jun Rao On May 24, 2015, 4:53 p.m., Gwen Shapira wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://rev

Re: Review Request 33065: Patch for KAFKA-1928

> On May 27, 2015, 6:44 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 109 > > <https://reviews.apache.org/r/33065/diff/16/?file=971264#file971264line109> > > > > Agreed with Joel's point. We should probably just

Re: Review Request 34734: Patch for KAFKA-2226

t is now null, we quit the loop, (5) the reinsert process adds the entry to a new list. At this point, a completed entry still exists in the list. - Jun Rao On May 27, 2015, 9 p.m., Yasuhiro Matsuda wrote: > > --- > Thi

Re: Review Request 34734: Patch for KAFKA-2226

> On May 28, 2015, 7:10 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/utils/timer/TimerTaskList.scala, lines 132-135 > > <https://reviews.apache.org/r/34734/diff/1/?file=973063#file973063line132> > > > > So, I guess the race condition is the following. T

Re: Review Request 34734: Patch for KAFKA-2226

> On May 28, 2015, 7:10 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/utils/timer/TimerTaskList.scala, lines 132-135 > > <https://reviews.apache.org/r/34734/diff/1/?file=973063#file973063line132> > > > > So, I guess the race condition is the following. T

Re: [DISCUSSION] Can we move the ack timeout in ProducerRequest to broker?

Hi, Jiangjie, The replication time may vary a bit for different partitions. For example, a partition with more replicas may take a bit more time to propagate the messages. Also, the replication time depends on network latency. If you have a Kafka cluster across availability zones, data will be rep

Re: [VOTE] KIP-21 Dynamic Configuration

CreateTopic will still allow setting configs as it is > nice to be able to specify configs while creating the topic. > - TopicMetadataResponse shoudn't return config for the topic. > DescribeConfig is the way to go. > - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig&

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

able clients to who don't want to create a topic > with default partitions, if it does not exist and then can manually create > the topic with their own configs(#partitions). > > Thanks, > > Mayuresh > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao wrote: > > > M

Re: Review Request 34734: Patch for KAFKA-2226

Not sure if I follow this comment. - Jun Rao On May 29, 2015, 12:19 a.m., Yasuhiro Matsuda wrote: > > --- > This is an automatically generated e-mail. To reply, visit

Re: [Bump] Code review for KIP-11

Parth, I will take a look. Thanks, Jun On Fri, May 29, 2015 at 10:49 AM, Parth Brahmbhatt < pbrahmbh...@hortonworks.com> wrote: > Hi, > > Can someone please review the following CRs: > > Public entities and interfaces with changes to KafkaAPI and KafkaServer: > https://reviews.apache.org/r/344

Re: Review Request 34734: Patch for KAFKA-2226

celed -> cancelled core/src/main/scala/kafka/utils/timer/TimingWheel.scala <https://reviews.apache.org/r/34734/#comment137520> canceled -> cancelled - Jun Rao On May 29, 2015, 5:49 p.m., Yasuhiro Matsuda wrote: > > ---

Re: Review Request 34734: Patch for KAFKA-2226

performance degradation when running TestPurgatoryPerformance? - Jun Rao On May 29, 2015, 10:10 p.m., Yasuhiro Matsuda wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Review Request 34492: Patch for KAFKA-2210

ent the acl value stored in ZK? core/src/test/scala/unit/kafka/security/auth/AclTest.scala <https://reviews.apache.org/r/34492/#comment137746> This is probably not needed if we turn ACL to a case class. core/src/test

Re: Anyone else sees SocketServerTest failing on trunk?

Hmm, don't see the failure myself. Thanks, Jun On Sun, May 31, 2015 at 2:30 PM, Gwen Shapira wrote: > Hi, > > I'm running: > ./gradlew cleanTest test > > on trunk and all of SocketServerTest tests are failing with: > java.net.SocketException: Socket closed > at java.net.PlainSocket

Re: Review Request 33065: Patch for KAFKA-1928

33065/#comment138032> Incorrect prefix. core/src/main/scala/kafka/server/KafkaServer.scala <https://reviews.apache.org/r/33065/#comment138033> config.metricReporterClasses is only used here. I am wondering if it's better to define it as a java List to avoid the back-and-f

Re: [DISCUSSION] Partition Selection and Coordination By Brokers for Producers

Bhavesh, I am not sure if load balancing based on the consumption rate (1.b) makes sense. Each consumer typically consumes all partitions from a topic. So, as long as the data in each partition is balanced, the consumption rate will be balanced too. Selecting a partition based on the size of each

notes from Kafka KIP hangout Jun. 2

The following are my notes. 1. Security: Harsh will upload a new SSL patch in the next day or two. 2. KIP-21 (dynamic configuration): How to serve DescribeConfigRequest? Currently, topic config is only cached on brokers storing the topic. We will need to either cache the config on every broker or

Re: notes from Kafka KIP hangout Jun. 2

We have a proposal to propagate ISR changes to every broker in KAFKA-1367. Thanks, Jun On Tue, Jun 2, 2015 at 12:55 PM, Ashish Singh wrote: > Jun, > > Thanks for the notes! > > @Jun/ Joel, An inline question on ISR info in TMR. > > On Tuesday, June 2, 2015, Jun Rao wrote

Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

The wiki references "network.request.timeout.ms" in ProducerConfig. It should be "request.timeout.ms", right? Thanks, Jun On Tue, Jun 2, 2015 at 2:06 PM, Jiangjie Qin wrote: > Hi folks, > > Thanks a lot for all the input and help. It looks we do not have further > concerns on KIP-19. I’ve upda

Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

+1. Thanks, Jun On Tue, Jun 2, 2015 at 3:13 PM, Jiangjie Qin wrote: > Ah, yes. Just changed it to ³request.timeout.ms². > > Thanks. > > Jiangjie (Becket) Qin > > On 6/2/15, 2:31 PM, "Jun Rao" wrote: > > >The wiki references "network.request.t

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

> clients > > > > > could > > > > > > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the > > > list > > > > of > > > > > > mechanisms enabled in the broker. I think Jun's point was that by > > >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

gt; > > > That said, I'd like to talk about use cases, to make sure > > that > > > this is in fact useful. The KIP should be updated with whatever use > > cases > > > we come up with. > > > > > > First of all, an ob

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

Hi, Rajini, A couple of other questions on the KIP. 10. For the config values stored in ZK, are those keys (s, t, k, i, etc) stored under scram-sha-256 standard? 11. Could KIP-48 (delegation token) use this KIP to send delegation tokens? In KIP-48, the client sends a HMAC as the delegation token

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

ipal and HMAC for authentication. The server > callback handler can be updated when delegation tokens are implemented to > generate Kafka principal accordingly. > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao wrote: > > > Hi, Rajini, > > > > A couple of other questi

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

alue pairs. We can add an extension key to the first client message to > indicate delegation token. Broker can then obtain credentials and principal > using a different code path for delegation tokens. > > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao wrote: > > > Magnus, > >

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi, Radai, Thanks for the KIP. Some comments below. 1. The KIP says "to facilitate faster implementation (as a safety net) the pool will be implemented in such a way that memory that was not release()ed (but still garbage collected) would be detected and "reclaimed". this is to prevent "leaks" in

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

licit "OFF" (or basically anything <=0). > Long.MAX_VALUE would still create a pool, that would still waste time > tracking resources. I dont really mind though if you have a preferred magic > value for off. > > > > > > On Wed, Nov 9, 2016 at 9:28 AM, Jun Rao wr

Re: ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

Hmm, was the corruption detected inside processPartitionData()? Typically, the corruption should be detected at https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L130 when we do a shallow iteration of the message set. If the corruption is dete

Re: [VOTE] KIP-84: Support SASL SCRAM mechanisms

Hi, Rajini, Thanks for the proposal. +1. A few minor comments. 30. Could you add that the broker config sasl.enabled.mechanisms can now take more values? 31. Could you document the meaning of s,t,k,i used in /config/users/alice in ZK? 32. In the rejected section, could you document why we decid

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

> >> > > > > 100 for 1MB and queued.max.bytes = 0.25 for 25% of > available > > >> > heap? > > >> > > > > > > >> > > > > jun's 2.2 - queued.max.bytes + socket.request.max.bytes still > &

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

at > done require it, as rajini suggested earlier) > > On Wed, Nov 16, 2016 at 9:04 AM, Jun Rao wrote: > > > Hi, Radai, > > > > Thanks for the updated proposal. +1 overall. A couple of comments below. > > > > 1. Our current convention is to avoid using get

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

t value > from pool) and "Throttles" (some moving-avg of how often does throttling > due to no mem kicks in). maybe also "BuffersOutstanding" ? > > On Thu, Nov 17, 2016 at 7:01 PM, Jun Rao wrote: > > > Hi, Radai, > > > > 2. Yes, on the server

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi, Radai, Could you add a high level description of the newly added metrics to the KIP wiki? Thanks, Jun On Wed, Nov 23, 2016 at 3:45 PM, radai wrote: > Hi Jun, > > I've added the sensor you requested (or at least I think I did ) > > On Fri, Nov 18, 2016 at 12:3

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi, Michael, Thanks for the KIP. A few comments below. 1. The message format change contains "HeadersLength Headers". Is that intended? 2. For compressed messageset, is the tombstone bit only set at the shallow level? Do we always leave that bit in the wrapper message unset? An alternative is to

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi, Michael, In order to answer the first two questions, it would be helpful if we could identify 1 or 2 strong use cases for headers in the space for third-party vendors. For use cases within an organization, one could always use other approaches such as company-wise containers to get around w/o

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

down convert and remove the > value so the old consumer had existing behavior, or is there something we > haven’t thought about? > > Cheers > Mike > > On 30/11/2016, 18:12, "Jun Rao" wrote: > > Hi, Michael, > > Thanks for the KIP. A few comm

Re: [DISCUSS] KIP-82 - Add Record Headers

wire format > changes > > >> and > > >> > im > > >> > >> not > > >> > >> >> >>> saying it could have been completely done with headers, > but > > >> > >> >> exact

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

e updated. Thanks again for the feedback. > > Agree yes we should align up when it gets to that, I assume you’ve flagged > the same in the other KIP? > > I think for now let’s get this KIP approved, then we can start the work to > get an initial PR, then we can discuss how to align t

Re: [DISCUSS] KIP-82 - Add Record Headers

is quite simply. > > As per how some other brokers do this (cough qpid, artemis) message uuid > are used to dedupe where message is sent and produced but the client didn’t > receive the ack, and there for replays the send, by having a unique message > id per message, this can be filtere

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

discussed? > > The second point I understand what you're getting at now my apologies. Yes > this makes sense to save on touching the message, if we're the only kip > going in, in this release. > > Cheers > Mike > > Sent using OWA for iPhone > __

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

> The second point I understand what you're getting at now my apologies. > Yes > > this makes sense to save on touching the message, if we're the only kip > > going in, in this release. > > > > Cheers > > Mike > > > > Sent using OWA for i

Re: [DISCUSS] KIP-101: Alter Replication Protocol to use Leader Generation rather than High Watermark for Truncation

Hi, Onur, The reason for keeping track of the CZXID of the broker registration path is the following. There is one corner case bug (KAFKA-1120) that Ben mentioned where the controller could miss a ZK watcher event if the broker deregisters and registers quickly. Always triggering a leader election

Re: [DISCUSS] KIP-82 - Add Record Headers

ve our producer move, where > its IP would change. > > 107) > UUID is set on the message by interceptors before actual producer > transport send. This is for platform level message dedupe guarantee, the > business payload should be agnostic to this. Please see > https://activemq.

Re: [VOTE] 0.10.1.1 RC1

Hi, Guozhang, Thanks for preparing the release. Verified quickstart. +1 Jun On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the second, and hopefully the last candidate for the release of > Apache Kafka 0.10.1.1 before th

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Just to follow up on Radai's idea of pushing the buffering logic to the broker. It may be possible to do this efficiently if we assume aborted transactions are rare. The following is a draft proposal. For each partition, the broker maintains the last stable offset (LSO) as described in the document

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

onds, > >> > > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.? > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > Done. > >> >

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

Hi, Mani Thanks for the proposal. Looks good at the high level. A few comments below. 101. DelegationTokenRequest/DelegationTokenResponse 101.1 I am wondering if renewer should be principalType + name to match what we have in kafka-acls tool? 101.2 Do we need to return owner in DelegationTokenRe

Re: [DISCUSS] KIP-124: Request rate quotas

limit. It is necessary for user to be > able > > to > > > > know this information to figure how whether they have reached there > > quota > > > > limit. How about we add log4j log on the server side to periodically > > > print > > >

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

s also not handled by > ZK event handling thread. The LeaderAndIsrReponse can be handled by the > same thread is that currently handling ControllerShutdownRequest so that we > don't require new type of synchronization. Further, it should be rare to > require additional synchroniz

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

may > not be a problem in practice with additional requirement (e.g. don't use > more than one log directory if using RAID-10), ideally we want to avoid > such limit. > > 3) Extra ZK read of failed-log-directory/[directory] when broker starts > > > My main concern with t

Re: add contributor

Hi, Clement, Thanks for your interest in Kafka. Added you to the contributor list. Jun On Fri, Feb 24, 2017 at 6:10 AM, Eno Thereska wrote: > Hello, > > Sending on behalf on user who wants to contribute. Could we add Clement > Valiente to contributor list please? Jira user id is "cvaliente". >

Re: [DISCUSS] KIP-124: Request rate quotas

ah you are right. I thought it wasn't because at LinkedIn it > will > > be > > >> > too > > >> > > much pressure on inGraph to expose those per-clientId metrics so > we > > >> ended > > >> > > up printing them periodically

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Dong, Thanks for the reply. Personally, I'd prefer not to write the created flag per replica in ZK. Your suggestion of disabling replica creation if there is a bad log directory on the broker could work. The only thing is that it may delay the creation of new replicas. I was thinking that an

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

lowing > broker to remove offline replica from fetcher thread will increase churns > in ISR. Do you think this is concern with this approach? > > I have updated the KIP to remove created flag from ZK and change the filed > name to isNewReplica. Can you check if there is any issu

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

6 is not typically advised anymore due to failure > issues, as Dong mentions, e.g.: http://www.zdnet.com/article/ > why-raid-6-stops-working-in-2019/ <http://www.zdnet.com/article/ > why-raid-6-stops-working-in-2019/> > > Eno > > > > On 27 Feb 2017, at 21:16, Jun Rao w

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

elieves it is useful and worth the effort. Yeah, it > will be useful to see what everyone else think about it. > > > Thanks, > Dong > > > On Mon, Feb 27, 2017 at 1:16 PM, Jun Rao wrote: > > > Hi, Dong, > > > > For RAID5, I am not sure the rebuild c

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

Hi, Dong, Yes, this change makes sense to me. Thanks, Jun On Mon, Feb 27, 2017 at 8:26 PM, Dong Lin wrote: > Hi Jun and everyone, > > I would like to change the KIP in the following way. Currently, if any > replica if offline, the purge result for a partition will > be NotEnoughReplicasExcept

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

u, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat < > > > > > > gharatmayures...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > Thanks

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

o start discussion on KIP-113 after we have > reached agreement on KIP-112. I have actually opened KIP-113 discussion > thread on 1/12 together with this thread. I have yet to add the ability to > list offline directories in KIP-113 which we discussed in this thread. > > Thanks for all yo

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

ession and then deprecate the KafkaPrincipal later. > > > > I am fine either ways. What do you think? > > > > Thanks, > > > > Mayuresh > > > > On Tue, Feb 28, 2017 at 9:53 AM, Jun Rao wrote: > > > > > Hi, Ismael, > > > > >

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

Hi, Dong, It seems that delete means removing everything while purge means removing a portion. So, it seems that it's better to be able to distinguish the two? Thanks, Jun On Wed, Mar 1, 2017 at 1:57 PM, Dong Lin wrote: > Hi all, > > I have updated the KIP to include a script that allows user

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the KIP. A few comments below. 1. For moving data across directories 1.1 I am not sure why we want to use ReplicaFetcherThread to move data around in the leader. ReplicaFetchThread fetches data from socket. For moving data locally, it seems that we want to avoid the socket ov

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

complexity to controller's > >> implementation, e.g. log destination broker per LeaderAndIsrRequet. > Becket > >> also shares this opinion with me. Is the only purpose of doing so to > make > >> controller log simpler? > >> > >> And certainly, I

Re: [DISCUSS] KIP-124: Request rate quotas

reports CPU usage and may be a bit easier for people to understand. Thanks, Jun On Fri, Feb 24, 2017 at 10:31 AM, Jun Rao wrote: > Hi, Jay, > > 2. Regarding request.unit vs request.percentage. I started with > request.percentage too. The reasoning for request.unit is the following. &g

Re: [DISCUSS] KIP-124: Request rate quotas

script to decrease/increase quotas by a multiple for all users. > > So it really boils down to which quota unit is most intuitive in terms of > configuration. And from the discussion so far, it feels like opinion is > divided on whether quotas should be carved out of an absolute 10

Re: [DISCUSS] KIP-124: Request rate quotas

s or percent). Just not sure > it makes it easier for admin in all cases. > > Regards, > > Rajini > > > On Fri, Mar 3, 2017 at 6:03 AM, Jun Rao wrote: > > > Hi, Rajini, > > > > Consider modeling as n * 100% unit. For 2), the question is what's &

Re: [DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

Hi, Guozhang, Thanks for the KIP. A couple of comments. 1. About the impact on producer batching. My understanding is that typically different sub-topologies in the same task are publishing to different topics. Since the producer batching happens at the topic/partition level, using a producer per

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

e should probably do > the > > same in this KIP for consistency. Once we do that, we should also > consider > > if `Before` should be in the method name or should be in the parameter > > class. Just an example to describe what I mean, one could say > > `deleteRecords(

Re: [DISCUSS] KIP-124: Request rate quotas

:04 AM, Rajini Sivaram > wrote: > > > Jun, > > > > Thank you for the explanation, I hadn't realized you meant percentage of > > the total thread pool. If everyone is OK with Jun's suggestion, I will > > update the KIP. > > > > Thanks, > > >

Re: [DISCUSS] KIP-124: Request rate quotas

> > I just asked Todd to take a look. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Tue, Mar 7, 2017 at 2:17 PM, Jun Rao wrote: > > > > > Hi, Jiangjie, > > > > > > Yes, I agree that byte rate already protects the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

n On Thu, Mar 2, 2017 at 11:18 PM, Dong Lin wrote: > Hey Jun, > > Thanks for all the comments! Please see my answer below. I have updated the > KIP to address most of the questions and make the KIP easier to understand. > > Thanks, > Dong > > On Thu, Mar 2, 2017 at 9:35

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

will ask SREs for their opinion. > > Thanks, > Dong > > > > > > > > > On Wed, Mar 8, 2017 at 1:24 PM, Jun Rao wrote: > > > Hi, Dong, > > > > Thanks for the updated KIP. A few more comments below. > > > > 1.1 and 1.2: I am still not su

Re: [DISCUSS] KIP-124: Request rate quotas

he utilization > > percentage as being fixed, not what the percentage reflects. I’m not tied > > to either way of doing it, provided that we do not lock clients to a > single > > thread. For example, if I specify that a given client can use 10% of a > > single thread, that shou

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

address your > comments. > > Please see my reply inline. > > > > Can you let me know if the latest KIP has addressed your comments? > > > > On Wed, Mar 8, 2017 at 9:56 PM, Jun Rao wrote: > > > >> Hi, Dong, > >> > >> Thanks for the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

DirRequest with dir = any. Jun On Mon, Mar 13, 2017 at 3:32 PM, Dong Lin wrote: > Hey Jun, > > Thanks much for your detailed comments. Please see my reply below. > > On Mon, Mar 13, 2017 at 9:09 AM, Jun Rao wrote: > > > Hi, Dong, > > > > Thanks for the upd

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi, Everyone, Jason has been working on the new message format related to EOS ( https://github.com/apache/kafka/pull/2614). He has included the header changes proposed in the KIP, which reduces the overhead for supporting an additional message format change if done separately. Since the message fo

Re: [VOTE] KIP-117: Add a public AdminClient API for Kafka admin operations

Hi, Colin, Thanks for the KIP. Looks good overall. A few comments below. 1. Sometimes we return CompletableFuture> and some other times we return Map> , which doesn't seem consistent. Is that intentional? 2. We support batching in CreateTopic/DeleteTopic/ListTopic, but not in DescribeTo

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

me know your thoughts on this. > > Thanks, > > Mayuresh > > On Tue, Feb 28, 2017 at 2:33 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Jun, > > > > Sure. > > I had an offline discussion with Joel on how we

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the update. A few replies inlined below. On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin wrote: > Hey Jun, > > Thanks for your comment! Please see my reply below. > > On Wed, Mar 15, 2017 at 9:45 PM, Jun Rao wrote: > > > Hi, Dong, > > > &

Re: [VOTE] KIP-124: Request rate quotas

Hi, Rajini, Thanks for the proposal. +1 from me. Jun On Mon, Mar 20, 2017 at 7:47 AM, Rajini Sivaram wrote: > I would like to initiate the voting process for KIP-124: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 124+-+Request+rate+quotas > > The KIP proposes to add request proc

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

low. > > On Mon, Mar 20, 2017 at 7:45 AM, Jun Rao wrote: > > > Hi, Dong, > > > > Thanks for the update. A few replies inlined below. > > > > On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin wrote: > > > > > Hey Jun, > > > > > >

Re: [VOTE] KIP-82 Add Record Headers

Hi, Michael, The KIP looks good to me overall. Just one comment. The wiki says "This will be done by calling "close()" method". However, there is no close() in Headers. Thanks, Jun On Thu, Mar 23, 2017 at 9:34 AM, Michael Pearce wrote: > Thanks all for the +1 votes so far, just one more commi

Re: [VOTE] KIP-82 Add Record Headers

called from user code. > > Ismael > > On 24 Mar 2017 1:51 am, "Jun Rao" wrote: > > > Hi, Michael, > > > > The KIP looks good to me overall. Just one comment. The wiki says "This > > will be done by calling "close()" method". How

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

if > they want to make sure that the ChangeReplicaDirRequest is executed. > > Thanks, > Dong > > > > On Tue, Mar 21, 2017 at 8:37 AM, Jun Rao wrote: > > > Hi, Dong, > > > > 10. I was mainly concerned about the additional complexity needed to > >

<    1   2   3   4   5   6   7   8   9   10   >