Re: [DISCUSS] KIP-354 Time-based log compaction policy

Hi Wesley, Thanks for the KIP and sorry for being late to the party. I wanted to understand, the scenario you mentioned in Proposed changes : - > > Estimate the earliest message timestamp of an un-compacted log segment. we > only need to estimate earliest message timestamp for un-compacted log >

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Colin/Zahari, I have created a ticket for the similar/same feature : https://issues.apache.org/jira/browse/KAFKA-7548 We (Linkedin) had a use case in Samza at Linkedin when they moved from the SimpleConsumer to KafkaConsumer and they wanted to do this pause and resume pattern. They realized the

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

f configuration as this > data should not be thrown away at all. Submitting a PR sounds great, > although I feel a bit jealous you (LinkedIn) beat me to my first kafka > commit ;) Not sure how things stand with the voting process ? > > Zahari > > > > On Th

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, Created the patch here : https://github.com/apache/kafka/pull/5844 Thanks, Mayuresh On Thu, Oct 25, 2018 at 4:42 PM Mayuresh Gharat wrote: > Hi Zahari, > > Oops. We had planned to put this patch upstream but somehow slipped my > mind. We were recently going over hotf

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

o reflect the fact > that > > as > > > already discussed - we do not really need any kind of configuration as > > this > > > data should not be thrown away at all. Submitting a PR sounds great, > > > although I feel a bit jealous you (LinkedIn) beat me to my f

Re: Apache Kafka blog on more partitions support

Thanks Jun for sharing this. Looks nice ! Do we intend to shed light on how much time is required, on an average, for new Leader election. Also would it be good to add "if the controller waits for the LeaderAndIsrResponses before sending shutDown_OK to the shutting down broker". Thanks, Mayuresh

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

Hi Lincong, Thanks for the KIP. As Colin pointed out, would it better to expose certain specific pieces of information from the request/response like api key, request headers, record counts, client ID instead of the entire request/response objects ? This enables us to change the request response

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for the KIP and sorry for being late to the party. This KIP is really useful for us at Linkedin. I had a few questions : The idea of having static member name seems nice, but instead of a config, would it be possible for it to be passed in to the consumer at runtime? This is be

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

s topic. I will do more analysis and make a trade-off > comparison. Nice catch! > > > I hope the explanations make sense to you. I will keep polishing on the > edge cases and details. > > > Best, > > Boyang > > > From: Mayuresh Gharat

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for updating the KIP. This is a step good direction for stateful applications and also mirroring applications whose latency is affected due to the rebalance issues that we have today. I had a few questions on the current version of the KIP : For the effectiveness of the KIP, con

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

to replace session timeout". Currently the > > default max.poll.interval.ms< > > > https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Feur04.safelinks.protection.outlook.com%2F%3Furl%3Dhttp%253A%252F%252Fmax.poll.interval.ms%26data%3D02%257C01%257C%257Cb48d52bf63324

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

t? If yes, should we specify it in the KIP ? 7. Are there any specific failure scenarios when you say "other potential failure cases."? It would be good to mention them explicitly, if you think there are any. 8. It would be good to have a rollback plan as you have for roll

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Edoardo, Thanks a lot for the KIP. I have a few questions/suggestions in addition to what Radai has mentioned above : 1. Is this meant only for 1:1 replication, for example one Kafka cluster replicating to other, instead of having multiple Kafka clusters mirroring into one Kafka clus

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

y operation, > then it will be fenced with an UNKNOWN_MEMBER_ID error. As long as the > subscription of the new instance hasn't changed, then we can skip the > rebalance and return the current assignment without forcing a rebalance. > > The trick to making this work is in the err

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

__ > > From: Guozhang Wang > > Sent: Wednesday, November 28, 2018 4:51 AM > > To: dev > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by > > specifying member id > > > > Regarding Jason's question and Boyang's respo

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

he list of embedded consumer / producer client ids, > 2) the producer's txn ids if EOS is turned on, and > 3) the consumer's instance ids. > > So that Streams operators can read those values from KafkaStreams directly > before shutting it down and use the list in the LeaveGro

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads. Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote: > Thanks Guozhang for the reply! > > 1. RemoveMembe

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Thanks for the KIP Boyang and great to see the progress on solving the rebalance issues with this and KIP-345. Thanks, Mayuresh On Mon, Dec 3, 2018 at 4:57 AM Stanislav Kozlovski wrote: > Everything sounds good to me. > > On Sun, Dec 2, 2018 at 1:24 PM Boyang Chen wrote: > > > In fact, it's p

Re: [VOTE] KIP-394: Require member.id for initial join group request

+1 (non-binding) Thanks, Mayuresh On Wed, Dec 5, 2018 at 3:59 AM Boyang Chen wrote: > Hey friends, > > I would like to start a vote for KIP-394< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request>. > The goal of this KIP is to impro

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

+1 (non-binding) Thanks, Mayuresh On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger wrote: > +1 (non binding) > > On 12/4/18, 9:43 AM, "Patrick Williams" > wrote: > > Pls take me off this VOTE list > > Best, > > Patrick Williams > > Sales Manager, UK & Ireland, Nordics & Israel

Re: Kafka client Metadata update on demand?

Hi Ming, Kafka clients do update there metadata on NotLeaderForPartitionException. The metadata update happens asynchronously. Also if you are getting this exception for a longer time, it might mean that your client is fetching metadata from a broker whose metadata cache is not updated with the l

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Hi ChienHsing, Thanks for the KIP. It would be great if you can explain with an example, what you mean by " Currently the implementation will return available records starting from the last partition the last poll call retrieves records from. This leads to unfair patterns of record consumption fro

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

ould alternate between A and B. I was proposing to move to > the next one fetch for the next poll call based on the order in the > completed fetch queue, so the order becomes A, B, A, B, A, B. The > implementation parses the completed fetch only once. > > Thanks, CH > > -Orig

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

hanges does not add any additional calls to the > broker nor does it introduce significant processing logic; it just parses > the completed fetch earlier and have a list to manage them. > > > CH > > -Original Message- > From: Mayuresh Gharat > Sent: Tuesday, Dec

[DISCUSS] : KIP-410: Add metric for request handler thread pool utilization by request type

I would like to get feedback on the proposal to add a metric for request handler thread pool utilization by request type. Please find the KIP here : KIP-410: Add metric for request handler thread pool utilization by request type

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Regarding "However, we shall still attempt to remove the member static info if the given `member.id` points to an existing `group.instance.id` upon LeaveGroupRequest, because I could think of the possibility that in long term we could want to add static membership leave group logic for

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

starts while group is in stable, the static map > will be cleaned up and rebalance will be triggered as expected (we are > already in dynamic mode now). > > I hope this answers your question, thanks! > > Boyang > > > From: Mayuresh Ghar

Re: [DISCUSS] : KIP-410: Add metric for request handler thread pool utilization by request type

e-rate" > > > connection-close-rate > Connections closed per second in the window. > > kafka.[producer|consumer|connect]:type=[producer|consumer|connect]-metrics,client-id=([-.\w]+) > > > > A separate nit - you may want to update the discussion thread lin

Re: [ANNOUNCE] New Committer: Vahid Hashemian

congrats !! On Tue, Jan 15, 2019 at 3:42 PM Matthias J. Sax wrote: > Congrats! > > On 1/15/19 3:34 PM, Boyang Chen wrote: > > This is exciting moment! Congrats Vahid! > > > > Boyang > > > > > > From: Rajini Sivaram > > Sent: Wednesday, January 16, 2019 6:50 AM >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Lucas, Thanks for the KIP. I am trying to understand why you think "The memory consumption can rise given the total number of queued requests can go up to 2x" in the impact section. Normally the requests from controller to a Broker are not high volume, right ? Thanks, Mayuresh On Tue, Jul 1

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

adding the extra > >> config with a default value, say 20, guards us from issues in those > >> troublesome times, and IMO there isn't much downside of adding the extra > >> config. > >> > >> @Mayuresh > >> Good catch, this sentence is an obsol

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

y only concern is that this design is tied to the coincidence > that > > > > > > we have two request priorities and there are two ends to a deque. > > > > > > Hence by using the proposed design, it seems the network layer is > > > > > > more tightly coupled with upper layer logic, e.g. if we were to > add > > > >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Actually nvm, correlationId is reset in case of connection loss, I think. Thanks, Mayuresh On Thu, Jul 19, 2018 at 11:11 AM Mayuresh Gharat wrote: > I agree with Dong that out-of-order processing can happen with having 2 > separate queues as well and it can even happen today. > Can w

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

ing and never reset, > hence a broker can leverage that to properly reject obsolete requests. > Thoughts? > > Thanks, > Lucas > > On Thu, Jul 19, 2018 at 12:11 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Actually nvm, correlationId is

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

>> I agree that there is no strong ordering when there are more than one > >>> socket connections. Currently, we rely on controllerEpoch and > leaderEpoch > >>> to ensure that the receiving broker picks up the latest state for each > >>> partition.

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

gt; If that's the case, I think this probably deserves a separate doc and > > > > design independent of this KIP. > > > > > > > > Lucas > > > > > > > > > > > > > > > > On Mon, Jul 23, 2018 at 12:39 PM, Dong Lin

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong !!! Thanks, Mayuresh On Mon, Aug 20, 2018 at 1:36 PM Gwen Shapira wrote: > Congrats Dong Lin! Well deserved! > > On Mon, Aug 20, 2018, 3:55 AM Ismael Juma wrote: > > > Hi everyone, > > > > Dong Lin became a committer in March 2018. Since then, he has remained > > active in the c

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Sure. Thanks, Mayuresh On Wed, Aug 23, 2017 at 5:07 PM, Jun Rao wrote: > Hi, Mayuresh, > > Since this KIP covers the requirement in KIP-111, could you review it too? > > Thanks, > > Jun > > > On Tue, Aug 22, 2017 at 3:04 PM, Jason Gustafson > wrote: > >> Bump. I'll open a vote in a few days i

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Principal would look like and its details, since it will just pass it to the third party library in Kafka Authorizer's authorize() call. Thanks, Mayuresh On Thu, Aug 24, 2017 at 10:21 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Sure. > > Thanks, > >

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

e out this use case. As long as the authorizer and the > principal builder both agree on how user principals are identified, then > they can still be used together. But I am explicitly leaving out support in > the ACL command tool for this use case in this KIP. This is mostly about > clar

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

; library through the `Extended_Plugged_In_Class` as you've suggested above. > Is that reasonable for this use case? > > Thanks, > Jason > > > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Jason, > >

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

+1 (non-binding) Thanks, Mayuresh On Wed, Aug 30, 2017 at 8:51 AM, Jason Gustafson wrote: > I'd like to open the vote for KIP-189: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL. > Thanks to everyone who helped rev

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

Sure . Thanks, Mayuresh On Wed, Sep 13, 2017 at 3:12 PM, Jun Rao wrote: > Hi, Mayuresh, > > Does this KIP obviate the need for KIP-111? If so, could you close that > one? > > Thanks, > > Jun > > On Wed, Sep 13, 2017 at 8:43 AM, Jason Gustafson > wrote: > >> Hi All, >> >> I wanted to mention o

KAFKA-3722 : Discussion about custom PrincipalBuilder and Authorizer configs

Hi All, I came across an issue with plugging in a custom PrincipalBuilder class using the config "principal.builder.class" along with a custom Authorizer class using the config "authorizer.class.name". Consider the following scenario : For PlainText we don't supply any PrincipalBuilder. For SSL

Re: KAFKA-3722 : Discussion about custom PrincipalBuilder and Authorizer configs

method that its need to be handled . In the interest > > of having less config I think option 4 seems to be > > better even though it breaks the interface. > > > > Thanks, > > Harsha > > On Fri, May 20, 2016, a

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

eira, Francois Visconte, Frank Scholten, > > Gabriel Zhang, gaob13, Geoff Anderson, glikson, Grant Henke, Greg > > Fodor, Guozhang Wang, Gwen Shapira, Igor Stepanov, Ishita Mandhan, > > Ismael Juma, Jaikiran Pai, Jakub Nowak, James Cheng, Jason Gustafson, > > Jay Kreps,

Re: Reporting security issues

Excellent :) Thanks, Mayuresh On Tue, May 24, 2016 at 2:55 AM, Ismael Juma wrote: > Hi all, > > Since Kafka implements a number of security features, we need a procedure > for reporting potential security vulnerabilities privately (as per > http://www.apache.org/security/). We have added a sim

Re: KAFKA-3722 : Discussion about custom PrincipalBuilder and Authorizer configs

n, May 23, 2016, at 10:30 AM, Mayuresh Gharat wrote: > > Hi Harsha and Ismael, > > > > Option 2 sounds like a good idea if we want to make this quick fix I > > think. > > Option 4 might require a KIP as its public interface change. I can > > resubmit >

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Nice write up Radai. I think what Jun said is a valid concern. If I am not wrong as per the proposal, we are depending on the entire pipeline to flow smoothly from accepting requests to handling it, calling KafkaApis and handing back the responses. Thanks, Mayuresh On Mon, Aug 8, 2016 at 12:22

Re: [DISCUSS] KIP-73: Replication Quotas

Nice write up Ben. I agree with Joel for keeping this simple by excluding the partitions from the fetch request/response when the quota is violated at the follower or leader instead of having a separate set of threads for handling the quota and non quota cases. Even though its different from the c

Re: [jira] [Issue Comment Deleted] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

You can take a look at this : https://issues.apache.org/jira/browse/KAFKA-1848 Thanks, Mayuresh On Sun, Mar 29, 2015 at 4:45 PM, Jiangjie Qin (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > J

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

I suppose adding this timeout will help. In cases were the broker is not completely down but stops responding to the produce request, tools like Mirror Makers will hang since they are waiting for responses. Adding this timeout enables it to fail the current request and retry with fresh metadata. T

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Agreed we also need to change in the code of Sender.java to indicate that it resembles REPLICATION_TIMEOUT and not the request Timeout. Thanks, Mayuresh On Thu, Apr 16, 2015 at 1:08 PM, Jiangjie Qin wrote: > Hi Guozhang, > > By implicit timeout for close() and flush(), I meant that currently w

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Just a quick question, can we handle REQUEST TIMEOUT as disconnections and do a fresh MetaDataRequest and retry instead of failing the request? Thanks, Mayuresh On Mon, May 4, 2015 at 10:32 AM, Jiangjie Qin wrote: > I incorporated Ewen and Guozhang’s comments in the KIP page. Want to speed >

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

+1 Becket. That would give enough time for clients to move. We should make this change very clear. Thanks, Mayuresh On Tue, May 12, 2015 at 1:45 PM, Jiangjie Qin wrote: > Hey Ewen, > > Very good summary about the compatibility. What you proposed makes sense. > So basically we can do the follow

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

t; 2. If a broker is down, hopefully metadata refresh will find the new > broker and we will not try to reconnect to the broker anymore. > > Comments are welcome! > > Thanks. > > Jiangjie (Becket) Qin > > On 5/12/15, 2:59 PM, "Mayuresh Gharat" wrote: > > >

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

d not see many new connections > > get created. > > 2. If a broker is down, hopefully metadata refresh will find the new > > broker and we will not try to reconnect to the broker anymore. > > > > Comments are welcome! > > > > Thanks. > > > > Jiang

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

M, Jiangjie Qin > >> > >> wrote: > >> > >>> I modified the WIKI page to incorporate the feedbacks from mailing list > >>> and KIP hangout. > >>> > >>> - Added the deprecation plan for TIMEOUT_CONFIG > >>> - Added t

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

include both the time in > >the accumulator an the time taken for the first request, etc. In other > >words this is the upper bound on the time to the Future being satisfied. > > > >*replication.timeout* will default to something reasonable but maybe you > >can override

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

quest.timeout* is the bound on the time after send() complete until > you > > >get an acknowledgement. This covers the connection timeout, and the time > > >in > > >the accumulator. So to implement this, the time we set in the request > sent > > >via N

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi, I had a question about TopicMetadata Request. Currently the way it works is : 1) Suppose a topic T1 does not exist. 2) Client wants to produce data to T1 using producer P1. 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. This in turn creates the default number of partit

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

config in the producer to control whether > TopicCreateRequest should be issued or not on UnknownTopicException. If > this is disabled and the topic doesn't exist, send will fail and the user > is expected to create the topic manually. > > Thanks, > > Jun > > > On

Re: KIP Wiki

+1. Thanks, Mayuresh On Mon, Jun 1, 2015 at 6:51 PM, Joe Stein wrote: > We should probably have some release/vXYZ section so that over time we can > keep track of what KIP where approved for what release, etc/ > > Anything not in a release folder (we could do now release/v0.8.3.0 for > everyth

Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

+1 (non-binding). Thanks Jiangjie. Thanks, Mayuresh On Tue, Jun 2, 2015 at 4:49 PM, Ewen Cheslack-Postava wrote: > +1 non-binding. Thanks for all the work on this Jiangjie! > > Aditya -- see discussion of partitionsFor() API, which needs to also use > one of those timeouts, and is a fetch rat

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

n is, where did this delete come from? What processing > > job > > > > emitted it? What input to the processing job caused this delete > to > > be > > > > produced? For example, if a record in topic A was processed and > > > caused a > > &

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

> > > > > > > > 1) Tracability. I would like to know who issued > this > > delete > > > > > tombstone. It might include the hostname, IP of the > producer of > > the > > > > delete. > > > >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

port a null value non-tombstone message in a log compacted topic, but I > am not sure if that has any use case. > > Thanks, > > Jiangjie (Becket) Qin > > > On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh Gharat < > gharatmayures...@gmail.com> > wrote: > > > I thin

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

v-store, so a > map API. > map.put(key, null) is not the same as map.remove(key), which to me > means a > null value should not represent a delete. a delete should be > explicit > (meaning flag). > > > On Wed, Nov 9, 2016 at

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

omebody sends null > > - broker accepts flags/variable/headers and zero length values as > > tombstones > > > > ​Conversion from one to the other may be possible but I would rather not > > have to do it. > > > > Nacho > > > > > > On Wed, No

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

stage approach is because we want to end up just supporting > tombstone marker (not both) > > But we accept we need to allow organisations and systems a period of > transition (this is what stage 1 provides) > > > > > > Fro

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

ersion config (i.e. magic value), we know for > sure when to down convert the messages to adapt to older clients. Otherwise > we will have to always scan all the messages. It would probably work but > relies on guess or inference. > > Thanks, > > Jiangjie (Becket) Qin &

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

And the old apis > > would not be allowed to be called as the broker now expects all clients > to > > be using latest api only. > > > > At this second phase stage only also we can support a null value in a > > compacted topic without it being treated as deletion. W

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

ting magic byte. The interpretation of an old message format should not > be changed and should always be supported until deprecated. > > Thanks, > > Jiangjie (Becket) Qin > > On Mon, Nov 14, 2016 at 11:28 AM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > &g

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

n, Nov 14, 2016 at 1:43 PM, Michael Pearce > > > > wrote: > > > > > > > I like the idea of up converting and then just having the logic to > look > > > > for tombstones. It makes that quite clean in nature. > > > > > > > > It

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

16, 2016 at 5:08 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > >- If we don't bump up the magic byte, on the broker side, the broker > >will always have to look at both tombstone bit and the value when do > the > >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

missing something, it sounds like kafka will be stuck with supporting both null value and tombstone bit for log compaction for life long, which does not look like a good end state. Thanks, Mayuresh On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat wrote: > Hi Ismael, > > That'

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

l always be interpreted to a tombstone. > > One option is that we keep the current way, i.e. do not support such > message. It would be good to know if there is a concrete use case for such > message. If there is not, we can probably just not support it. > > Thanks, > &g

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

n step (3) is this correct? > > Cheers > Mike > > > > Sent using OWA for iPhone > ________ > From: Mayuresh Gharat > Sent: Thursday, November 17, 2016 5:18:41 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-87 - A

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Michael, I have updated the migration section of the KIP. Can you please take a look? Thanks, Mayuresh On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat wrote: > Hi Michael, > > That whilst sending tombstone and non null value, the consumer can expect > only to receive the non-

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

11/2016, 15:52, "Michael Pearce" > wrote: > > > > Hi Mayuresh, > > > > LGTM. Ive just made one small adjustment updating the wire > protocol to > > show the magic byte bump. > >

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hi Becket, Seems like an interesting idea. I had couple of questions : 1) How do we decide when the batch should be split? 2) What do you mean by slowly lowering the "actual" compression ratio? An example would really help here. Thanks, Mayuresh On Fri, Feb 24, 2017 at 3:17 PM, Becket Qin wrot

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

n-binding) > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com > > > wrote: > > > > > Hi Jun, > > > > > > Thanks a lot for the comments and reviews. > > > I agree we should log the user

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

to 1.0 and split > the batch. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Feb 27, 2017 at 10:30 AM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Hi Becket, > > > > Seems like an interesting idea. > > I had couple

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

d to `Session` with > the original principal. We can potentially deprecate the existing one, if > we're sure we don't need it (or we can leave it for now). > > Ismael > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

omething we only do as > a > > last resort. Is there strong justification for doing it here? > > > > Ismael > > > > On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat < > > gharatmayures...@gmail.com> wrote: > > > > > Hi Ismael, >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Koshy wrote: > > > If we deprecate KafkaPrincipal, then the Authorizer interface will also > > need to change - i.e., deprecate the getAcls(KafkaPrincipal) method. > > > > On Tue, Feb 28, 2017 at 10:11 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com> wrote:

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

on will help us to support other implementation of Java.security.Principal as well which seems necessary especially since Kafka provides pluggable Authorizer and PrincipalBuilder. Let me know your thoughts on this. Thanks, Mayuresh On Tue, Feb 28, 2017 at 2:33 PM, Mayuresh Gharat wrote: >

[DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi All, We have created KIP-135 to propose that Kafka should return a non-retriable error when the producer produces a message with null key to a log compacted topic. Please find the KIP wiki in the link : https://cwiki.apache.org/confluence/display/KAFKA/KIP-135+%3A+Send+of+null+key+to+a+compac

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Principal generated from the > SSL or SASL library. How do we make sure that the same type of > java.security.Principal > can be created and will match? > > Thanks, > > Jun > > > On Wed, Mar 15, 2017 at 8:48 PM, Mayuresh Gharat < > gharatmayures...@gmail.com >

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

change, that you would > recommend upgrading clients before upgrading brokers? > > Thanks, > -James > > > > On Mar 22, 2017, at 3:07 PM, Mayuresh Gharat > wrote: > > > > Hi All, > > > > We have created KIP-135 to propose that Kafka should return a

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

> opinion. > > Ismael > > On Wed, Mar 22, 2017 at 10:07 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Hi All, > > > > We have created KIP-135 to propose that Kafka should return a > non-retriable > > error when the producer

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

setRequest. We don't return DuplicatedPartitionException in this > case. > > Thanks, > Dong > > > > On Wed, Mar 22, 2017 at 3:07 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > We have created KIP-135 to propose that Kafka should

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

ncrease overtime and become > harder to maintain. > > Thanks, > Dong > > > On Mon, Mar 27, 2017 at 7:20 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Dong, > > > > I had thought about this before and wanted to do similar thin

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

reply. I > guess the question is, do we still need this KIP if we can have custom > error message specified in the exception via the other KIP? > > > On Mon, Mar 27, 2017 at 11:00 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Hi Dong, > >

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

clients to work with newer brokers. > That's why I recommended that we only throw this error if the > ProduceRequest is version 3 or higher. > > Ismael > > P.S. Note that we already send error messages back for the CreateTopics > protocol API (I added that in the

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi Ismael, I have updated the KIP. Let me know if everything looks fine then I will begin voting. Thanks, Mayuresh On Wed, Mar 29, 2017 at 9:06 AM, Mayuresh Gharat wrote: > Hi Ismael, > > I agree. I will change the compatibility para and start voting. > > Thanks, > > May

[VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi All, It seems that there is no further concern with the KIP-135. At this point we would like to start the voting process. The KIP can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-135+%3A+Send+of+null+key+to+a+compacted+topic+should+throw+non-retriable+error+back+to+user

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Bumping up this thread. On Mon, Apr 3, 2017 at 4:22 PM, Mayuresh Gharat wrote: > Hi All, > > It seems that there is no further concern with the KIP-135. At this point > we would like to start the voting process. The KIP can be found at > https://cwiki.apache.org/confluence/di

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

ion of > this exception and use it in the various places where it makes sense. > > Ismael > > On Tue, Apr 4, 2017 at 12:22 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > It seems that there is no further concern wi

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Ping Ismael. Thanks, Mayuresh On Thu, Apr 6, 2017 at 9:50 AM, Mayuresh Gharat wrote: > Hi Ismael, > > Are you suggesting to use the InvalidRecordException when the key is null? > > Thanks, > > Mayuresh > > On Thu, Apr 6, 2017 at 8:49 AM, Ismael Juma wrote: >

  1   2   3   4   5   6   7   >