Re: [DISCUSS] KIP-124: Request rate quotas

internal distribution of the work between the two thread pools in > Kafka. > > > Regards, > > Rajini > > > On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao wrote: > > > Hi, Rajini, > > > > Thanks for the proposal. > > > > The benefit of using

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

cleaner. Is there specific advantage of having controller send > tells broker to stop fetching data for offline replicas? > > Regarding 22, I agree with your observation that it will happen. I will > update the KIP and specify that broker will exist with proper error message > in the log and

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

ard Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen > Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang, > Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour, > huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason > Gust

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

n each > request. Would it be OK to just specify the name of the principal. > Is there any major reason, we don't want to change the logging format? > > Thanks, > > Mayuresh > > > > On Mon, Feb 20, 2017 at 10:18 PM, Jun Rao wrote: > > > Hi, Mayuresh, > &g

Re: [DISCUSS] KIP-124: Request rate quotas

n that VMs and containers get from hypervisors or OS > schedulers. > > While different client access patterns can use wildly different amounts > of > > request thread resources per request, a given application will generally > > have a stable access pattern and can figure out

Kafka KIP meeting Mar 15 at 11:00am PST

Hi, Everyone, We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to attend but haven't received an invite, please let me know. The following is the agenda. Agenda: KIP-35 - Retrieving protocol version KIP-45 - Standardize all client sequence interaction on j.u.Collection KIP-3

Re: Kafka KIP meeting Mar 15 at 11:00am PST

+Proposals . Thanks, Jun On Mon, Mar 14, 2016 at 1:37 PM, Jun Rao wrote: > Hi, Everyone, > > We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to > attend but haven't received an invite, please let me know. The following is > the agenda. > > Agenda: >

Re: [VOTE] KIP-43: Kafka SASL enhancements

t; If I can see correctly, Harsha and I are the only committers who voted, > > so > > > we are missing a 3rd vote. > > > > > > Gwen > > > > > > On Thu, Mar 24, 2016 at 11:24 AM, Rajini Sivaram < > > > rajinisiva...@googlemail.com> wrote:

Re: [VOTE] KIP-43: Kafka SASL enhancements

he KIP to clarify these points. > > Thanks, > > Rajini > > > On Mon, Mar 28, 2016 at 6:17 PM, Jun Rao wrote: > > > Hi, Rajini, > > > > Sorry for the late response. The revised KIP looks good overall. Just a > few > > minor comments below. > > >

Re: Jira

Mattias, Just added you to the contributor list. Thanks, Jun On Thu, Mar 31, 2016 at 3:44 PM, Matthias J. Sax wrote: > Hi, > > I was browsing Kafka JIRAs and realized, that I cannot assign JIRAs to > myself. Can anyone give me credentials for this, please? > > username: mjsax > > Thanks. :) >

Re: [VOTE] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

Ashish, Thanks for the KIP. It seems that a specific implementation of Authorizer can reject invalid user type in addAcl/removeAcl without needing the new getSupportedPrincipalTypes() method, right? It's probably useful to provide the supported user types as information through CLI (e.g., when --

Re: [VOTE] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

hould getSupportedPrincipalTypes() right now. It's easy to add a new method to the interface, but hard to remove/change an existing method. Grant, could you comment on the latter? Thanks, Jun On Sat, Apr 2, 2016 at 11:43 AM, Ashish Singh wrote: > Hello Jun, > > > On Fri, Apr 1, 2016 at 9

Re: [VOTE] KIP-4 Metadata Schema

Grant, Thanks for the writeup. A few comments. 1. It seems a bit weird to return just a list of internal topics w/o the corresponding metadata. It also seems a bit weird to return the internal topics even if the client doesn't ask for it. Would it be better to just add a flag in topic_metadata to

Re: KIP-4 Wiki Update

Ismael, The reason that we only apply the regex for topics on the client side is that regex is java specific. So, for non-java clients, it would be a bit weird for them to include java specific stuff in the wire protocol. Thanks, Jun On Sun, Apr 3, 2016 at 12:36 AM, Ismael Juma wrote: > Hi Gr

Re: [VOTE] KIP-4 Metadata Schema

eans. I do prefer the booleans from an > > expressiveness standpoint but was not strongly opinionated on the > > structure. > > > > Thank you, > > Grant > > > > On Sun, Apr 3, 2016 at 8:00 PM, Ismael Juma wrote: > > > >> Hi Jun, &g

Re: [VOTE] KIP-4 Metadata Schema

t errored as unavailable. You are > suggesting to drop the unavailable ones and return just the replicas with > no-errors and return no error code on the partition. Is that correct? > > Under what scenario does the MetadataCache have a replica that is not > available? > > Thanks, >

Re: How will KIP-35 and KIP-43 play together?

I think with KIP-43, the existing way of sasl handshake during connection still works. It's just that if you want to support non-GSSAPI, you will need a new sasl handshake implementation in the client. It's unfortunate that Protocol currently only covers the communication after the connection is re

Re: [VOTE] KIP-4 Metadata Schema

ICA_NOT_AVAILABLE error code and only replicas 4,6 in the > > assigned replicas. It's more intuitive to send no error code and 4,5,6 in > > the assigned replicas in this case. > > > Should the list with no error code just be 4,6 since 5 is not available? > > > On Mon,

Re: [VOTE] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

Ashish Singh wrote: > > > Jun, > > > > Your suggested approach works, will update the KIP and re-initiate > voting. > > Thanks! > > > > On Sun, Apr 3, 2016 at 8:37 AM, Jun Rao wrote: > > > >> Ashish, > >> > >>

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

suppose I want my Kafka installation to > > > > >>> support both > > > > >>> > > >> >> Kerberos and SmartCard mechanisms, can you tell me how > to > > > > >>> configure > > > > >>> > > >> >> Kafka? (as admin, not developer)? > > >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus, A while back, we had another proposal for the broker to just send the correlation id and an empty payload if it receives an unsupported version of the request. I didn't see that in the rejected section. It seems simpler than the current proposal where the client has to issue an ApiVersionR

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

java code is organized (do we process things in the KafkaApis layer, etc). > These are separate questions, right? > > -Jay > > On Fri, Apr 8, 2016 at 4:19 PM, Jun Rao wrote: > > > Hi, Magnus, > > > > You brought up a few things in your proposal. I am trying t

Re: [VOTE] KIP-4 Metadata Schema

;> >> > can't/don't return the topic. Though the topic may still exist if > its > >> >> not > >> >> > ready to be completely deleted or is in the process of being > deleted. > >> >> > > >> >> > Th

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

t the least is > needed but it could be more), right? I think this is what the authenticate > api in zookeeper does, right? > > -Jay > > On Sun, Apr 10, 2016 at 10:08 AM, Jun Rao wrote: > > > The way that we want to extend SASL mechanism is to first add the > mechanism

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

Ashish, A 3rd option is to in 0.10.0, just sanity check the principal type in the implementation of addAcls/removeAcls of Authorizer, but don't change the Authorizer api to add the getDescription() method. This fixes the immediate issue that an acl rule with the wrong principal type is silently ig

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Ismael, My responses are inlined below. On Sun, Apr 10, 2016 at 12:25 PM, Ismael Juma wrote: > Hi Jun, > > A couple of points below. > > On Sat, Apr 9, 2016 at 12:19 AM, Jun Rao wrote: > > > 5. Your main request is how can a client know that the broker is now

Re: [DISCUSS] KIP-35 - Retrieve protocol version

r? What if the C client just follows the java client model by implementing one version of protocol per C client release (which seems easier to implement)? Thanks, Jun On Fri, Apr 8, 2016 at 4:20 PM, Jun Rao wrote: > Magnus, > > A while back, we had another proposal for the brok

Re: [VOTE] KIP-33 - Add a time based log index

Hi, Jiangjie, Thanks for the update. Looks good to me overall. Just a few minor comments below. 10. On broker startup, it's not clear to me why we need to scan the log segment to retrieve the largest timestamp since the time index always has an entry for the largest timestamp. Is that only for re

Re: [DISCUSS] KIP-35 - Retrieve protocol version

n one case, we require ApiVersionRequest to happen before authentication and in another case we require the reverse. Thanks, Jun On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill wrote: > 2016-04-11 3:01 GMT+02:00 Jun Rao : > > > Thinking about ApiVersionRequest a bit more. There

Re: [DISCUSS] KIP-35 - Retrieve protocol version

ey Jun, see inline > > 2016-04-11 19:19 GMT+02:00 Jun Rao : > > > Hi, Magnus, > > > > Let me understand your proposal in more details just from the client's > > perspective. My understanding of your proposal is the following. > > > >

Re: [VOTE] KIP-4 Metadata Schema (Round 2)

Grant, Thanks for the updated version. +1 from me. Jun On Mon, Apr 11, 2016 at 10:42 AM, Grant Henke wrote: > Based on the discussion in the previous vote thread > < > http://search-hadoop.com/m/uyzND1xlaiU10QlYX&subj=+VOTE+KIP+4+Metadata+Schema > > > I also would like to include a behavior ch

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

mplementations. However, can we assist Sentry users to > avoid that breaking upgrade? I think it is worth a shot. If the changes are > not done by 0.10 code freeze, then sure lets punt it to next release. Does > this seem reasonable to you? > > On Sun, Apr 10, 2016 at 11:42 AM, Jun Rao w

Re: [DISCUSS] KIP-35 - Retrieve protocol version

ose messages may also help from a debugging perspective as would then > include client id and correlation id along with the message. > > Ismael > > On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao wrote: > > > Magnus, > > > > That sounds reasonable. To reduce the ch

Re: [VOTE] KIP-33 - Add a time based log index

question > > about the how we are loading the log segments, we can discuss it in the > PR) > > > > I will update the wiki to clarify the question raised in the comments and > > submit a PR by tomorrow. I am currently cleaning up the documentation. > > > > Thank

Re: [VOTE] KIP-33 - Add a time based log index

g. I commented on PR. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Thu, Apr 14, 2016 at 10:00 AM, Jun Rao wrote: > > > Hi, Jiangjie, > > > > 11. Rebuilding all missing time indexes will make the upgrade process > > longer since the log segments p

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

onse added by KIP-43. > I haven't changed the overall structure of the Java code. Feedback is > appreciated. > > Thanks, > > Rajini > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma wrote: > > > Hi Jun, > > > > Comments inline. > > > > O

Re: [VOTE] KIP-35: Retrieve protocol version - Round 2

Ashish, Just a couple of clarifications. 1. In ApiVersionRequest, we should get rid of ApiKeys since the request has an empty body, right? 2. In ApiVersionResponse, we should list ErrorCode before ApiVersions, right? Thanks, Jun On Thu, Apr 21, 2016 at 4:48 PM, Ashish Singh wrote: > Hey Guy

Re: [ANNOUNCE] New committer: Ismael Juma

Congratulations, Ismael! Jun On Mon, Apr 25, 2016 at 10:52 PM, Neha Narkhede wrote: > The PMC for Apache Kafka has invited Ismael Juma to join as a committer and > we are pleased to announce that he has accepted! > > Ismael has contributed 121 commits >

Re: [VOTE] KIP-57: Interoperable LZ4 Framing

Hi, Dana, Could you explain the following from the KIP a bit more? The KIP is intended to just fix the HC checksum, but the following seems to suggest there are other format changes? KafkaLZ4* code: - add naive support for optional header flags (ContentSize, ContentChecksum) to enable inte

Re: [VOTE] KIP-50: Move Authorizer to o.a.k.common package - Round 2

Ashish, Thanks for the proposal. +1 Jun On Mon, Apr 25, 2016 at 10:04 AM, Ashish Singh wrote: > Hey Guys, > > I would like to re-initiate the voting process for KIP-50: Move Authorizer > to o.a.k.common package. KIP-50 is available here > < > https://cwiki.apache.org/confluence/display/KAFKA/K

Re: [VOTE] KIP-57: Interoperable LZ4 Framing

ly skipped by a conformant decoder." We skip it. > > ContentChecksum is "Content Checksum validates the result, that all > blocks were fully transmitted in the correct order and without error, > and also that the encoding/decoding process itself generated no > distortion."

Re: [VOTE] 0.10.0.0 RC4

Tom, Thanks for reporting this. A few quick comments. 1. Did you send the right command for producer-perf? The command limits the throughput to 100 msgs/sec. So, not sure how a single producer can get 75K msgs/sec. 2. Could you collect some stats (e.g. average batch size) in the producer and see

Re: [VOTE] 0.10.0.0 RC4

can try those particular commits tomorrow. Before I try a bisect, > I'm going to replicate with a less intensive to iterate on smaller scale > perf test. > > Jun, inline: > > On Thursday, 12 May 2016, Jun Rao wrote: > > > Tom, > > > > Thanks for reporting thi

Re: [ANNOUNCE] New Kafka Committer Ewen Cheslack-Postava

Congratulations, Ewen! Well deserved. Jun On Tue, Dec 8, 2015 at 11:37 AM, Neha Narkhede wrote: > I am pleased to announce that the Apache Kafka PMC has voted to > invite Ewen Cheslack-Postava as a committer and Ewen has accepted. > > Ewen is an active member of the community and has contribute

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

Allen, Thanks for the proposal. A few comments. 1. Since this KIP changes the inter broker communication protocol (UpdateMetadataRequest), we will need to document the upgrade path (similar to what's described in http://kafka.apache.org/090/documentation.html#upgrade). 2. It might be useful to i

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

Hi, Becket, Thanks for the proposal. Looks good overall. A few comments below. 1. KIP-32 didn't say what timestamp should be set in a compressed message. We probably should set it to the timestamp of the latest messages included in the compressed one. This way, during indexing, we don't have to d

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

nt to positions in the corresponding .log file, not previous ones, right? Thanks, Jun On Mon, Dec 14, 2015 at 3:10 PM, Becket Qin wrote: > Hi Jun, > > Thanks a lot for the comments. Please see inline replies. > > Thanks, > > Jiangjie (Becket) Qin > > On Mon, Dec 1

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

g segment does not have a valid offset in current log > segment to point to. Maybe in that case we should build an empty log index. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Dec 14, 2015 at 5:51 PM, Jun Rao wrote: > > > 1. I was thinking more about sa

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Jiangjie, Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor comments on the KIP. KIP-32: 50. 6.c says "The log rolling has to depend on the earliest timestamp", which is inconsistent with KIP-33. 51. 8.b "If the time difference threshold is set to 0. The timestamp in the mes

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

oker security protocol in TopicMetadataResponse? > > > > > > > > The other option is to make it a separate KIP to make > > > > TopicMetadataResponse version aware and decide what to include, and > > make > > > > this KIP focus on the rack awa

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

: > Thanks a lot for the careful reading, Jun. > Please see inline replies. > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > Jiangjie, > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > > comments on the KIP. > >

Upcoming Kafka Summit this April

Hi, Everyone, I wanted to remind you of a couple of Kafka Summit related deadlines coming up. On Monday Jan 11 the Call for Proposals closes at 11:59pm Pacific Time. We encourage you to participate - share your stories and best practices. Submit your proposal at www.kafka-summit.org On Friday Ja

Kafka KIP meeting Jan. 12 at 11:00am PST

Hi, Everyone, We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to attend but haven't received an invite, please let me know. The following is the agenda. Agenda: KIP-36: Rack-aware replica assignment KIP-41: KafkaConsumer Max Records Thanks, Jun

Re: Kafka KIP meeting Jan. 12 at 11:00am PST

oting. The video will be uploaded soon in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals . Thanks, Jun On Mon, Jan 11, 2016 at 2:09 PM, Jun Rao wrote: > Hi, Everyone, > > We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to > a

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

KIP according to Jun's comment and included changes to TMR. > > > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao wrote: > > > >> Hi, Allen, > >> > >> A couple of minor comments on the KIP. > >> > >> 1. The version of the broker JSON

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

That's a good suggestion. However, it does not solve the problem for the > clients or thirty party tools that get broker information directly from > ZooKeeper. > > Thanks, > Allen > > > On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao wrote: > > > Allen, > >

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

a good suggestion. However, it does not solve the problem for > the > > > clients or thirty party tools that get broker information directly from > > > ZooKeeper. > > > > > > Thanks, > > > Allen > > > > > > > > > On Tue, Jan 12, 2016

Re: [VOTE] KIP-36 - Rack aware replica assignment

Allen, Thanks for updating the KIP wiki. +1 Could other people take a look at the latest proposal and vote on it? Thanks, Jun On Mon, Jan 4, 2016 at 4:30 PM, Allen Wang wrote: > I would like to call for a vote for KIP-36 - Rack aware replica assignment. > > The latest proposal is at > > http

Re: [VOTE] KIP-41: Consumer Max Records

Thanks for the proposal. +1. Jun On Wed, Jan 13, 2016 at 3:18 PM, Jason Gustafson wrote: > Hi All, > > I'd like to open up the vote on KIP-41. This KIP adds a new consumer > configuration option "max.poll.records" which sets an upper bound on the > number of records returned in a call to poll()

Re: KIP for extension of SASL to include additional mechanisms

Rajini, Thanks for your interest. I just gave you the permission to Kafka wiki. Jun On Thu, Jan 21, 2016 at 5:51 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Can I have access to write up a KIP for extending the SASL implementation > in Kafka to include more mechanisms? We have t

Kafka KIP meeting Jan 26 at 11:00am PST

Hi, Everyone, We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to attend but haven't received an invite, please let me know. The following is the agenda. Agenda: KIP-42: Add Producer and Consumer Interceptors Thanks, Jun

Re: Kafka KIP meeting Jan 26 at 11:00am PST

d soon in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals . Thanks, Jun On Mon, Jan 25, 2016 at 2:56 PM, Jun Rao wrote: > Hi, Everyone, > > We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to > attend but haven't received an

Kafka 0.9.0.1 plan

Hi, Everyone, We have fixed a few critical bugs since 0.9.0.0 was released and are still investigating a few more issues. The current list of issues tracked for 0.9.0.1 can be found below. Among them, only KAFKA-3159 seems to be critical. https://issues.apache.org/jira/issues/?jql=project%20%3D%2

0.9.0.1 RC1

This is the first candidate for release of Apache Kafka 0.9.0.1. This a bug fix release that fixes 70 issues. Release Notes for the 0.9.0.1 release https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, Feb. 16, 7pm PT Kafka's K

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Bill, Thanks for the proposal. A couple of comments. 1. It seems that this new policy should work for CreateTime as well. If a topic is configured with CreateTime, messages may not be added in strict order in the log. However, to build a time-based index, we will be maintaining the largest timest

Re: 0.9.0.1 RC1

Since the unit test failures are transient. +1 from myself. Thanks, Jun On Thu, Feb 11, 2016 at 6:55 PM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.9.0.1. This a > bug fix release that fixes 70 issues. > > Release Notes for the 0.9.0.1 rel

Re: 0.9.0.1 RC1

Thanks everyone for voting. The results are: +1 binding = 4 votes (Ewen Cheslack-Postava, Neha Narkhede, Joel Koshy and Jun Rao) +1 non-binding = 3 votes -1 = 0 votes 0 = 0 votes The vote passes. I will release artifacts to maven central, update the dist svn and download site. Will send out an

Re: 0.9.0.1 RC1

, Christian Posta wrote: > BTW, what's the etiquette for votes (non-binding) for this community? > welcomed? noise? > happy to see the non-binding votes, I'd like to contribute, just don't want > to pollute the vote call. thoughts? > thanks! > > On Tue, Feb

[ANNOUCE] Apache Kafka 0.9.0.1 Released

stava, Geoff Anderson, Grant Henke, Guozhang Wang, Gwen Shapira, Ismael Juma, Jaikiran Pai, James Cheng, Jason Gustafson, Jay Kreps, Jesse Anderson, Jiangjie Qin, Jin Xing, Jun Rao, Kim Christensen, Kishore Senji, Luciano Afranllie, Magnus Edenhill, Maksim Logvinenko, Mayuresh Gharat, Michael Blume,

Re: Please add me to the contributor list

Aarti, Thanks for your interest. Added to you to the contributor list. Jun On Fri, Feb 19, 2016 at 2:03 PM, Aarti Gupta wrote: > Hi folks! > > Want to start contributing to Kafka, > found some newbie JIRAs such as > https://issues.apache.org/jira/browse/KAFKA-3118 > > Please, can I be added t

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

works in a live system. Is there more discussion that > needs to be had about this KIP, or should I start a VOTE thread? > > > > On Tue, Feb 16, 2016 at 5:06 PM, Jun Rao wrote: > > > Bill, > > > > Thanks for the proposal. A couple of comments. > > > > 1.

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

; do that tonight. > > Bill > > On Fri, Feb 19, 2016 at 7:02 PM, Jun Rao wrote: > > > Hi, Bill, > > > > I replied with the following comments earlier to the thread. Did you see > > that? > > > > Thanks for the proposal. A couple of comments. > >

Kafka KIP meeting Feb 23 at 11:00am PST

Hi, Everyone, We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to attend but haven't received an invite, please let me know. The following is the agenda. Agenda: KIP-33 - Add a time based log index to Kafka KIP-47 - Add timestamp-based log deletion policy Thanks, Jun

Re: Kafka KIP meeting Feb 23 at 11:00am PST

eems to be in favor of making the next major release 0.10.0, instead of 0.9.1. The video will be uploaded soon in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals . Thanks, Jun On Mon, Feb 22, 2016 at 2:16 PM, Jun Rao wrote: > Hi, Everyone, > > We wi

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1. Thanks, Jun On Fri, Feb 19, 2016 at 11:00 AM, Becket Qin wrote: > Hi All, > > We would like to start this voting thread on making next Kafka release > 0.10.0.0 instead of 0.9.1.0. > > The next Kafka release will have several significant important new > features/changes such as Kafka Stream

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

Hi, Becket, I thought that your proposal to build time-based index just based off index.interval.bytes is reasonable. Is there a particular need to also add time. index.interval.bytes? Compute the pre-allocated index file size based on log segment file size can be useful. However, the tricky thin

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Rajini, Thanks for the updates. Just a couple of minor comments. 1. With the default GSSAPI, what's the first packet that the client sends to the server? Is that completely different from the packet format that we will use for non-GSSAPI mechanisms? 2. In the server response, it doesn't seem tha

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

checksum. This is completely different from the mechanism packet from > Kafka >clients which start with a two-byte version set to zero currently, > followed >by just a String mechanism. >2. Agreed, I have removed the version from the server response in the >KIP. Thank

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

ing startup > we will remove all the segments and roll out a new empty segment. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Feb 29, 2016 at 6:09 PM, Jun Rao wrote: > > > Hi, Becket, > > > > I thought that your proposal to

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

previously > latest timestamp has already been deleted, there is no need to remember > that any more. > > That said, I believe this corner case is really because user is not > configuring the acceptable time difference threshold appropriately. > > Thanks, > > Jiangjie (Bec

Re: [VOTE] Release plan - Kafka 0.10.0

+1 Thanks, Jun On Mon, Mar 7, 2016 at 9:27 AM, Gwen Shapira wrote: > Greetings Kafka Developer Community, > > As you all know, we have few big features that are almost complete > (Timestamps! Interceptors! Streams!). It is time to start planning our > next release. > > I suggest the following:

Re: Review Request 34492: Patch for KAFKA-2210

> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote: > > core/src/main/scala/kafka/security/auth/Acl.scala, lines 57-62 > > > > > > I tried exactly that but it tunrs out our current Json parser does not > > work when

Kafka KIP hangout June 9

Hi, Everyone, We will have a KIP hangout at 11 PST on June 9. The following is the agenda. If you want to attend and is not on the invite, please let me know. Agenda: KIP-12 (sasl/ssl authentication): status check KIP-4: keeping ISR in TopicMetadataResponse KIP-25: (system test) Thanks, Jun

Re: Review Request 34789: Patch for KAFKA-2168

4789/#comment139578> To be consistent with the naming convention with the rest of the methods, should we just name it offsetRestNeeded()? - Jun Rao On June 5, 2015, 7:45 p.m., Jason Gustafson wrote: > > --

Re: Review Request 34789: Patch for KAFKA-2168

> On June 9, 2015, 6:49 p.m., Jay Kreps wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > line 322 > > > > > > Do we need this? There is no real guarantee on the poll time, so

Re: Review Request 34789: Patch for KAFKA-2168

> On June 9, 2015, 7:58 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > line 1212 > > <https://reviews.apache.org/r/34789/diff/8/?file=980075#file980075line1212> > > > > -1 makes the pollClient b

Re: Review Request 34789: Patch for KAFKA-2168

> On June 9, 2015, 7:58 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > line 1212 > > <https://reviews.apache.org/r/34789/diff/8/?file=980075#file980075line1212> > > > > -1 makes the pollClient b

Re: KIP Call Notes

Jayesh, Most of the metadata you mentioned already exist. For example, ISR is already in the TopicMetadataResponse and the high water mark can be returned from the OffsetRequest. The details of the current wire protocol can be found at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+T

Re: Review Request 34789: Patch for KAFKA-2168

> On June 9, 2015, 7:58 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, > > line 1212 > > <https://reviews.apache.org/r/34789/diff/8/?file=980075#file980075line1212> > > > > -1 makes the pollClient b

Re: Review Request 33204: Patch for KAFKA-1646 merge to latest trunk

SEt => FileMessageSet core/src/test/scala/unit/kafka/log/FileMessageSetTest.scala <https://reviews.apache.org/r/33204/#comment139769> clear shutdown => cleanly shut down core/src/test/scala/unit/kafka/log/LogSegmentTest.scala <https://reviews.apache.org/r/33204/#comment1397

Re: [VOTE] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

+1 Thanks, Jun On Tue, Jun 9, 2015 at 11:24 PM, Honghai Chen wrote: > Hi Kafka, > > After a long discussion, please help vote again for the > KIP. Thanks. > > > > I wrote a KIP for this after some discussion on KAFKA-1646. > https://issues.apache.org/jira/browse/KAFKA-1646 > >

Re: Review Request 35201: Fix KAFKA-2253

if you have 1000 requests/sec and each request has 1000 partitions, that lock is going to be access 1million times in a sec. Could you do some tests/profiling before and after we introduced the global lock to see if this could be an issue? - Jun Rao On June 8, 2015, 6:47 p.m., Guozhang

Re: Review Request 35201: Fix KAFKA-2253

> On June 11, 2015, 1:07 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 > > <https://reviews.apache.org/r/35201/diff/2/?file=980805#file980805line264> > > > > Not sure if we need this check. Since all writes t

Re: Review Request 35347: Patch for KAFKA-2249

even need this method and just let LogConfig access KafkaConfig.originals(). KafkaConfig.getMetricClasses() uses toProps, but is never used. We can just remove that method. Also, perhaps it's useful and convenient to turn LogConfig into an AbstractConfig too. - Jun Rao

Re: Review Request 35261: Patch for KAFKA-2232

t; We should pass in the key/value object, instead of null to the partitioner. - Jun Rao On June 9, 2015, 7 p.m., Alexander Pakulov wrote: > > --- > This is an automatically generated e-mail. To reply, visit

Re: [VOTE] KIP-25 System test improvements

+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson wrote: > Hi Kafka, > > After a few rounds of discussion on KIP-25, there doesn't seem to be > opposition, so I'd like to propose a vote. > > Thanks, > Geoff > > On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson > wrote: > > >

Re: [VOTE] KIP-25 System test improvements

+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson wrote: > Hi Kafka, > > After a few rounds of discussion on KIP-25, there doesn't seem to be > opposition, so I'd like to propose a vote. > > Thanks, > Geoff > > On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson > wrote: > > >

Re: [Discussion] New Consumer API / Protocol

Guozhang, Perhaps we can discuss this in our KIP hangout next week? Thanks, Jun On Tue, Jun 9, 2015 at 1:12 PM, Guozhang Wang wrote: > This email is to kick-off some discussion around the changes we want to > make on the new consumer APIs as well as their semantics. Here are a > not-comprehen

Re: Review Request 35371: Patch for KAFKA-2266

/src/test/java/org/apache/kafka/common/network/SelectorTest.java <https://reviews.apache.org/r/35371/#comment140081> Waiting a second on this could be long and is not always reliable. Perhaps, we can introduce a waitUntil util like in kafka.utils.TestUtils and use it here. - Jun Rao O

Re: Review Request 33614: Patch for KAFKA-2132

log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java <https://reviews.apache.org/r/33614/#comment140086> We can just use ConfigException. - Jun Rao On April 30, 2015, 10:53 p.m., Ashish Singh wrote: > > --

<    4   5   6   7   8   9   10   11   12   13   >