Re: Protocol documentation draft

2012-12-04 Thread Jun Rao
Thanks for drafting this. Made some minor edits and added some comments. Thanks, Jun On Thu, Nov 29, 2012 at 3:17 PM, Jay Kreps jay.kr...@gmail.com wrote: I started trying to document the 0.8 protocol from the code and write a guide to client implementation. This is meant to be a more

Re: Objections to doing this in 0.8?

2012-12-10 Thread Jun Rao
That sounds goods to me. Thanks, Jun On Mon, Dec 10, 2012 at 3:05 PM, Jay Kreps jay.kr...@gmail.com wrote: https://issues.apache.org/jira/browse/KAFKA-667 Goal would just be forward compatibility with a more sane naming scheme... -Jay

can't commit to Kafka svn

2012-12-12 Thread Jun Rao
Hi, I can't seem to commit to Kafka svn since this morning. The commit is on a fresh checkout. Do you know what the issue is? jrao-mn:0.8_top_temp jrao$ svn commit --username junrao -m trivial fix to increase offset commit interval to reduce ZK load Sending

Re: can't commit to Kafka svn

2012-12-13 Thread Jun Rao
the tires before we cut over? -Jay On Wed, Dec 12, 2012 at 6:34 PM, Gavin McDonald ga...@16degrees.com.auwrote: Issue is you are migrating to git. ** ** https://issues.apache.org/jira/browse/INFRA-5111 ** ** ** ** ** ** *From:* Jun Rao [mailto:jun...@gmail.com] *Sent

Re: can't commit to Kafka svn

2012-12-13 Thread Jun Rao
- origin/trunk origin/consumer_redesign origin/legacy_client_libraries origin/trunk On Thu, Dec 13, 2012 at 9:49 AM, Jun Rao jun...@gmail.com wrote: https://git-wip-us.apache.org/repos/asf/kafka.git seems to be pointing to trunk. Do you know where is the 0.8 branch? Thanks

Re: can't commit to Kafka svn

2012-12-13 Thread Jun Rao
maintenance. remote: I've updated the ticket to see if they are really undergoing maintenance. On Thu, Dec 13, 2012 at 10:07 AM, Jun Rao jun...@gmail.com wrote: Ok. Thanks. Is the idea to leave site under svn after the git move? Thanks, Jun On Thu, Dec 13, 2012 at 9:55 AM, Joel

Re: Offset commit api

2012-12-18 Thread Jun Rao
Thanks for the proposal. Added a couple of comments to the wiki. Thanks, Jun On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps jay.kr...@gmail.com wrote: Hey Guys, David has made a bunch of progress on the offset commit api implementation. Since this is a public API it would be good to do as

Re: git workflow

2013-01-03 Thread Jun Rao
Thanks for documenting this. Could you also add how to resolve conflicts during rebase? Jun On Wed, Jan 2, 2013 at 1:45 PM, Jay Kreps jay.kr...@gmail.com wrote: I don't know about other people but I find git kind of confusing. I thought it would be useful to try to document the normal

Re: kafka 0.8 producer throughput

2013-01-08 Thread Jun Rao
The 50MB/s number is for 0.7. We haven't carefully measured the performance in 0.8 yet. We do expect the throughput that a single producer can drive in 0.8 to be less. This is because the 0.8 producer needs to wait for an RPC response from the broker while in 0.7, there is no ack for the producer.

Re: kafka 0.8 producer throughput

2013-01-09 Thread Jun Rao
broker? Can this be disabled or it's a major design change? On Wed, Jan 9, 2013 at 12:40 AM, Jun Rao jun...@gmail.com wrote: The 50MB/s number is for 0.7. We haven't carefully measured the performance in 0.8 yet. We do expect the throughput that a single producer can drive in 0.8

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

2013-01-14 Thread Jun Rao
Basically, we spread partitions among multiple brokers. If a message is sent without a key, the producer picks a random partition to balance the load. If a message has a key, the default partitioner hashes the key to one of the partitions deterministically. Then, the load may not always be

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

2013-01-15 Thread Jun Rao
All apis for 0.8 are documented in https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol However, there is no need for a producer to call the metadata api directly. Our producer client does this for you automatically. For your first question, if a new broker is added,

Re: About kafka 0.8 producer auto detect broker

2013-01-15 Thread Jun Rao
That's an experiment for a future release. The consumer logic in 0.8 is more or less the same as 0.7, ie, running in a decentralized way. Thanks, Jun On Tue, Jan 15, 2013 at 12:10 AM, gj1989lh gj198...@163.com wrote: Hi, How is the consumer rebalancing works? In the design part of the

Re: Need mailing list moderators

2013-01-15 Thread Jun Rao
I am already moderating the mailing list. Thanks, Jun On Tue, Jan 15, 2013 at 7:37 AM, Alan Cabrera a...@toolazydogs.com wrote: It seems that I am the only moderator for the Kafka mailing lists. We need some volunteers from the PMC to take my place, at least two. Regards, Alan

Re: dynamic topic config

2013-01-17 Thread Jun Rao
Posted a couple of comments on the wiki. Thanks, Jun On Wed, Jan 16, 2013 at 8:18 PM, Jay Kreps jay.kr...@gmail.com wrote: I added a first pass at a proposal for moving topic configuration into zookeeper so that you would specify the config when you created the topic and config could be

Re: zookeeper partition data

2013-01-17 Thread Jun Rao
Using separate ZK paths for replicas assignment and configs seems better since it allows each of them to be updated independently. Yes, we should make a path of values stored in ZK for future extension. Could you file a jira? Thanks, Jun On Wed, Jan 16, 2013 at 7:47 PM, Jay Kreps

Re: Abou Kafka 0.8 producer throughput test

2013-01-17 Thread Jun Rao
We also did some perf test on 0.8 using the following command. All configs on the broker are the defaults. bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list localhost:9092 --initial-message-id 0 --messages 200 --topics topic_001 --request-num-acks -1 --batch-size 100

Re: kafka pull request: KAFKA-294

2013-01-31 Thread Jun Rao
The change looks good. Would you mind attaching a patch to the jira? This will indicate that you are ok with donating the code to Apache. Thanks, Jun On Wed, Jan 30, 2013 at 1:21 PM, fsaintjacques g...@git.apache.org wrote: GitHub user fsaintjacques opened a pull request:

Re: kafka replication blog

2013-02-07 Thread Jun Rao
and get up to speed on the current replication architecture's design. -- Felix On Tue, Feb 5, 2013 at 2:21 PM, Jun Rao jun...@gmail.com wrote: I just posted the following blog on Kafka replication. This may answer some of the questions that a few people have

Re: Typo's...

2013-02-11 Thread Jun Rao
Netz, Thanks a lot. I just updated the site with those changes. Jun On Thu, Feb 7, 2013 at 10:17 PM, Nelz nelz9...@gmail.com wrote: Hello, I was recently doing some reading of your materials, and I found a couple of typo's. In the spirit of Open Source, I thought I'd just jot a quick note

Clients and partition leaders

2013-02-12 Thread Jun Rao
David, The benefit of the strategy used in Solr is that it simplifies client routing. The downside is potential additional RPC overhead and a bit more logic in the server. Technically, you can achieve what Solr does in the client layer too. You can run a proxy that runs the java version of Kafka

Re: Plans for a 0.7.3 release?

2013-02-15 Thread Jun Rao
If there is enough interest, we could do a 0.7.3 release. However, there weren't that many fixes after 0.7.2 and 0.8 will probably be out in Mar. I am wondering if it's worth doing another 0.7 release. Thanks, Jun On Thu, Feb 14, 2013 at 11:31 AM, David Arthur mum...@gmail.com wrote: I think

Re: Plans for a 0.7.3 release?

2013-02-18 Thread Jun Rao
it see suller adoption having the best posible .7.x in place give folks confidence in the overall Kafka project. Joe On Fri, Feb 15, 2013 at 10:14 PM, Jun Rao jun...@gmail.com wrote: If there is enough interest, we could do a 0.7.3 release. However, there weren't that many fixes after

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

2013-02-27 Thread Jun Rao
Did you upgrade your console consumer? Thanks, Jun On Wed, Feb 27, 2013 at 3:40 AM, ben fleis b...@monkey.org wrote: Sorry, I should have clarified something -- using the normal ConsoleConsumer does *not* produce this error. So something in my formatter, or in the logic of a custom

Re: Flume-NG Kafka Channel

2013-03-04 Thread Jun Rao
Jonathan, My understanding is that channel is a local concept in Flume. So, are you using a single local Kafka broker to build channel? Will it make more sense to build a Kafka sink in Fume? Thanks, Jun On Fri, Mar 1, 2013 at 10:56 PM, Jonathan Creasy j...@box.com wrote: I started hacking on

Re: Producer-side persistence and re-sending

2013-03-04 Thread Jun Rao
Matan, Thanks for picking this up. Yes, this is likely a post 0.8 item and should go into trunk. Could you file a jira and post your initial design there for discussion? Jun On Sat, Mar 2, 2013 at 12:53 PM, matan ma...@cloudaloe.org wrote: Hi, I am designing a patch on top the 0.8 code

Apache Kafka git unavailable this morning

2013-03-06 Thread Jun Rao
Hi, It seems that git push or pull hang on Apache Kafka git repo this morning. Could you help? Thanks, Jun

Re: Apache Kafka git unavailable this morning

2013-03-06 Thread Jun Rao
Thanks, Daniel. It works now. Jun On Wed, Mar 6, 2013 at 10:02 AM, Daniel Shahaf d...@daniel.shahaf.namewrote: Should work now Jun Rao wrote on Wed, Mar 06, 2013 at 09:34:34 -0800: Hi, It seems that git push or pull hang on Apache Kafka git repo this morning. Could you help

Re: How do we get involved in beta testing Kafka 0.8?

2013-03-17 Thread Jun Rao
Darren, Thanks for volunteering for testing 0.8. The current stable version of 0.8 can be found in the 0.8 branch (NOT trunk) of Kafka Apache git repo. You can follow https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.8+Quick+Start to get things started. At LinkedIn, we have been testing

Re: Question about Kafka code contribution guidelines

2013-03-28 Thread Jun Rao
All committers have to file the following agreement: http://www.apache.org/licenses/icla.txt Contributor submits patches to a jira. The patches will be reviewed and if accepted, they will be checked in by a committer. If a contributor makes significant contribution, he/she will be nominated as a

Re: How to create a deployable release?

2013-04-01 Thread Jun Rao
Probably we should add this as a subtask for the 0.8 beta release? Thanks, Jun On Mon, Apr 1, 2013 at 8:46 AM, Neha Narkhede neha.narkh...@gmail.comwrote: Cosmin, The release-zip target worked for a short while but has been broken for a long time now. Its purpose was to create a deployable

Re: 0.8 blockers

2013-04-04 Thread Jun Rao
David, The release related blockers are listed in https://issues.apache.org/jira/browse/KAFKA-833. Do you think you can help out on https://issues.apache.org/jira/browse/KAFKA-843? Thanks, Jun On Thu, Apr 4, 2013 at 6:29 AM, David Arthur mum...@gmail.com wrote: What issues are blocking the

pushing delete topic feature out of 0.8

2013-04-04 Thread Jun Rao
Hi, We started the work on deleting topics in 0.8 (kafka-330). We realized that it touches quite a few critical components such as controller, replica manager, and log, and it will take some time to stabilize this. In order not to delay the 0.8 release too much, I propose that we push this

Re: Thoughts on using Ant+Ivy for the build?

2013-04-07 Thread Jun Rao
David, We'd be open to a new build tool if SBT is deemed too hard to use. We can revisit this post 0.8. Do you want to open a jira to track this? Thanks, Jun On Fri, Apr 5, 2013 at 12:55 PM, David Arthur mum...@gmail.com wrote: After getting frustrated with SBT, and being unable to figure

Re: Kafka and Scala versions

2013-04-07 Thread Jun Rao
Scott, Yes, we are still using scala 2.8.0 at LinkedIn. We do plan to upgrade to a later version of scala at some point. However, this may take some time since we will need to upgrade other scala projects at LinkedIn at the same time. It's unfortunate that code change is needed in order to

Re: Kafka and Scala versions

2013-04-09 Thread Jun Rao
that differs, instead of ~1500 lines). See my recent patches in KAFKA-717. On 4/7/13 10:17 PM, Jun Rao jun...@gmail.com wrote: Scott, Yes, we are still using scala 2.8.0 at LinkedIn. We do plan to upgrade to a later version of scala at some point. However, this may take some time since we

Re: Permission to translate your page at http://kafka.apache.org/

2013-04-11 Thread Jun Rao
That should be fine. Thanks for doing this. Jun On Wed, Apr 10, 2013 at 1:36 PM, Anja Skrba an...@webhostinggeeks.comwrote: Dear Sir, I am writing to inquire regarding your web page about Kafka where I have found a lot of useful information. My name is Anja and I'm currently studying at

Re: interesting paper on log replication

2013-04-14 Thread Jun Rao
Thanks for the link. This paper provides an alternative, but similar implementation to that in Zookeeper. The key difference seems to be that the former supports membership reconfiguration. Kafka replication is simpler because it separates the leader election part from log replication. Such

Re: Kafka branching model

2013-04-15 Thread Jun Rao
Cosmin, At this moment, 0.8 is only open for blockers. Everything else should go to trunk. We plan to merge changes from 0.8 to trunk as we stabilize 0.8. Thanks, Jun On Mon, Apr 15, 2013 at 7:31 AM, Cosmin Lehene cleh...@adobe.com wrote: Hi, The documentation on the website mentions

Re: interesting paper on log replication

2013-04-16 Thread Jun Rao
, where we allow the logs to divert and don't reconcile perfectly. Thanks, Neha On Sun, Apr 14, 2013 at 9:42 PM, Jun Rao jun...@gmail.com wrote: Thanks for the link. This paper provides an alternative, but similar implementation to that in Zookeeper. The key difference seems

Re: Kafka 0.8 release schedule?

2013-04-17 Thread Jun Rao
At this moment, the only blocker for the 0.8 beta release is the doc (KAFKA-834). We will try to get those docs updated. We can plan to do the 0.8 beta release next week. Thanks, Jun On Wed, Apr 17, 2013 at 9:21 AM, Dragos Manolescu dragos.manole...@servicenow.com wrote: Hi -- As far as I

Re: kafka messages out of order

2013-04-21 Thread Jun Rao
In 0.7, we create a new socket after reconnect.interval messages. Messages sent through different sockets may not arrive at the broker in order. The reason to have reconnect.interval is for the producer to get a chance to see the new broker added to a load balancer (if one uses a VIP to send

Re: SVN or Git

2013-05-05 Thread Jun Rao
) I apologize, please let me know the right way to contribute. Cheers, Andrea On 04/24/2013 04:56 PM, Jun Rao wrote: We have moved to git. Updated the wiki. Thanks, Jun On Wed, Apr 24, 2013 at 6:41 AM, Andrea Gazzarini andrea.gazzar...@gmail.com wrote: Hi, looking at these pages

0.8 beta release status

2013-05-08 Thread Jun Rao
We are investigating a metadata related issue when there are a large number of clients (1000+). This issue, if not resolved, may cause the whole cluster to be unavailable. We are testing a fix. Once the issue is resolved, we can start the release process. Thanks, Jun

Re: produce request wire format question

2013-05-22 Thread Jun Rao
Dave, Currently, the broker expects each producer request to specify the exact partition id (-1 is on longer valid). The mapping from a message to a partition is done at the producer client. The producer can choose a random partition (from the existing list of partitions) or deterministically

Re: metadata request/response question

2013-06-12 Thread Jun Rao
It will only send those brokers on which those topics are assigned. Thanks, Jun On Wed, Jun 12, 2013 at 2:24 PM, Dave Peterson dspeter...@tagged.comwrote: Suppose I send a metadata request for a single topic or some small subset of the entire set of available topics. Will the response

Re: [VOTE] Kafka 0.8.0 Beta 1 (Candidate 1)

2013-06-13 Thread Jun Rao
+1. Verified unit tests and quick start. Thanks, Jun On Thu, Jun 13, 2013 at 2:41 PM, Joe Stein crypt...@gmail.com wrote: Hello, this is the first candidate release for Kafka 0.8.0 Beta 1 This release fixes the following issues

Re: [VOTE] Kafka 0.8.0 Beta 1 (Candidate 1)

2013-06-19 Thread Jun Rao
Hi, Joe, When do you plan to cut an RC2 for vote? Thanks, Jun On Mon, Jun 17, 2013 at 4:58 AM, Joe Stein crypt...@gmail.com wrote: Hey folks, I am going to vote -1 on this this release. I know, I know, I know I prepared it but I was working for hours and hours last night going through so

Kafka User Group Meeting

2013-06-21 Thread Jun Rao
Hi, Everyone, We will be having our user group meeting next Thursday evening at LinkedIn in Mountain View. The following is the venue and the tentative agenda. http://www.meetup.com/http-kafka-apache-org/events/125887332/ Hope to see many of you there! Jun

Re: FAQ

2013-06-24 Thread Jun Rao
Yes, I think that would be better. Thanks, Jun On Mon, Jun 24, 2013 at 10:30 AM, Jay Kreps jay.kr...@gmail.com wrote: I have noticed we don't do a good job of updating the FAQs. Would we do better if I migrated it to the wiki so it was easier to edit? -Jay

Re: Helping out

2013-06-25 Thread Jun Rao
As for offset management, we have a summer intern starting this week and our plan is to have the intern work on this. Thanks, Jun On Tue, Jun 25, 2013 at 7:53 AM, Sam Meder sam.me...@jivesoftware.comwrote: On Jun 24, 2013, at 9:41 AM, Jay Kreps jay.kr...@gmail.com wrote: Hey Sam,

Re: Helping out

2013-06-25 Thread Jun Rao
Sam, Also, just want to say thank you for fixing kafka-938. That's an awesome fix! Jun On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder sam.me...@jivesoftware.comwrote: Hey, I now have roughly a day a week I can dedicate to working on Kafka, so I am looking for issues in the 0.8.1 batch that

Kafka User Group Meeting Jun. 27

2013-06-26 Thread Jun Rao
Hi, Everyone, We have finalized our agenda of the meetup Thursday evening, with Speakers from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the meetup link below. For remote people, we will post the streaming link at meetup before the meeting and we will use our IRC for remote

Re: [VOTE] Apache Kafka 0.8.0-beta1 candidate 2

2013-06-27 Thread Jun Rao
! On Thu, Jun 20, 2013 at 8:25 AM, Jun Rao jun...@gmail.com wrote: +1. Verified both unit tests and quick start. Thanks, Jun On Wed, Jun 19, 2013 at 9:28 PM, Joe Stein crypt...@gmail.com wrote: This is the second release candidate vote for the Apache Kafka 0.8.0-beta1

Re: criteria for fixes on 0.8 beta

2013-07-02 Thread Jun Rao
My vote is that a patch can go into 0.8 if (1) it fixes a critical issue or (2) the change is trivial and it makes the 0.8 experience better (e.g., improving log4j readability). kafka-946 may fall into (2). Thanks, Jun On Tue, Jul 2, 2013 at 7:34 PM, Joe Stein crypt...@gmail.com wrote: How

Re: Helping out

2013-07-03 Thread Jun Rao
Sam, Interested in taking a look at KAFKA-959https://issues.apache.org/jira/browse/KAFKA-959 ? Thanks, Jun On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder sam.me...@jivesoftware.comwrote: Hey, I now have roughly a day a week I can dedicate to working on Kafka, so I am looking for issues in

Re: Helping out

2013-07-03 Thread Jun Rao
/DefaultEventHandler.scala#L152 On Wed, Jul 3, 2013 at 2:18 PM, S Ahmed sahmed1...@gmail.com wrote: Jun, Looks like the update would happen around here? On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao jun...@gmail.com wrote: Sam, Interested in taking a look at KAFKA-959https://issues.apache.org

Re: having problem with 0.8 gzip compression

2013-07-09 Thread Jun Rao
On Tue, Jul 9, 2013 at 8:06 AM, Jun Rao jun...@gmail.com wrote: Could you try starting the consumer first (and enable gzip in the producer)? Thanks, Jun On Mon, Jul 8, 2013 at 9:37 PM, Scott Wang scott.w...@rumbleentertainment.com wrote: No, I did not start the consumer

Re: Make documentation part of new features acceptance criteria?

2013-07-10 Thread Jun Rao
Cosmin, That's a good idea. In the past, for major new features, we tend to create a wiki page to outline the design. The wiki pages can be organized better. Is this what you are looking for? Thanks, Jun On Wed, Jul 10, 2013 at 1:17 AM, Cosmin Lehene cleh...@adobe.com wrote: I'm not sure if

Re: Running 0.8 against 0.7 data and zk

2013-07-11 Thread Jun Rao
No. If you point an 0.8 broker to a ZK with 0.7 layout, it will still hit some weird exception, maybe not exactly the same as specified in the jira. Thanks, Jun On Thu, Jul 11, 2013 at 3:15 PM, Jay Kreps jay.kr...@gmail.com wrote: This JIRA has a patch but I thought we already did something

Re: Kafka User Group Meeting Jun. 27

2013-07-12 Thread Jun Rao
to get the recording there, nm. -Jay On Tue, Jul 2, 2013 at 12:47 PM, Jay Kreps jay.kr...@gmail.com wrote: The recording for the user group talks is available here: http://www.ustream.tv/linkedin-events -Jay On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao jun...@gmail.com wrote: Hi

Re: Meetup in Raleigh/Durham, NC

2013-07-21 Thread Jun Rao
David, Thanks for sharing this. Jun On Thu, Jul 18, 2013 at 9:19 PM, David Arthur mum...@gmail.com wrote: There is a Hadoop meetup happening in Durham next week. I'm presenting an intro to Kafka. I don't suspect there are any Kafka users in the area who are not already members of TriHUG,

Re: Does Kafka message manager tool (read messages from console, delete messages) exist?

2013-07-23 Thread Jun Rao
For 1, you can write a tool that uses our getOffsetBefore api and SimpleConsumer. The offset returned is at the log segment boundary. So it's not going to match the specified time precisely. For 2, you can configure a topic to be rolled by timestamp (log.roll.hours and topic.log.roll.hours). Then

Re: [VOTE] Logo runoff vote

2013-07-31 Thread Jun Rao
+1 for 296 Thanks, Jun On Wed, Jul 31, 2013 at 1:34 PM, Jay Kreps jay.kr...@gmail.com wrote: Okay folks we did a survey on the JIRA (KAFKA-982) and it was more or less a tie between two logos: https://issues.apache.org/jira/secure/attachment/12593545/296.png

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Jun Rao
) Not sure is message.max.bytes only counts payload or whole message + any headers, so it may be that it should be a bit larger even. /Sam On Aug 1, 2013, at 7:04 AM, Jun Rao jun...@gmail.com wrote: server: replica.fetch.max.bytes should be = message.max.bytes. Otherwise, the follower will get

Re: Two patches needing review

2013-08-04 Thread Jun Rao
Sorry for the delay. Reviewed both. Jun On Sat, Aug 3, 2013 at 10:52 AM, Jay Kreps jay.kr...@gmail.com wrote: This is a set of good housekeeping things for 0.8 open source release: https://issues.apache.org/jira/browse/KAFKA-718 This is just a small set of trivial cleanups that are very

Re: All Lead Replica are on a Single Broker

2013-08-14 Thread Jun Rao
I think Joel answered a similar question in the user mailing list. You can run the preferred replica leader election tool after the restarts ( https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools) Thanks, Jun On Wed, Aug 14, 2013 at 6:22 AM, Kumar, Vaibhav

Re: kafka pull request: Escape the . in the kafka.Kafka chain

2013-08-27 Thread Jun Rao
Could you create a jira in Kafka and attach a patch there? This is to make sure that you grant permission to Apache. Thanks, Jun On Tue, Aug 27, 2013 at 9:04 AM, vpernin g...@git.apache.org wrote: GitHub user vpernin opened a pull request: https://github.com/apache/kafka/pull/8

Re: Review Request 14013: Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

2013-09-08 Thread Jun Rao
? core/src/main/scala/kafka/utils/Annotations_2.9+.scala https://reviews.apache.org/r/14013/#comment50707 How does this affect IDEs like Intellij? Are you able to build the project in IDE with both version of threadsafe annotation? - Jun Rao On Sept. 8, 2013, 9:47 p.m., Christopher Freeman

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread Jun Rao
This means the broker somehow closed the socket connection. Anything in the broker log around the same time? Thanks, Jun On Wed, Sep 11, 2013 at 6:07 PM, prashant amar amasin...@gmail.com wrote: Also noticed another issue Specified below is the current configuration Topic1 - n Partitions

preparing for the 0.8 final release

2013-09-13 Thread Jun Rao
Hi, Everyone, We have been stabilizing the 0.8 branch since the beta1 release. I think we are getting close to an 0.8 final release. I made an initial list of the remaining jiras that should be fixed in 0.8.

Re: [VOTE] Bylaws!

2013-09-13 Thread Jun Rao
+1 Jun On Fri, Sep 13, 2013 at 9:53 AM, Jay Kreps jay.kr...@gmail.com wrote: It was pointed out to us that Apache projects should formalize their bylaws--basically how votes are done, who votes on what, and what the criteria are for voting. We have been acting informally so far and haven't

Re: Random Partitioning Issue

2013-09-14 Thread Jun Rao
Joe, Thanks for bringing this up. I want to clarify this a bit. 1. Currently, the producer side logic is that if the partitioning key is not provided (i.e., it is null), the partitioner won't be called. We did that because we want to select a random and available partition to send messages so

Re: Random Partitioning Issue

2013-09-22 Thread Jun Rao
of a partitioner interface is to make it possible to plug in non-standard behavior like this, right? -Jay On Sat, Sep 14, 2013 at 8:15 PM, Jun Rao jun...@gmail.com wrote: Joe, Thanks for bringing this up. I want to clarify this a bit. 1. Currently, the producer side logic

Re: Random Partitioning Issue

2013-09-27 Thread Jun Rao
wanted on this extra item (including the discussion) to 0.8.1 or greater? I am always of the mind of reduce complexity unless that complexity is in fact better than not having it. On Sun, Sep 22, 2013 at 8:56 PM, Jun Rao jun...@gmail.com wrote: It's reasonable to make the behavior of random

Re: Random Partitioning Issue

2013-09-27 Thread Jun Rao
} } /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Fri, Sep 27, 2013 at 12:46 PM, Jun Rao jun...@gmail.com wrote: Joe

Re: Random Partitioning Issue

2013-09-27 Thread Jun Rao
, right? private class RandomMetaRefreshPartitioner[T](props: VerifiableProperties = null) extends Partitioner[T] { def partition(key: T, numPartitions: Int): Int = { Utils.abs(key.hashCode) % numPartitions } } On Fri, Sep 27, 2013 at 1:10 PM, Jun Rao jun...@gmail.com wrote

Re: Random Partitioning Issue

2013-09-29 Thread Jun Rao
in DefaultEventHandler, and do that in partitioner. One thing about this idea is any customized partitioner also has to consider key == null case then. Guozhang On Fri, Sep 27, 2013 at 9:12 PM, Jun Rao jun...@gmail.com wrote: We have the following code in DefaultEventHandler: val partition

Review Request 14399: OfflinePartitionCount metrics may be incorrect after the controller failover

2013-09-29 Thread Jun Rao
/browse/KAFKA-1068 Repository: kafka Description --- kafka-1068 Diffs - core/src/main/scala/kafka/controller/KafkaController.scala aef41ad5f087817862dcbd9dfbb4674e49e21f91 Diff: https://reviews.apache.org/r/14399/diff/ Testing --- Thanks, Jun Rao

Re: Random Partitioning Issue

2013-10-01 Thread Jun Rao
, 2013 at 12:14 PM, Jun Rao jun...@gmail.com wrote: The main issue is that if we do that, when key is null, we can only select a random partition, but not a random and available partition, without changing the partitioner api. Being able to do the latter is important in my opinion

Re: Random Partitioning Issue

2013-10-01 Thread Jun Rao
://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Oct 1, 2013 at 11:26 AM, Jun Rao jun...@gmail.com wrote: This proposal still doesn't address the following fundamental issue: The random partitioner cannot

Re: Registering #kafka channel

2013-10-02 Thread Jun Rao
I will follow up on this. Thanks, Jun On Wed, Oct 2, 2013 at 5:56 AM, David Arthur mum...@gmail.com wrote: This morning I noticed that the topic in the #kafka IRC channel was thanks., so I changed it to something more appropriate. This led me to discover that #kafka has no OPs and is not

Review Request 14496: Patch for KAFKA-1073

2013-10-04 Thread Jun Rao
f333d29bf36bb7fdc66b3bf9af16e7ee19ad7e48 Diff: https://reviews.apache.org/r/14496/diff/ Testing --- Thanks, Jun Rao

Re: Review Request 14496: incorporating review feedback

2013-10-05 Thread Jun Rao
/diff/ Testing --- Thanks, Jun Rao

Re: preparing for the 0.8 final release

2013-10-07 Thread Jun Rao
%20Available%22) Do people agree with this list? Joe, I don't have good understanding of KAFKA-1018. Do you think this needs to be fixed in 0.8 final? Thanks, Jun On Fri, Sep 13, 2013 at 9:18 AM, Jun Rao jun...@gmail.com wrote: Hi, Everyone, We have been stabilizing the 0.8 branch since

Review Request 14534: Patch for KAFKA-1069

2013-10-08 Thread Jun Rao
/browse/KAFKA-1069 Repository: kafka Description --- kafka-1069 Diffs - core/src/main/scala/kafka/cluster/Partition.scala a9bb3c8c450f99d8e110ae6709c5fe1fe09a8f77 Diff: https://reviews.apache.org/r/14534/diff/ Testing --- Thanks, Jun Rao

Review Request 14538: Patch for KAFKA-954

2013-10-08 Thread Jun Rao
/browse/KAFKA-954 Repository: kafka Description --- kafka-954 Diffs - README.md 9879427d4103354c7c3872c0b6caf454c1204a76 Diff: https://reviews.apache.org/r/14538/diff/ Testing --- Thanks, Jun Rao

Re: preparing for the 0.8 final release

2013-10-09 Thread Jun Rao
KAFKA-1008 has been checked into the 0.8 branch and needs to be manually double-committed to trunk. To avoid merging problems, I suggest that for all future changes in the 0.8 branch, we double commit them to trunk. Any objections? Thanks, Jun On Mon, Oct 7, 2013 at 5:33 PM, Jun Rao jun

Re: Review Request 14554: Move AddPartitions into TopicCommand in trunk

2013-10-09 Thread Jun Rao
://reviews.apache.org/r/14554/#comment52173 Ditto for the comment above. - Jun Rao On Oct. 9, 2013, 6:48 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14554

Re: Review Request 14571: Patch for KAFKA-1018

2013-10-10 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14571/#review26865 --- Ship it! Do we want to address issue #2 in the patch too? - Jun

Re: Review Request 14582: Upgrading zkclient to verion 0.4. Adds retry logic in case of a failed ZooKeeper session re-establishment.

2013-10-13 Thread Jun Rao
logic in zkclient. Assuming there is already a unit test in zkclient itself, is there any value in including this unit test in Kafka? - Jun Rao On Oct. 10, 2013, 10:14 p.m., Anatoly Fayngelerin wrote: --- This is an automatically

Re: Review Request 14638: Patch for KAFKA-1086

2013-10-15 Thread Jun Rao
/tools/GetOffsetShell.scala https://reviews.apache.org/r/14638/#comment52623 Should we name this option partitions? - Jun Rao On Oct. 14, 2013, 10:50 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail

Re: Conditional leader and ISR update fails forever

2013-10-16 Thread Jun Rao
The value of the state path for a partition in ZK can be updated by both the controller (for choosing the new leader and potentially shrinking Isr) and the leader replica (for expanding/shrinking Isr). After the controller modified the state path, we don't want the old leader to modify Isr

Review Request 14675: Patch for KAFKA-1090

2013-10-16 Thread Jun Rao
/browse/KAFKA-1090 Repository: kafka Description --- kafka-1090 Diffs - core/src/test/scala/unit/kafka/network/SocketServerTest.scala 94b5a2a36b2b82377576c2479a91e6799ae2e326 Diff: https://reviews.apache.org/r/14675/diff/ Testing --- Thanks, Jun Rao

Review Request 14676: Patch for KAFKA-1091

2013-10-16 Thread Jun Rao
/browse/KAFKA-1091 Repository: kafka Description --- kafka-1091 Diffs - core/src/main/scala/kafka/server/KafkaApis.scala 338d1cc6533fd219941f2afb9bc0ea122b368bbe Diff: https://reviews.apache.org/r/14676/diff/ Testing --- Thanks, Jun Rao

Re: Review Request 14675: address previous review comments

2013-10-17 Thread Jun Rao
94b5a2a36b2b82377576c2479a91e6799ae2e326 Diff: https://reviews.apache.org/r/14675/diff/ Testing --- Thanks, Jun Rao

Re: Review Request 14730: Patch for KAFKA-1001

2013-10-18 Thread Jun Rao
the importance of the ordering of the steps, i.e., removing fetchers, truncate log(and checkpoint flushing offset) and adding fetchers? - Jun Rao On Oct. 18, 2013, 2:15 a.m., Guozhang Wang wrote: --- This is an automatically generated

Re: preparing for the 0.8 final release

2013-10-19 Thread Jun Rao
one. https://issues.apache.org/jira/browse/KAFKA-1008 -Jay On Wed, Oct 9, 2013 at 8:56 AM, Jun Rao jun...@gmail.com wrote: KAFKA-1008 has been checked into the 0.8 branch and needs to be manually double-committed to trunk. To avoid merging problems, I suggest that for all

Re: preparing for the 0.8 final release

2013-10-20 Thread Jun Rao
with a filter (actually a relatively large number of topics, some of which are newly created over time). I'm still not sure whether the new version is solving this issue (since it was a rare transient thing anyway). Thanks, Jason On Sat, Oct 19, 2013 at 2:03 AM, Jun Rao jun

Re: Review Request 14771: Log.getOffsetsBefore(t, …) does not return the last confirmed offset before t

2013-10-20 Thread Jun Rao
is no longer accurate after a broker restart. Also, this patch should probably be done in trunk, instead of 0.8 since it doesn't seem to be a blocker. - Jun Rao On Oct. 19, 2013, 10:34 p.m., Swapnil Ghike wrote: --- This is an automatically

  1   2   3   4   5   6   7   8   9   10   >