Okay, I looked into this more. I think there are actually a number of changes that should be made here, and it might be better to really think it through and do a more serious project to get this right. So three things to discuss:
- I propose we make no change in the producer api in 0.8. Returning the offset is a nice feature but not really a blocker. It can come in 0.8 or 0.9. Storing the key can be done using the existing ProducerData definition. This is not ideal since it is unintuitive to have the same key for many values, but since most people will not be using the key de-duplication I think this is a tolerable intermediate state. - I wrote up a few thoughts on what we should do to the producer API (in 0.8.1 or 0.9 or whatever) here: https://cwiki.apache.org/confluence/display/KAFKA/Producer+API+Proposal The rationale is that I think a good producer API is actually a little more involved. Properly handling async requests means making each response a future, which while not rocket science will involve pretty big changes to the producer internals. Making this change without refactoring the existing code would be very hard to do, but the refactoring is probably fairly involved. I suspect if we try to cram this into 0.8 we will probably do a half-assed job and just end up redoing it again. -Jay On Thu, Oct 4, 2012 at 1:24 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey guys, > > Two quick things: > 1. There is a proposed change to ProducerData to allow us to retain the > partition key with the message here: > https://issues.apache.org/jira/browse/KAFKA-544 > This would also mean changing the Encoder/Decoder interface. Since this > impacts the user-facing APIs I think it would be useful to get any feedback > up front before I write a bunch of code. > 2. There is another JIRA to make the producer return the offset(s) to the > caller. Since these are both in the same chunk of code, I would like to > grab this if no one else is actively working on it to avoid merge hell. > https://issues.apache.org/jira/browse/KAFKA-496 > > -Jay >
