This is a valuable follow up: After flushing about 5.5 G of data onto a test kafka node (localhost) SimpleConsumer.getOffsetsBefore() semi works. At least it doesn't return 0's anymore.
I think the implementation is fine, what I think is missing is api documentation, perhaps I'll push a doc change. Here is what happens. 1. After you run the consumer for long enough you start getting some real offsets, the question becomes ** when ** and how (predictably) 2. Answer is easy. Here are the offsets for my test. 5368709220 4831838298 4294967376 3758096454 3221225532 2684354610 2147483688 1610612766 1073741844 536870922 0 Here are the kafka logs. -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:51 00000000000000000000.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53 00000000000536870922.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53 00000000001073741844.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:54 00000000001610612766.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55 00000000002147483688.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55 00000000002684354610.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56 00000000003221225532.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56 00000000003758096454.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:57 00000000004294967376.kafka │ -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:58 00000000004831838298.kafka │ -rw-r--r-- 1 agallego wheel 534062760 Sep 26 11:59 00000000005368709220.kafka To point it out. *** The offsets are the name of the files. **** In other words, the offsets don't get updated until a new file is generated. I'm not sure how to submit a push request for documentation for this finding. Perhaps a dev can comment if I missed the docs, certainly the api docs don't help much (they are not totally accurate) Thanks, Sincerely, Alexander Gallego ---*--- ------* * * * On Wed, Sep 26, 2012 at 10:22 AM, Alexander Gallego <gallego.al...@gmail.com> wrote: > Hello, > > I don't mean to spam the mailing list. I emailed yesterday, but I had > made the repo containing the examples 'private' by accident. It is now > open. > > Thanks again for reading. > > > Sincerely, > Alex > > ---*--- > ------* > * * * > > > > > On Tue, Sep 25, 2012 at 9:11 PM, Alexander Gallego > <gallego.al...@gmail.com> wrote: >> Hello, >> >> I'm having problems with Kafka 0.7: >> >> Here is a project I've put together to show the problem: >> https://bitbucket.org/agallego/kafkaoffset (make sure to run twice: >> once to generate data, and one to see the problem. ) >> >> I can't seem to get valid offsets from the queue using a >> SimpleConsumer class. >> (https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13) >> >> I can read and write to the queue no problem given an offset. >> Everything works just fine. However, I want to read from the queue the >> last offset from 'now()' . That is to say, I want to pass in an >> arbitrary time in the past and I want to get a list of correct >> offsets. (off course given that there is data before that time) >> >> >> ** What am I seeing instead ? *** >> >> I always get an array of 0's >> >> quite literally long[] tmp = {0L, 0L, 0L....} >> >> >> >> ** What do I expect to see *** >> >> An array of valid offsets as stated by the api. >> >> >> >> >> Is this a bug ? What am I doing wrong. The whole code is about 20 >> lines of code. It assumes a running instance of kafka on localhost. >> (off course you can change it) >> >> >> Thanks in advance >> >> >> Sincerely, >> Alex >> >> ---*--- >> ------* >> * * *
