The answer you're looking for has already been said on this list, iirc. getOffsetsBefore works based on the mtime of the log segments, which is why the offsets you get are about 500 megs apart On Sep 26, 2012 7:22 PM, "Alexander Gallego" <gallego.al...@gmail.com> wrote:
> This is a valuable follow up: > > After flushing about 5.5 G of data onto a test kafka node (localhost) > SimpleConsumer.getOffsetsBefore() semi works. At least it doesn't > return 0's anymore. > > > I think the implementation is fine, what I think is missing is api > documentation, perhaps I'll push a doc change. Here is what happens. > > > 1. After you run the consumer for long enough you start getting some > real offsets, the question becomes ** when ** and how (predictably) > 2. Answer is easy. > > Here are the offsets for my test. > > 5368709220 > 4831838298 > 4294967376 > 3758096454 > 3221225532 > 2684354610 > 2147483688 > 1610612766 > 1073741844 > 536870922 > 0 > > > Here are the kafka logs. > > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:51 > 00000000000000000000.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53 > 00000000000536870922.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53 > 00000000001073741844.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:54 > 00000000001610612766.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55 > 00000000002147483688.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55 > 00000000002684354610.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56 > 00000000003221225532.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56 > 00000000003758096454.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:57 > 00000000004294967376.kafka │ > -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:58 > 00000000004831838298.kafka │ > -rw-r--r-- 1 agallego wheel 534062760 Sep 26 11:59 > 00000000005368709220.kafka > > > To point it out. > > *** The offsets are the name of the files. **** > > > In other words, the offsets don't get updated until a new file is > generated. > > > > I'm not sure how to submit a push request for documentation for this > finding. Perhaps a dev can comment if I missed the docs, certainly the > api docs don't help much (they are not totally accurate) > > > Thanks, > > > > > > Sincerely, > Alexander Gallego > > ---*--- > ------* > * * * > > > > > On Wed, Sep 26, 2012 at 10:22 AM, Alexander Gallego > <gallego.al...@gmail.com> wrote: > > Hello, > > > > I don't mean to spam the mailing list. I emailed yesterday, but I had > > made the repo containing the examples 'private' by accident. It is now > > open. > > > > Thanks again for reading. > > > > > > Sincerely, > > Alex > > > > ---*--- > > ------* > > * * * > > > > > > > > > > On Tue, Sep 25, 2012 at 9:11 PM, Alexander Gallego > > <gallego.al...@gmail.com> wrote: > >> Hello, > >> > >> I'm having problems with Kafka 0.7: > >> > >> Here is a project I've put together to show the problem: > >> https://bitbucket.org/agallego/kafkaoffset (make sure to run twice: > >> once to generate data, and one to see the problem. ) > >> > >> I can't seem to get valid offsets from the queue using a > >> SimpleConsumer class. > >> ( > https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13 > ) > >> > >> I can read and write to the queue no problem given an offset. > >> Everything works just fine. However, I want to read from the queue the > >> last offset from 'now()' . That is to say, I want to pass in an > >> arbitrary time in the past and I want to get a list of correct > >> offsets. (off course given that there is data before that time) > >> > >> > >> ** What am I seeing instead ? *** > >> > >> I always get an array of 0's > >> > >> quite literally long[] tmp = {0L, 0L, 0L....} > >> > >> > >> > >> ** What do I expect to see *** > >> > >> An array of valid offsets as stated by the api. > >> > >> > >> > >> > >> Is this a bug ? What am I doing wrong. The whole code is about 20 > >> lines of code. It assumes a running instance of kafka on localhost. > >> (off course you can change it) > >> > >> > >> Thanks in advance > >> > >> > >> Sincerely, > >> Alex > >> > >> ---*--- > >> ------* > >> * * * >
