There may be some confusion of terminology hw mark == highest offset the server will give to consumers, based on what is sync'd to disk (0.7) or replicated to multiple machines (0.8) consumer offset == consumer's position in the stream (always less than hw mark, by definition)
As Jun says in 0.7 we take the HW mark from the log and cache it in memory. The consumer offset is stored in zookeeper. Hope that helped clarify. -Jay On Fri, Jun 15, 2012 at 7:12 AM, Jun Rao <jun...@gmail.com> wrote: > In 0.7, HW is only kept in memory in the broker. In 0.8, it will be > checkpointed to disk too. > > Thanks, > > Jun > > On Fri, Jun 15, 2012 at 3:51 AM, Raymond Ng <raymond...@gmail.com> wrote: > > > Hi > > > > I'm trying to locate where the watermark is written to, is it on the > kafka > > server / zookeeper or where kafka spouts are deployed? > > > > -- > > Rgds > > Ray > > >
