Thanks for the follow-up and the bug report, David.
We're taking a look at that.
On Mon, Oct 10, 2016 at 4:36 PM, David Garcia <dav...@spiceworks.com> wrote:
> Thx for the responses. I was able to identify a bug in how the times are
> obtained (offsets resolved as unknown cause the issue):
> “Actually, I think the bug is more subtle. What happens when a consumed
> topic stops receiving messages? The smallest timestamp will always be the
> static timestamp of this topic.
> On 10/7/16, 5:03 PM, "David Garcia" <dav...@spiceworks.com> wrote:
> Ok I found the bug. Basically, if there is an empty topic (in the
> list of topics being consumed), any partition-group with partitions from
> the topic will always return -1 as the smallest timestamp (see
> To reproduce, simply start a kstreams consumer with one or more empty
> topics. Punctuate will never be called.
> -David ”
> On 10/10/16, 1:55 AM, "Michael Noll" <mich...@confluent.io> wrote:
> > We have run the application (and have confirmed data is being
> for over 30 mins…with a 60-second timer.
> Ok, so your app does receive data but punctuate() still isn't being
> > So, do we need to just rebuild our cluster with bigger machines?
> That's worth trying out. See
> for some EC2 instance types recommendations.
> But I'd also suggest to look into the logs of (1) your application,
> (2) the
> log files of the Kafka broker(s), and (3) the log files of ZooKeeper
> to see
> whether you see anything suspicious?
> Sorry for not being able to provide more actionable feedback at this
> point. Typically we have seen such issues only (but not exclusively)
> cases where there have been problems in the environment in which your
> application is running and/or the environment of the Kafka clusters.
> Unfortunately these environment problems are a bit tricky to debug
> via the mailing list.
> On Fri, Oct 7, 2016 at 8:11 PM, David Garcia <dav...@spiceworks.com>
> > Yeah, this is possible. We have run the application (and have
> > data is being received) for over 30 mins…with a 60-second timer.
> So, do we
> > need to just rebuild our cluster with bigger machines?
> > -David
> > On 10/7/16, 11:18 AM, "Michael Noll" <mich...@confluent.io> wrote:
> > David,
> > punctuate() is still data-driven at this point, even when you're
> > the
> > WallClock timestamp extractor.
> > To use an example: Imagine you have configured punctuate() to be
> > every
> > 5 seconds. If there's no data being received for a minute, then
> > punctuate
> > won't be called -- even though you probably would have expected
> this to
> > happen 12 times during this 1 minute.
> > (FWIW, there's an ongoing discussion to improve punctuate(),
> part of
> > which
> > is motivated by the current behavior that arguably is not very
> > intuitive to
> > many users.)
> > Could this be the problem you're seeing? See also the related
> > discussion
> > at
> > http://stackoverflow.com/questions/39535201/kafka-problems-with-
> > timestampextractor
> > .
> > On Fri, Oct 7, 2016 at 6:07 PM, David Garcia <
> > wrote:
> > > Hello, I’m sure this question has been asked many times.
> > > We have a test-cluster (confluent 3.0.0 release) of 3 aws
> > m4.xlarges. We
> > > have an application that needs to use the punctuate() function
> to do
> > some
> > > work on a regular interval. We are using the WallClock
> > > Unfortunately, the method is never called. I have checked the
> > > filedescriptor setting for both the user as well as the
> process, and
> > > everything seems to be fine. Is this a known bug, or is there
> > something
> > > obvious I’m missing?
> > >
> > > One note, the application used to work on this cluster, but
> now it’s
> > not
> > > working. Not really sure what is going on?
> > >
> > > -David
> > >