Hello Greg,
Sorry for the long delay in responses, but very glad to see you have gone
this far to resolve the problem. All of your solutions make sense to me.
I think for the first problem your raised, it is a general problem not only
for Streams' sending changelog records but for any clients
After more digging, a good solution seems to be to set the topic timetstamp
policy to LogAppendTime and also update our timestamp extractor to respect
this (by just returning record.timestamp() if the timestamp type is set to
LogAppendTime.) This gets us the semantics we want for the changelog
I managed to answer some of my own questions :)
For future google'ers:
to deal with retention.ms see
https://issues.apache.org/jira/browse/KAFKA-4340
to deal with early rejection of bad timestamps the
message.timestamp.difference.max.ms config is relevant discussion here
Hey all, we are currently working on migrating our system to kafka 10.2
from 10.0 and one thing that we have hit that I wanted some advice on is
dealing with the new log retention/rolling semantics that are based on
timestamps.
We send telemetry data from installed clients into kafka via kafka
