Hi Jiri,
Thank you for the follow up.
I guess, it can happen that during start-up and the respective
rebalances some partitions are read more often than others and that
consequently the timestamps in the repartition topic are mixed up more
than during normal operation. Unfortunately, I do not
Hi all,
I am writing to describe where I got with the issue.
The next thing I wanted to do was to check topics if they contain records
with mixed timestamps in single partition that could cause the
warning "Skipping record for expired segment" - meaning, timestamp of
incoming record is behind
Hi Jiří,
Hmm, that is a mystery indeed. to inspect the log, you could try
kafka-dump-log (I've never used it).
What I have done before is use kafka-console-consumer, with the
following option specified:
--property
properties include:
Is it possible that during normal processing these records were actually
dropped (eg due to a deserialization exception)? During restoration we
actually just copy plain bytes from the changelog directly, so if this was
an
optimized source table it's possible that records which were supposed to
be
Hi Bruno, John and Sophie,
thank you very much for quick responses, you are the best. After thinking
about it a little bit more, it seems fishy.
>From logs, I see that it is not happening when application is running
normally.
I have checked timestamps (windowStartTimestamp) - connecting local
Hey all,
Sorry for the confusion. Bruno set me straight offline.
Previously, we had metrics for each reason for skipping records, and the
rationale was that you would monitor the metrics and only turn to the logs
if you needed to *debug* unexpected record skipping. Note that skipping
records by
While I agree that seems like it was probably a refactoring mistake, I'm
not
convinced it isn't the right thing to do. John, can you reiterate the
argument
for setting it to debug way back when?
I would actually present this exact situation as an argument for keeping it
as
warn, since something
Hi,
I’m sorry for the trouble. It looks like it was a mistake during
https://github.com/apache/kafka/pull/6521
Specifically, while addressing code review comments to change a bunch of other
logs from debugs to warnings, that one seems to have been included by accident:
Hi,
I am pretty sure this was intentional. All skipped records log
messages are on WARN level.
If a lot of your records are skipped on app restart with this log
message on WARN-level, they were also skipped with the log message on
DEBUG-level. You simply did not know about it before. With an
Hi,
in
https://github.com/apache/kafka/commit/9f5a69a4c2d6ac812ab6134e64839602a0840b87#diff-a5cfe68a5931441eff5f00261653dd10R134
log level of "Skipping record for expired segment" was changed from debug
to warn. Was it intentional change? Should it be somehow handled by user?
How can user handle
10 matches
Mail list logo
