Using mirror maker 2 to replicate one cluster to another, we're getting a
lot of message rejections due to timestamp out of range.
[2025-11-12 00:58:46,727] ERROR [MirrorSourceConnector|task-0]
WorkerSourceTask{id=MirrorSourceConnector-0} failed to send record to
our.topic.name.goes.here:
(org.apache.kafka.connect.runtime.AbstractWorkerSourceTask:419)
org.apache.kafka.common.InvalidRecordException: Timestamp 1725218013643 of
message with offset 7 is out of range. The timestamp should be within
[1762822726723, 1762995526723]
The range is one day plus or minus current time, which is a problem for a
lot of our topics that might have 30 day retention, but also compacted
topics which could have years old messages that need to be replicated.
My analysis so far suggests that it is message.timestamp.difference.max.ms
- even though this is a deprecated setting, we're using kafka 3.9 until our
migration has completed, and that seems to be the setting that relates to
the message. message.timestamp.before.max.ms is correctly set to Long.MAX
on both source and target.
Our source topics have message.timestamp.difference.max.ms set to Long.MAX,
but the destination topics have it set to a day. Looking into
DefaultConfigPropertyFilter, the default behaviour is to exclude this
property from topic config sync, but also the default behaviour is to use
the target's default property.
I've tried overriding config.properties.exclude at the connector level to
not exclude message.timestamp.difference.max.ms (this override shows up
correctly in the logs) and overriding use.defaults.from to source (this
doesn't show up in the logs, but I think that's expected from the code) but
the target topic configs still set message.timestamp.difference.max.ms to
a day.
Part of the problem is working out at what level to set the values
(config.properties.exclude seems to work best on
source->target.config.properties.exclude, for example)
Is there anything else I can try to get replication working without having
to update target topic configuration (which is a problem as mirror maker
creates the topics with the incorrectly replicated config and then
immediately starts syncing messages, so we lose messages immediately). I
can recreate the cluster to replay all the messages, but the target
configuration needs to be correct.
Thanks
Will
--
Disclaimer: This e-mail and any attachments may contain Confidential
Information. If you are not the intended recipient, any disclosure,
copying, distribution or use of any information contained herein is
strictly prohibited. If you have received this transmission in error,
please immediately notify the Security Officer, and destroy the original
transmission and any attachments without reading or saving.
