Prateek Maheshwari created SAMZA-2356:
-----------------------------------------
Summary: Do not trim changelog if time since last checkpoint is
greater than min.compaction.lag
Key: SAMZA-2356
URL: https://issues.apache.org/jira/browse/SAMZA-2356
Project: Samza
Issue Type: Task
Reporter: Prateek Maheshwari
Assignee: Prateek Maheshwari
In the current implementation for transactional state, if the checkpointed
changelog offset gets compacted, (e.g. if the processor was down for more than
min compaction lag), and the checkpointed changelog offset is b/w the oldest
and newest post compaction, we will end up deleting part of the current state
(potentially everything after checkpointed offset). We should not do this.
Instead, we should write the checkpoint timestamp along with the the
checkpointed changelog offset, and not do a trim during restore if time since
last checkpoint is older than (or close to) min.compaction.lag. We should
instead restore to latest and log a warning that transactional state guarantee
doesn't hold (i.e., same as the current behavior). The fix for users would then
be to increase the min.compaction.lag.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
