Jake Maes created SAMZA-1480:
--------------------------------
Summary: TaskStorageManager improperly initializes changelog
consumer position when restoring a store from disk
Key: SAMZA-1480
URL: https://issues.apache.org/jira/browse/SAMZA-1480
Project: Samza
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Jake Maes
Assignee: Jake Maes
Fix For: 0.14.0
For the Host Affinity state restore, an OFFSET file is written to disk on each
commit. This offset file contains the most recently written changelog event
which is also reflected in the on-disk state. When the container is restarted,
it restores the on-disk store and then replays the changelog from the offset
recorded in the OFFSET file in order to restore any changelog events that were
produced when the job ran on a different host.
http://samza.apache.org/learn/documentation/0.13/yarn/yarn-host-affinity.html
When TaskStorageManager initializes the consumer, it uses the offset from the
OFFSET file, which is already reflected in the state.
Instead, it should use the SystemAdmin.getOffsetsAfter() method to get the next
offset to consume. This will avoid the replay of 1 extra message for state
restore.
It should then use SystemAdmin.offsetComparator() to use the larger of the next
offset (calculated above) and the oldest offset (according to the metadata).
This is necessary for changelogs configured with TTL retention rather than
infinite retention where the offset from the OFFSET file may no longer be
valid.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
