[
https://issues.apache.org/jira/browse/SAMZA-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lakshmi Manasa Gaduputi updated SAMZA-2664:
-------------------------------------------
Description:
TO change the grouper factory of the job, checkpoint validation should be
disabled (using the config = ). This ensures the new checkpoints can be written
with the new factory and job can use the new checkpoints. However, since the
factory is part of the checkpoint key, the older checkpoints are never cleaned
up. This leads to job reading those old checkpoints on every deploy and failing
checkpoint validation. to get job running, checkpoint validation needs to be
disabled forever.
Keeping checkpoint validation disabled forever is risky, as that would ignore
the exceptions caused by serialization errors for checkpoint key.
one solution is to clean out the older checkpoints when validation fails due to
grouper factory.
was:
TO change the grouper factory of the job, checkpoint validation should be
disabled (using the config = ). This ensures the new checkpoints can be written
with the new factory and job can use the new checkpoints. However, since the
factory is part of the checkpoint key, the older checkpoints are never cleaned
up. This leads to job reading those old checkpoints on every deploy and failing
checkpoint validation. to get job running, checkpoint validation needs to be
disabled forever.
one solution is to clean out the older checkpoints when validation fails due to
grouper factory.
> Checkpoint validation enabled needs to be false forever if grouper factory
> was changed.
> ---------------------------------------------------------------------------------------
>
> Key: SAMZA-2664
> URL: https://issues.apache.org/jira/browse/SAMZA-2664
> Project: Samza
> Issue Type: Bug
> Reporter: Lakshmi Manasa Gaduputi
> Assignee: Lakshmi Manasa Gaduputi
> Priority: Major
>
> TO change the grouper factory of the job, checkpoint validation should be
> disabled (using the config = ). This ensures the new checkpoints can be
> written with the new factory and job can use the new checkpoints. However,
> since the factory is part of the checkpoint key, the older checkpoints are
> never cleaned up. This leads to job reading those old checkpoints on every
> deploy and failing checkpoint validation. to get job running, checkpoint
> validation needs to be disabled forever.
> Keeping checkpoint validation disabled forever is risky, as that would ignore
> the exceptions caused by serialization errors for checkpoint key.
>
> one solution is to clean out the older checkpoints when validation fails due
> to grouper factory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
