[
https://issues.apache.org/jira/browse/MESOS-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14653531#comment-14653531
]
Bernd Mathiske commented on MESOS-2073:
---------------------------------------
In phase 1, the server should enact dynamic content updates as follows.
1. Remove the checksum file. This will cause the fetcher to not cache content
until we are in a stable situation again.
2. Swap in the new content file.
3. Put a fresh checksum file in place.
This way there is no race between content and checksum preventing cache
refreshing when it must be indicated. However, the saved checksum may not
actually correspond to the downloaded content. It may be older. This is
tolerable, because
- in any case newer content than previously in the cache did get downloaded
and
- if the checksum is indeed out of sync, the next request will see either
no checksum or a newer checksum
and hence refresh the cache again.
This means that in no case an update is missed. Worst case, the same content
gets downloaded twice, once with the old checksum and then again with the
correct one. This is also tolerable.
In case multiple updates happen while holding on to an outdated checksum, this
also works out OK, as long as every new checksum is different from all others.
If the latter cannot be guaranteed, for instance because the checksum
calculations in use are idempotent and old content may reappear, then we have a
problem. Two approaches may help:
A) Look at "Last-Modified" headers and such to still establish different
identities of equal-looking checksums.
B) Download the checksum before AND after downloading the content. Only if it
matches, proceed. Otherwise start over.
Approach A strongly hints that phase 1 would be obsolete if we simply honored
HTTP cache directives. But this is a protocol-specific approach. I suggest that
in the long run we implement both. But we start with phase 1, as this leads us
to phase 2, which I reckon is what we typically really want.
Note (again) that in phase 1 the checksum is allowed to say nothing about the
integrity of the downloaded content. For example, simple serial numbers can be
used.
> Fetcher cache file verification, updating and invalidation
> ----------------------------------------------------------
>
> Key: MESOS-2073
> URL: https://issues.apache.org/jira/browse/MESOS-2073
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher, slave
> Reporter: Bernd Mathiske
> Assignee: Bernd Mathiske
> Priority: Minor
> Labels: mesosphere
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> The other tickets in the fetcher cache epic do not necessitate a check sum
> (e.g. MD5, SHA*) for files cached by the fetcher. Whereas such a check sum
> could be used to verify whether the file arrived without unintended
> alterations, it can first and foremost be employed to detect and trigger
> updates.
> Scenario: If a UIR is requested for fetching and the indicated download has
> the same check sum as the cached file, then the cache file will be used and
> the download forgone. If the check sum is different, then fetching proceeds
> and the cached file gets replaced.
> This capability will be indicated by an additional field in the URI protobuf.
> Details TBD, i.e. to be discussed in comments below.
> In addition to the above, even if the check sum is the same, we can support
> voluntary cache file invalidation: a fresh download can be requested, or the
> caching behavior can be revoked entirely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
