[
https://issues.apache.org/jira/browse/MESOS-2057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bernd Mathiske updated MESOS-2057:
----------------------------------
Description:
Add a URI flag to CommandInfo messages that indicates caching. Cache files
downloaded by the fetcher in a repository. When a URI is "cached", it is only
ever downloaded once for the same user on the same slave as long as the slave
keeps running.
This even holds if multiple tasks request the same URI concurrently. If
multiple requests for the same URI occur, perform only one of them and reuse
the result. Make concurrent requests for the same URI wait for the one
download.
Different URIs from different CommandInfos can be downloaded concurrently.
No cache eviction, cleanup or failover will be handled for now. Additional
tickets will be filed for these enhancements. (So don't use this feature in
production until the whole epic is complete.)
Note that implementing this does not suffice for production use. This ticket
contains the main part of the fetcher logic, though. See the epic MESOS-336 for
the rest of the features that lead to a fully functional fetcher cache.
The proposed general approach is to keep all bookkeeping about what is in which
stage of being fetched and where it resides in the slave's
MesosContainerizerProcess, so that all concurrent access is disambiguated and
controlled by an "actor" (aka libprocess "process").
Depends on MESOS-2056.
was:
Add a URI flag to CommandInfo messages that indicates caching. When a URI is
"cached" it is only ever downloaded once for the same user on the same slave as
long as the slave keeps running. This even holds if multiple tasks request the
same URI concurrently. Different URIs from different CommandInfos can be
downloaded concurrently.
No cache eviction, cleanup or failover will be handled for now. Additional
tickets will be filed for these enhancements. (So don't use this feature in
production until the whole epic is complete.)
Depends on MESOS-2056.
> Add cache functionality with concurrent downloading to fetcher
> --------------------------------------------------------------
>
> Key: MESOS-2057
> URL: https://issues.apache.org/jira/browse/MESOS-2057
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher, slave
> Reporter: Bernd Mathiske
> Assignee: Bernd Mathiske
>
> Add a URI flag to CommandInfo messages that indicates caching. Cache files
> downloaded by the fetcher in a repository. When a URI is "cached", it is only
> ever downloaded once for the same user on the same slave as long as the slave
> keeps running.
> This even holds if multiple tasks request the same URI concurrently. If
> multiple requests for the same URI occur, perform only one of them and reuse
> the result. Make concurrent requests for the same URI wait for the one
> download.
> Different URIs from different CommandInfos can be downloaded concurrently.
> No cache eviction, cleanup or failover will be handled for now. Additional
> tickets will be filed for these enhancements. (So don't use this feature in
> production until the whole epic is complete.)
> Note that implementing this does not suffice for production use. This ticket
> contains the main part of the fetcher logic, though. See the epic MESOS-336
> for the rest of the features that lead to a fully functional fetcher cache.
> The proposed general approach is to keep all bookkeeping about what is in
> which stage of being fetched and where it resides in the slave's
> MesosContainerizerProcess, so that all concurrent access is disambiguated and
> controlled by an "actor" (aka libprocess "process").
> Depends on MESOS-2056.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
