[
https://issues.apache.org/jira/browse/MESOS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973753#comment-13973753
]
Benjamin Hindman commented on MESOS-336:
----------------------------------------
[~bernd-mesos], the concurrency issues are indeed a bit tricky. The fetcher is
currently invoked by the containerizer which means that coordinating what the
fetcher should actually be downloading might be a bigger refactor in the "phase
2" you had mentioned in your posts above. To keep things simple for this first
version what about creating a directory for each checksum and then store the
files that collide within that directory? For example, if we had the files
foo.tar.gz, bar.tar.gz, and baz.tar.gz and the checksums for foo and baz are
the same we'd have the directory structure:
9A32C8943E/foo.tar.gz
9A32C8943E/baz.tar.gz
472F293ECA/bar.tar.gz
This doesn't handle the fact that multiple fetchers may be racing to download
the same resource. We could use POSIX file locks here, or punt this for phase 2
and just deal with the inefficiency. What do you think?
> Mesos slave should cache executors
> ----------------------------------
>
> Key: MESOS-336
> URL: https://issues.apache.org/jira/browse/MESOS-336
> Project: Mesos
> Issue Type: Improvement
> Components: slave
> Reporter: brian wickman
> Assignee: Bernd Mathiske
> Labels: newbie
>
> The slave should be smarter about how it handles pulling down executors. In
> our environment, executors rarely change but the slave will always pull it
> down from regardless HDFS. This puts undue stress on our HDFS clusters, and
> is not resilient to reduced HDFS availability.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
