[
https://issues.apache.org/jira/browse/MAHOUT-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205666#comment-15205666
]
Dmitriy Lyubimov commented on MAHOUT-1755:
------------------------------------------
Interesting. this gets back to the caching checkpoints: caching checkpoints is
a way to optimize for common computational paths in DAGs.
so the only way to implement checkpoints in Flink (other than with cacheHint =
NONE) is therefore always dumping stuff to DFS? If yes, this is a serious
limitation as it prevents the most effective form of caching, i.e., when the
object trees themselves are used. Even if hdfs checkpoint hits memory cache, we
still need to spend time serializing and deserializing partitions back for
every new tiny bit of computation (such as taking mean, or sum, or
reducing-rebroadcasting intermediate statistics). This loop is very common for
algorithms running till convergence. If we have a heavyweight scheduling in
these type of systems inside the loop (as opposed to one-time scheduling
outside the loop in superstep systems), it is already bad enough. If we on top
of that need to serialize and deserialize when we run 50 conversion iterations,
this is pretty disastrous.
So there's absolutely no way to keep datasets in object trees inside the worker
vms between computations?
> Mahout DSL for Flink: Flush intermediate results to FS
> ------------------------------------------------------
>
> Key: MAHOUT-1755
> URL: https://issues.apache.org/jira/browse/MAHOUT-1755
> Project: Mahout
> Issue Type: Task
> Components: Flink
> Affects Versions: 0.10.2
> Reporter: Alexey Grigorev
> Assignee: Suneel Marthi
> Priority: Minor
> Fix For: 0.12.0
>
>
> Now Flink (unlike Spark) doesn't keep intermediate results in memory -
> therefore they should be flushed to a file system, and read back when
> required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
