[
https://issues.apache.org/jira/browse/SYSTEMML-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm resolved SYSTEMML-1274.
--------------------------------------
Resolution: Done
Assignee: Matthias Boehm
Fix Version/s: SystemML 0.13
> Unnecessary rdd computation for nnz maintenance on write
> --------------------------------------------------------
>
> Key: SYSTEMML-1274
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1274
> Project: SystemML
> Issue Type: Bug
> Components: Runtime
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 0.13
>
>
> Our primitive for writing binary block RDDs to HDFS (as used in guarded
> collect), first computes the number of non-zeros (nnz) and subsequently
> writes out the data. This leads to redundant RDD computation, which can be
> expensive for large DAGs of RDD operations. Explicitly computing the nnz is
> unnecessary as we could simply piggyback this computation onto the write via
> an accumulator as done in multiple other places in SystemML.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
