[
https://issues.apache.org/jira/browse/SYSTEMML-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm resolved SYSTEMML-1019.
--------------------------------------
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 0.11
> Perftest: MSVM, L sparse, unncessary under-utilization (parfor dop)
> -------------------------------------------------------------------
>
> Key: SYSTEMML-1019
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1019
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 0.11
>
>
> In execution modes spark and hybrid_spark, the parfor optimizer uses a
> smaller memory budget (50% of normal memory budget) for deciding on the local
> parfor degree of parallelism in order to avoid unnecessary "guarded collect"
> over hdfs. However, "guarded collect" can only happen in case of existing
> spark instructions; otherwise this decision leads to unnecessary
> under-utilization if the memory requirement is limiting the degree of
> parallelism.
> We should determine, if there are existing spark instructions in the body and
> accordingly adapt the memory budget the parfor optimizer works with. On this
> MSVM usecase, the performance differences are as follows:
> {code}
> #before the change
> MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 144
> MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 156
> #after the change
> MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 99
> MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 117
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
