[
https://issues.apache.org/jira/browse/SYSTEMML-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626422#comment-15626422
]
Glenn Weidner commented on SYSTEMML-1072:
-----------------------------------------
I modified sparkDML.sh to use below options:
--driver-java-options "-Xms20g -Xmn2g"
--conf spark.executor.extraJavaOptions="-Xmn5500m"
--conf spark.executor.memory="-Xms50g"
I used spark.executor.memory for -Xms50g setting to prevent
SparkConf.validateSettings exception:
spark.executor.extraJavaOptions is not allowed to alter memory settings
(was '-Xms50g -Xmn5500m'). Use spark.executor.memory instead.
The binomial 80g MultiLogReg produced consistent results:
-- Running runMultiLogReg on 10M_1k_dense (all configs):
MultiLogReg train ict=0 on mbperftest/binomial/X10M_1k_dense: 109
MultiLogReg train ict=1 on mbperftest/binomial/X10M_1k_dense: 151
MultiLogReg train ict=2 on mbperftest/binomial/X10M_1k_dense: 123
-- Running runMultiLogReg on 10M_1k_dense (all configs):
MultiLogReg train ict=0 on mbperftest/binomial/X10M_1k_dense: 114
MultiLogReg train ict=1 on mbperftest/binomial/X10M_1k_dense: 148
MultiLogReg train ict=2 on mbperftest/binomial/X10M_1k_dense: 119
> Perftest: MultiLogReg performance variation
> -------------------------------------------
>
> Key: SYSTEMML-1072
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1072
> Project: SystemML
> Issue Type: Bug
> Reporter: Glenn Weidner
>
> Times (8G, k150):
> SPARK RUN MULTINOMIAL EXPERIMENTS: Thu Oct 27
> -- Running runMultiLogReg on 1M_1k_dense (all configs)
> MultiLogReg train ict=0 on mbperftest/multinomial/X1M_1k_dense_k150: 141
> MultiLogReg train ict=1 on mbperftest/multinomial/X1M_1k_dense_k150: 166
> MultiLogReg train ict=2 on mbperftest/multinomial/X1M_1k_dense_k150: 417
> SPARK RUN MULTINOMIAL EXPERIMENTS: Wed Oct 12
> -- Running runMultiLogReg on 1M_1k_dense (all configs)
> MultiLogReg train ict=0 on mbperftest/multinomial/X1M_1k_dense_k150: 127
> MultiLogReg train ict=1 on mbperftest/multinomial/X1M_1k_dense_k150: 151
> MultiLogReg train ict=2 on mbperftest/multinomial/X1M_1k_dense_k150: 354
> Times (80G, dense):
> SPARK RUN BINOMIAL EXPERIMENTS: Wed Oct 26
> -- Running runMultiLogReg on 10M_1k_dense (all configs):
> MultiLogReg train ict=0 on mbperftest/binomial/X10M_1k_dense: 143
> MultiLogReg train ict=1 on mbperftest/binomial/X10M_1k_dense: 147
> MultiLogReg train ict=2 on mbperftest/binomial/X10M_1k_dense: 138
> SPARK RUN BINOMIAL EXPERIMENTS: Wed Oct 12
> -- Running runMultiLogReg on 10M_1k_dense (all configs):
> MultiLogReg train ict=0 on mbperftest/binomial/X10M_1k_dense: 90
> MultiLogReg train ict=1 on mbperftest/binomial/X10M_1k_dense: 84
> MultiLogReg train ict=2 on mbperftest/binomial/X10M_1k_dense: 94
> Times (80G, k150):
> SPARK RUN MULTINOMIAL EXPERIMENTS: Wed Oct 26
> -- Running runMultiLogReg on 10M_1k_dense (all configs)
> MultiLogReg train ict=0 on mbperftest/multinomial/X10M_1k_dense_k150: 1300
> MultiLogReg train ict=1 on mbperftest/multinomial/X10M_1k_dense_k150: 1265
> MultiLogReg train ict=2 on mbperftest/multinomial/X10M_1k_dense_k150: 1677
> SPARK RUN MULTINOMIAL EXPERIMENTS: Wed Oct 12
> -- Running runMultiLogReg on 10M_1k_dense (all configs)
> MultiLogReg train ict=0 on mbperftest/multinomial/X10M_1k_dense_k150: 1639
> MultiLogReg train ict=1 on mbperftest/multinomial/X10M_1k_dense_k150: 1790
> MultiLogReg train ict=2 on mbperftest/multinomial/X10M_1k_dense_k150: 2541
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
