[ https://issues.apache.org/jira/browse/SPARK-21688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125353#comment-16125353 ]
Vincent commented on SPARK-21688: --------------------------------- sorry for late reply. Yes, It's simple and easy to check the env variables in the code, but I don't think that's a right thing to do. First, I still believe that if a user decides to run on native blas to speed up his/her application, he/she should be aware of proper settings as mentioned in https://issues.apache.org/jira/browse/SPARK-21305. they can set 1, 2... or any arbitrary number of threads for native blas that can give them better performance; Second, there are a bunch of BLAS variations, MKL, Openblas, Atlas, Cublas ...etc, each one has a different variable name for this setting, check all these variant settings in the code doesn't seem right. > performance improvement in mllib SVM with native BLAS > ------------------------------------------------------ > > Key: SPARK-21688 > URL: https://issues.apache.org/jira/browse/SPARK-21688 > Project: Spark > Issue Type: Improvement > Components: MLlib > Affects Versions: 2.2.0 > Environment: 4 nodes: 1 master node, 3 worker nodes > model name : Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz > Memory : 180G > num of core per node: 10 > Reporter: Vincent > Priority: Minor > Attachments: ddot unitest.png, mllib svm training.png, > native-trywait.png, svm1.png, svm2.png, svm-mkl-1.png, svm-mkl-2.png > > > in current mllib SVM implementation, we found that the CPU is not fully > utilized, one reason is that f2j blas is set to be used in the HingeGradient > computation. As we found out earlier > (https://issues.apache.org/jira/browse/SPARK-21305) that with proper > settings, native blas is generally better than f2j on the uni-test level, > here we make the blas operations in SVM go with MKL blas and get an end to > end performance report showing that in most cases native blas outperformance > f2j blas up to 50%. > So, we suggest removing those f2j-fixed calling and going for native blas if > available. If this proposal is acceptable, we will move on to benchmark other > algorithms impacted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org