[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-12-08 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991098#comment-16991098
 ] 

WuZeyi commented on SPARK-29553:


[~srowen] Sir, I use the cmd line “export OPENBLAS_NUM_THREADS=1” to set env of 
OS on the machine where my executor is launched,  but it doesn't work either.

I propose a concrete change to the docs, PTAL.

https://github.com/apache/spark/pull/26801

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979784#comment-16979784
 ] 

WuZeyi commented on SPARK-29553:


I also try the cmd "export OPENBLAS_NUM_THREADS=1", but it doesn't work.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979781#comment-16979781
 ] 

WuZeyi commented on SPARK-29553:


[~srowen] I set it in the spark/conf/spark-env.sh which machine my executor is 
launched on instead of the client side, but It doesn't work. 

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979781#comment-16979781
 ] 

WuZeyi edited comment on SPARK-29553 at 11/22/19 2:29 AM:
--

[~srowen] I set it in the spark/conf/spark-env.sh which machine my executor is 
launched on instead of the client side, but it doesn't work. I'm confused.


was (Author: zeyiii):
[~srowen] I set it in the spark/conf/spark-env.sh which machine my executor is 
launched on instead of the client side, but It doesn't work. 

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979775#comment-16979775
 ] 

WuZeyi edited comment on SPARK-29553 at 11/22/19 2:18 AM:
--

[~srowen]  But if I set it with spark.env only, it doesn't take effect on 
executors.If so, the performance is pool. How should I do to solve?


was (Author: zeyiii):
[~srowen]  But if I set it with spark.env only, it doesn't take effect on 
executors.If so, the performance is pool.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979775#comment-16979775
 ] 

WuZeyi edited comment on SPARK-29553 at 11/22/19 2:10 AM:
--

[~srowen]  But if I set it with spark.env only, it doesn't take effect on 
executors.If so, the performance is pool.


was (Author: zeyiii):
But if I set it with spark.env only, it doesn't take effect on executors.If so, 
the performance is pool.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979775#comment-16979775
 ] 

WuZeyi edited comment on SPARK-29553 at 11/22/19 2:10 AM:
--

But if I set it with spark.env only, it doesn't take effect on executors.If so, 
the performance is pool.


was (Author: zeyiii):
But if I set it with spark.env only, it doesn't take effact on executors.If so, 
the performance is pool.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979775#comment-16979775
 ] 

WuZeyi commented on SPARK-29553:


But if I set it with spark.env, it doesn't take effact on executors.If so, the 
performance is pool.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979775#comment-16979775
 ] 

WuZeyi edited comment on SPARK-29553 at 11/22/19 2:09 AM:
--

But if I set it with spark.env only, it doesn't take effact on executors.If so, 
the performance is pool.


was (Author: zeyiii):
But if I set it with spark.env, it doesn't take effact on executors.If so, the 
performance is pool.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979116#comment-16979116
 ] 

WuZeyi commented on SPARK-29553:


[~srowen] Yes,you are right.

This isn't a bug.The processes run normally，but the performance is not good.

But the conf/spark-env.sh.template write that:

!image-2019-11-21-17-08-15-797.png|width=737,height=76!

But I think it is more reasonable to add it to 
conf/spark-defaults.conf.template.

Because I modify spark-defaults.conf to set 
spark.executorEnv.OPENBLAS_NUM_THREADS=1 ,the performance improves.It really 
came into effect.

 

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-21 Thread WuZeyi (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WuZeyi updated SPARK-29553:
---
Attachment: image-2019-11-21-17-08-15-797.png

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-20 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978889#comment-16978889
 ] 

WuZeyi edited comment on SPARK-29553 at 11/21/19 1:42 AM:
--

[~srowen] I use Hibench to run it on only one machine on Yarn. 
 The cmd is like this:

/home/installed/spark-2.3.0/bin/spark-submit --properties-file 
/home/heyan/hibench/report/kmeans/spark/conf/sparkbench/spark.conf --class 
com.intel.hibench.sparkbench.ml.DenseKMeans --master yarn-client 
--num-executors 19 --executor-cores 5 --executor-memory 20g 
/home/heyan/hibench/sparkbench/assembly/target/sparkbench-assembly-7.0-dist.jar 
-k 10 --numIterations 10 --initMode Random 
hdfs://172.19.31.21:9000/HiBench/Kmeans/Input/samples

 

All jvm processes:

[root]# jps
82979 ResourceManager
26115 Jps
24776 CoarseGrainedExecutorBackend
25449 CoarseGrainedExecutorBackend
25450 CoarseGrainedExecutorBackend
25451 CoarseGrainedExecutorBackend
24653 CoarseGrainedExecutorBackend
23694 Worker
21359 DataNode
23313 Master
25009 CoarseGrainedExecutorBackend
25010 CoarseGrainedExecutorBackend
21813 SecondaryNameNode
83483 JobHistoryServer
25978 CoarseGrainedExecutorBackend
23995 HistoryServer
25979 CoarseGrainedExecutorBackend
83165 NodeManager
25980 CoarseGrainedExecutorBackend
22845 SparkSubmit
24478 ExecutorLauncher
21055 NameNode


was (Author: zeyiii):
I use Hibench to run it on only one machine on Yarn. 
The cmd is like this:

/home/installed/spark-2.3.0/bin/spark-submit --properties-file 
/home/heyan/hibench/report/kmeans/spark/conf/sparkbench/spark.conf --class 
com.intel.hibench.sparkbench.ml.DenseKMeans --master yarn-client 
--num-executors 19 --executor-cores 5 --executor-memory 20g 
/home/heyan/hibench/sparkbench/assembly/target/sparkbench-assembly-7.0-dist.jar 
-k 10 --numIterations 10 --initMode Random 
hdfs://172.19.31.21:9000/HiBench/Kmeans/Input/samples

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-20 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978889#comment-16978889
 ] 

WuZeyi commented on SPARK-29553:


I use Hibench to run it on only one machine on Yarn. 
The cmd is like this:

/home/installed/spark-2.3.0/bin/spark-submit --properties-file 
/home/heyan/hibench/report/kmeans/spark/conf/sparkbench/spark.conf --class 
com.intel.hibench.sparkbench.ml.DenseKMeans --master yarn-client 
--num-executors 19 --executor-cores 5 --executor-memory 20g 
/home/heyan/hibench/sparkbench/assembly/target/sparkbench-assembly-7.0-dist.jar 
-k 10 --numIterations 10 --initMode Random 
hdfs://172.19.31.21:9000/HiBench/Kmeans/Input/samples

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-20 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977236#comment-16977236
 ] 

WuZeyi edited comment on SPARK-29553 at 11/20/19 12:22 PM:
---

[~srowen] Thanks for Looking.

I modify spark-env.sh,and the flame graph is like this:

!image-2019-11-19-16-11-43-130.png!

Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color}

{color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color}

It means that it is still multi-thread if I modify spark-env.sh.

If I modify spark.conf to set  
{color:#ff}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，it works and the 
the performance improve.

IMHO, spark-env.sh only set the evn of the spark-sumbit process, but it doesn't 
work in the executor processes.


was (Author: zeyiii):
Thanks for Looking.

I modify spark-env.sh,and the flame graph is like this:

!image-2019-11-19-16-11-43-130.png!

Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color}

{color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color}

It means that it is still multi-thread if I modify spark-env.sh.

If I modify spark.conf to set  
{color:#ff}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，it works and the 
the performance improve.

IMHO, spark-env.sh only set the evn of the spark-sumbit process, but it doesn't 
work in the executor processes.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-19 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977236#comment-16977236
 ] 

WuZeyi edited comment on SPARK-29553 at 11/19/19 8:26 AM:
--

Thanks for Looking.

I modify spark-env.sh,and the flame graph is like this:

!image-2019-11-19-16-11-43-130.png!

Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color}

{color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color}

It means that it is still multi-thread if I modify spark-env.sh.

If I modify spark.conf to set  
{color:#ff}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，it works and the 
the performance improve.

IMHO, spark-env.sh only set the evn of the spark-sumbit process, but it doesn't 
work in the executor processes.


was (Author: zeyiii):
Thanks for Looking.

I modify spark-env.sh,and the flame graph is like this:

!image-2019-11-19-16-11-43-130.png!

Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color}

{color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color}

It means that it is still multi-thread if I modify spark-env.sh.

If I modify spark.conf to set  
{color:#ff}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，it works and the 
the performance improve.

IMHO, spark-env,sh only set the evn of the spark-sumbit process, but it doesn't 
work in the executor processes.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-19 Thread WuZeyi (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977236#comment-16977236
 ] 

WuZeyi commented on SPARK-29553:


Thanks for Looking.

I modify spark-env.sh,and the flame graph is like this:

!image-2019-11-19-16-11-43-130.png!

Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color}

{color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color}

It means that it is still multi-thread if I modify spark-env.sh.

If I modify spark.conf to set  
{color:#ff}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，it works and the 
the performance improve.

IMHO, spark-env,sh only set the evn of the spark-sumbit process, but it doesn't 
work in the executor processes.

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-19 Thread WuZeyi (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WuZeyi updated SPARK-29553:
---
Attachment: image-2019-11-19-16-13-30-723.png

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png, 
> image-2019-11-19-16-13-30-723.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-11-19 Thread WuZeyi (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WuZeyi updated SPARK-29553:
---
Attachment: image-2019-11-19-16-11-43-130.png

> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
>  Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-10-22 Thread WuZeyi (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WuZeyi updated SPARK-29553:
---
Description: 
I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance on 
Yarn.

The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to disable 
multi-threading of OpenBLAS, but it does not take effect.

I modify {color:#ff}spark.conf{color} to set  
{color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
performance improve.
  
  
 I think MKL_NUM_THREADS is the same.
  

  was:
I use {color:#FF}native BLAS{color} to improvement ML/MLLIB performance on 
Yarn.

The file {color:#FF}spark-env.sh{color} which is modified by [SPARK-21305] 
said that I should set {color:#FF}OPENBLAS_NUM_THREADS=1{color} to disable 
multi-threading of OpenBLAS, but it does not take effect.

I modify {color:#FF}spark.conf{color} to set  OPENBLAS_NUM_THREADS=1，and 
the performance improve.
 
 
I think MKL_NUM_THREADS is the same.
 


> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
>  Issue Type: Improvement
>  Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Major
>  Labels: performance
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance 
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305 
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to 
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set  
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the 
> performance improve.
>   
>   
>  I think MKL_NUM_THREADS is the same.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance

2019-10-22 Thread WuZeyi (Jira) 
WuZeyi created SPARK-29553:
--

 Summary: This problemis about using native BLAS to improvement 
ML/MLLIB performance
 Key: SPARK-29553
 URL: https://issues.apache.org/jira/browse/SPARK-29553
 Project: Spark
  Issue Type: Improvement
  Components: ML, MLlib
Affects Versions: 2.4.4, 2.3.0
Reporter: WuZeyi


I use {color:#FF}native BLAS{color} to improvement ML/MLLIB performance on 
Yarn.

The file {color:#FF}spark-env.sh{color} which is modified by [SPARK-21305] 
said that I should set {color:#FF}OPENBLAS_NUM_THREADS=1{color} to disable 
multi-threading of OpenBLAS, but it does not take effect.

I modify {color:#FF}spark.conf{color} to set  OPENBLAS_NUM_THREADS=1，and 
the performance improve.
 
 
I think MKL_NUM_THREADS is the same.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org