[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-600185011 I'm fine with merging into 3.0 too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-599577586 jenkins, test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-599368497 LGTM. Could you add some tests to cover both <256 and >=256 cases? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-595595323 @yma11 You also need to try old generation CPUs / RAMs to confirm 256 is absolutely safe. We cannot assume all users are on the latest generation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-595592042 I'm okay with a hardcoded constant. We are start with a safer one, say 512, if there are concerns around environment differences. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-587132383 @yma11 Thanks for re-running the benchmark! Could you confirm whether AVX is used by MKL/OpenBLAS? I think that is why it is faster on level-1 on large vectors. It seems reasonable to switch between f2jBLAS and nativeBLAS based on vector size (256?). There is still one scenario we need to check. I don't think users will remember limiting native BLAS threads to 1. They might have 8 concurrent tasks running on a 8-core worker while each using multi-thread BLAS. I saw cases before that it significantly degrade the performance. It might be worth running a benchmark for this scenario. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-586387070 Please also limit the blas threads to 1 because one core per task is the common setting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
mengxr commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-585864274 Could you provide reproducible microbenchmark results for both OpenBLAS and MKL on arrays of different sizes? Last time we tested it it didn't help. But that was years ago. Note that JVM also improved SIMD. We should also check whether we can improve along that direction. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
