[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21777 @kiszk Can you close this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97552/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21777 **[Test build #97552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97552/testReport)** for PR 21777 at commit [`a7ebdfd`](https://github.com/apache/spark/commit/a7ebdfdb2dacbb73e21de5f6c5c627faa5d551cd). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4099/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21777 **[Test build #97552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97552/testReport)** for PR 21777 at commit [`a7ebdfd`](https://github.com/apache/spark/commit/a7ebdfdb2dacbb73e21de5f6c5c627faa5d551cd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21777 @maropu Based on my understanding, the bytecodes generated by these two compilers are different. That is why the performance should be different. Previously, I expected the bytecodes generated by JDK compiler can be better optimized by JIT, compared with the one generated by Janino. Maybe our JVM internal experts @kiszk and @rednaxelafx can give more guidance. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21777 btw, it seems this pr exceeds the current timeout...ãAny way to temporarily make the timeout longer? We always need to configure timeout in the Jenkins-side like https://github.com/apache/spark/pull/20222#issuecomment-357004091? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21777 > Except the TPC-DS queries, are we able to find some workloads that could perform faster using the bytecode generated by the JDK compiler? Since I don't have real workloads or non-TPCDS queries, it is difficult for me to find the workload can you? @kiszk > Or, does that mean Janino compiler is always better than JDK compiler? (that does not sound true to me) No, I meant averaged performance. I'm not a Java/JDK expert, so I don't completely understand how the two compilers work. IIUC, these compilers might apply simple optimization a little though, they simply convert Java code into bytecode? The hotspot has a responsibility for optimization? Anyway, we'd be better to wait for other developers. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21777 @maropu Except the TPC-DS queries, are we able to find some workloads that could perform faster using the bytecode generated by the JDK compiler? Or, does that mean Janino compiler is always better than JDK compiler? (that does not sound true to me) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21777 Also cc @rednaxelafx --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21777 Yea, as you said, the JDK compiler generates different bytecode though, I couldn't get obvious performance gains for TPCDS as compared to the janino one. So, I couldn't find the strong reason to implement this in terms of performance; https://docs.google.com/spreadsheets/d/1Mgdd9dfFaACXOUHqKfaeKrj09hB3X1j9sKTJlJ6UM6w/edit#gid=1236423798 As another viewpoint, I think it might be useful to check if the generated Java code of Spark could be compiled by the JDK compiler (JDK8 code compatibility checks). But, since the compilation of the JDK compiler is too slow (see the performance values in the google spread sheet above), IMO it is impractical to check this in Jenkins (I found it took 7~8 hours to run the tests of the `spark/sql` only in aws instances). WDYT? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21777 @maropu @kiszk Thank you for taking this effort! Based on my initial understanding, the code generated by the JDK compiler can be better optimized by JIT in many cases. Is my understanding right? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93038/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21777 **[Test build #93038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93038/testReport)** for PR 21777 at commit [`523bf3d`](https://github.com/apache/spark/commit/523bf3d96d21ebf07aa87b1842ea58a840d2c33b). * This patch **fails from timeout after a configured wait of \`300m\`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21777 **[Test build #93038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93038/testReport)** for PR 21777 at commit [`523bf3d`](https://github.com/apache/spark/commit/523bf3d96d21ebf07aa87b1842ea58a840d2c33b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21777: [WIP][SPARK-24498][SQL] Add JDK compiler for runtime cod...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/975/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
