[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16594 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73419/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73419/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73419/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73402/ Test FAILed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-23 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16594 LGTM, pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73402/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16594 LGTM except one comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73295/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73295/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73295/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73200/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73200/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #73200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73200/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-07 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 ok I'll modify it with this new command. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 me 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16594 I like the idea proposed by rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-02-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16594 ok here is an idea how about ``` explain stats xxx ``` as the way to add stats? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71921/ Test FAILed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71921/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 @gatorsmile I just did a quick fix to show how the improved stats look like. If @rxin @hvanhovell accept the change proposed in this pr, I'll update to remove the flag :) --- If your project is set

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 I still do not think using an internal configuration is a user friendly way to show the plan costs. Using this way, we do not want users to see it. --- If your project is set up for it, you can

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71906/ Test FAILed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71906/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71906/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 @rxin @gatorsmile @hvanhovell I've updated this pr and make stats much more readable: SizeInBytes is shown in units of B, KB, MB ... PB, e.g. `sizeInBytes=228.8 GB`, and if it's too

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 @ron8hu Yes, I've already updated this pr. I'll present an example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread ron8hu
Github user ron8hu commented on the issue: https://github.com/apache/spark/pull/16594 To show a very large Long number, there is no need to print out every digit in the number. We can use exponent. For example, a number 120,000,000,005,123 can be printed as 1.2*10**14, where 10**14

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71847/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71847/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71847/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 :- ) No perfect solution, but we should use the [metric prefix](https://en.wikipedia.org/wiki/Metric_prefix) when the number is huge. --- If your project is set up for it, you can reply to

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 SQLServer has three ways to show the plan: graphical plans, text plans, and XML plans. Actually, it is pretty advanced. When using the text plans, users can set the output formats: 1.

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 As of MySQL 5.7.3, the EXPLAIN statement is changed so that the effect of the EXTENDED keyword is always enabled. ``` mysql> EXPLAIN EXTENDED -> SELECT t1.a, t1.a IN (SELECT t2.a

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 PostgreSQL has [a few different options in the EXPLAIN command](https://www.postgresql.org/docs/9.3/static/sql-explain.html): ``` EXPLAIN SELECT * FROM foo WHERE i = 4;

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 DB2 has a tool to format the contents of the EXPLAIN tables. Below is an example of the output with explanation: ![screenshot 2017-01-22 21 05

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16594 Let us do some research how the other RDBMSs are doing it? For example, Oracle ``` SQL> explain plan for select * from product; Explained. SQL> select * from

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 @rxin Can we add a flag to enable or disable it? Currently there's no other way to see size and row count except debugging. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16594 sorry this explain plan makes no sense -- it is impossible to read. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 @hvanhovell I've updated the description which shows a simple example. The explained plan will become hard to read when joining many tables and sizeInBytes is computed by the simple way

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16594 @wzhfy could you add an example of this to the PR description? I am a bit worried that the explain plans will become (much) harder to read. I am also interested to see if this new explain output

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71588/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71588/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71588/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71508/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71508/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71508/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71430/ Test PASSed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71430/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71430/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16594 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71424/ Test FAILed. ---

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16594 **[Test build #71424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71424/testReport)** for PR 16594 at commit

[GitHub] spark issue #16594: [SPARK-17078] [SQL] Show stats when explain

2017-01-15 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16594 cc @rxin @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,