[GitHub] spark pull request: [SPARK-14489][SPARK-14153][ML][PYSPARK] Suppor...

2016-04-21 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/12577#issuecomment-213071809 It sounds like people are running into this often when using cross-validation - would it make sense to also mention this in the k-fold docstring or examples? (Just a mi

[GitHub] spark pull request: [SPARK-14369][SQL] Locality support for FileSc...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12527#issuecomment-213071922 **[Test build #56559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56559/consoleFull)** for PR 12527 at commit [`e0bfa3e`](https://g

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213070653 I like the idea of just warning, rather than making the rules slightly different. It's simpler. I suppose it only benefits those who are kind of looking for style failur

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213070508 **[Test build #56575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56575/consoleFull)** for PR 12553 at commit [`83b131a`](https://gi

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12561#issuecomment-213070005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12561#issuecomment-213069997 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/12580#discussion_r60637137 --- Diff: project/SparkBuild.scala --- @@ -254,7 +254,7 @@ object SparkBuild extends PomBuild { val mimaProjects = allProjects.filterNot { x =>

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12561#issuecomment-213069188 **[Test build #56555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56555/consoleFull)** for PR 12561 at commit [`f74b381`](https://g

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213067543 The maven plugin has a `failOnViolation` configuration; I assume the sbt one has something similar. Maybe just setting that to `false` during test or compile? Then peopl

[GitHub] spark pull request: [SPARK-12469][CORE] Consistent accumulators fo...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-213066879 **[Test build #56574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56574/consoleFull)** for PR 11105 at commit [`e598d88`](https://gi

[GitHub] spark pull request: [SPARK-14763] [SQL] fix subquery resolution

2016-04-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/12539#issuecomment-213066258 cc @hvanhovell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-14763] [SQL] fix subquery resolution

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12539#issuecomment-213065907 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-14801][SQL] Move MetastoreRelation to i...

2016-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12567 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-14763] [SQL] fix subquery resolution

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12539#issuecomment-213065908 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-14763] [SQL] fix subquery resolution

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12539#issuecomment-213065457 **[Test build #56556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56556/consoleFull)** for PR 12539 at commit [`fd8c75c`](https://g

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213065242 **[Test build #56572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56572/consoleFull)** for PR 12580 at commit [`30e541d`](https://g

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213065256 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213065258 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-14801][SQL] Move MetastoreRelation to i...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12567#issuecomment-213065135 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.inS...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12562#issuecomment-213064322 **[Test build #56573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56573/consoleFull)** for PR 12562 at commit [`c7a6d9b`](https://gi

[GitHub] spark pull request: [SPARK-14699][Core]Stop endpoints before closi...

2016-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12481 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: removed expectedType from Param __init__()

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12581#issuecomment-213064044 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your p

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213064363 **[Test build #56572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56572/consoleFull)** for PR 12580 at commit [`30e541d`](https://gi

[GitHub] spark pull request: [SPARK-14798][SQL] Move native command and scr...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12564#issuecomment-213063745 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: removed expectedType from Param __init__()

2016-04-21 Thread jasoncl
GitHub user jasoncl opened a pull request: https://github.com/apache/spark/pull/12581 removed expectedType from Param __init__() ## What changes were proposed in this pull request? Removed expectedType arg from PySpark Param __init__, as suggested by the JIRA. ## How wa

[GitHub] spark pull request: [SPARK-14798][SQL] Move native command and scr...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12564#issuecomment-213063751 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread ericl
Github user ericl commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213063456 Same with import ordering. We can probably turn those to WARN on compile step. Trying to figure out how to do that without forking the entire xml config... --- If your

[GitHub] spark pull request: [SPARK-14798][SQL] Move native command and scr...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12564#issuecomment-213063158 **[Test build #56557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56557/consoleFull)** for PR 12564 at commit [`a38eaf1`](https://g

[GitHub] spark pull request: [SPARK-14699][Core]Stop endpoints before closi...

2016-04-21 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/12481#issuecomment-213062730 Thanks, merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/12580#discussion_r60635076 --- Diff: project/SparkBuild.scala --- @@ -254,7 +254,7 @@ object SparkBuild extends PomBuild { val mimaProjects = allProjects.filterNot { x =>

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213061387 also this should probably go into sql? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/12580#discussion_r60635078 --- Diff: compatibility/pom.xml --- @@ -0,0 +1,66 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12580#issuecomment-213061073 I'd name this something like spark-1.x-compatibility (is dot allowed in the name?) --- If your project is set up for it, you can reply to this email and have your reply a

[GitHub] spark pull request: [SPARK-14801][SQL] Move MetastoreRelation to i...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12567#issuecomment-213058613 **[Test build #2848 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2848/consoleFull)** for PR 12567 at commit [`0a6bf1a`](https://g

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213058828 > There is an environment variable here that you can set in your profile Yeah, I saw that, but I don't start sbt every time I run something, it's just running in

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12580#discussion_r60634983 --- Diff: compatibility/pom.xml --- @@ -0,0 +1,66 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-inst

[GitHub] spark pull request: [SPARK-14807] Create a compatibility module

2016-04-21 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/12580 [SPARK-14807] Create a compatibility module ## What changes were proposed in this pull request? This PR creates a compatibility module, which will host HiveContext in Spark 2.0 (moving HiveC

[GitHub] spark pull request: [SPARK-13330][PYSPARK] PYTHONHASHSEED is not p...

2016-04-21 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/11211#issuecomment-213060214 Okay, so why don't we go ahead and use `setExecutorEnv` then? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-14798][SQL] Move native command and scr...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12564#issuecomment-213060413 **[Test build #56571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56571/consoleFull)** for PR 12564 at commit [`21b5397`](https://gi

[GitHub] spark pull request: [SPARK-9656] [MLlib] [Python] Add missing meth...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9441#issuecomment-213059784 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9656] [MLlib] [Python] Add missing meth...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9441#issuecomment-213059786 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/5

[GitHub] spark pull request: [SPARK-14570][ML] Log instrumentation in Rando...

2016-04-21 Thread BenFradet
Github user BenFradet commented on the pull request: https://github.com/apache/spark/pull/12536#issuecomment-213058212 @thunterdb thanks for review, will fix the points you brought up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [SPARK-9656] [MLlib] [Python] Add missing meth...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9441#issuecomment-213059645 **[Test build #56565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56565/consoleFull)** for PR 9441 at commit [`0f82902`](https://git

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12561 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12561#issuecomment-213058893 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: [SPARK-14571][ML]Log instrumentation in ALS

2016-04-21 Thread wangmiao1981
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/12560#issuecomment-213059111 @thunterdb train method has count information, but it will change the signature of the train method. I am learning how to avoid collect and changing signature. Tha

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread ericl
Github user ericl commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213058840 The println() is a good point. Let me see if I can turn that to a warn just in the compile check. --- If your project is set up for it, you can reply to this email and h

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213058313 There is an environment variable here that you can set in your profile to never trigger this. --- If your project is set up for it, you can reply to this email and have y

[GitHub] spark pull request: [SPARK-14433][PySpark][ML]:PySpark ml Gaussian...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12402#issuecomment-213057588 **[Test build #56570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56570/consoleFull)** for PR 12402 at commit [`7fa5713`](https://gi

[GitHub] spark pull request: [SPARK-14801][SQL] Move MetastoreRelation to i...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12567#issuecomment-213056872 **[Test build #2846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2846/consoleFull)** for PR 12567 at commit [`0a6bf1a`](https://

[GitHub] spark pull request: [SPARK-14433][PySpark][ML]:PySpark ml Gaussian...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12402#issuecomment-213056046 **[Test build #56569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56569/consoleFull)** for PR 12402 at commit [`a111e38`](https://gi

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213056053 **[Test build #56568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56568/consoleFull)** for PR 12553 at commit [`530f039`](https://gi

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213055878 I'm not strongly opposed, it's just that when debugging things I sometimes add a bunch of "println" statements in the code to trace what's going on. Having the style che

[GitHub] spark pull request: [SPARK-14669] [SQL] Fix some SQL metrics in co...

2016-04-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/12425#discussion_r60632391 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoin.scala --- @@ -57,8 +59,13 @@ case class ShuffledHashJoin( C

[GitHub] spark pull request: [SPARK-14669] [SQL] Fix some SQL metrics in co...

2016-04-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/12425#discussion_r60632404 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoin.scala --- @@ -57,8 +59,13 @@ case class ShuffledHashJoin( C

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213054626 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10574] [ML] [MLlib] HashingTF supports ...

2016-04-21 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/12498#discussion_r60631689 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala --- @@ -66,8 +83,9 @@ class HashingTF(val numFeatures: Int) extends Serializ

[GitHub] spark pull request: [SPARK-10001][Core] Interrupt tasks in repl wi...

2016-04-21 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/12557#discussion_r60631324 --- Diff: repl/src/main/scala/org/apache/spark/repl/Signaling.scala --- @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213049604 **[Test build #56566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56566/consoleFull)** for PR 12553 at commit [`530f039`](https://g

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213049696 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213049691 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.min...

2016-04-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12562#issuecomment-213048699 Thank you so much, @rxin and @marmbrus ! I will update soon like the following according to the comments. - spark.sql.optimizer.minSetSize -> spark.sql.opt

[GitHub] spark pull request: Investigate flaky test

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12565#issuecomment-213048303 Weird - it just passes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Investigate flaky test

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12565#issuecomment-213047409 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: Investigate flaky test

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12565#issuecomment-213047419 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213046520 I believe @vanzin was the main person that didn't like this. Copy him here just in case. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: Investigate flaky test

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12565#issuecomment-213046527 **[Test build #56558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56558/consoleFull)** for PR 12565 at commit [`64447a4`](https://g

[GitHub] spark pull request: [SPARK-14736][core] Deadlock in registering ap...

2016-04-21 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/12506#issuecomment-213046545 No, you are right - this is called only from the event loop - which should ensure thread safety. I misread where the re-registeration was happening as outside of the

[GitHub] spark pull request: [SPARK-14795][SQL] Remove the use of Hive's va...

2016-04-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12561#issuecomment-213045905 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request: [SPARK-14669] [SQL] Fix some SQL metrics in co...

2016-04-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/12425#discussion_r60630187 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Sort.scala --- @@ -87,10 +89,12 @@ case class Sort( // Remember spill data size o

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213045255 **[Test build #56567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56567/consoleFull)** for PR 12555 at commit [`8b88370`](https://gi

[GitHub] spark pull request: [SPARK-14801][SQL] Move MetastoreRelation to i...

2016-04-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12567#issuecomment-213045030 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread ericl
Github user ericl commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213044120 @rxin @srowen I added a check for `NOLINT_ON_COMPILE` in the environment, which disables this check if set. The user can add this to their shell profile if they want. I c

[GitHub] spark pull request: [SPARK-14669] [SQL] Fix some SQL metrics in co...

2016-04-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/12425#discussion_r60629466 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala --- @@ -26,6 +26,7 @@ import com.google.common.io.ByteStreams

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.min...

2016-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/12562#discussion_r60629326 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -54,10 +54,16 @@ object SQLConf { val OPTIMIZER_MAX_ITERAT

[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12537#issuecomment-213043117 **[Test build #2845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2845/consoleFull)** for PR 12537 at commit [`d48b19e`](https://

[GitHub] spark pull request: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12553#issuecomment-213042923 **[Test build #56566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56566/consoleFull)** for PR 12553 at commit [`530f039`](https://gi

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.min...

2016-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/12562#discussion_r60629117 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeInSuite.scala --- @@ -36,7 +39,7 @@ class OptimizeInSuite extends Pla

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.min...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12562#issuecomment-213042352 maybe inSetConversionThreshold? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.min...

2016-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/12562#discussion_r60628971 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeInSuite.scala --- @@ -17,11 +17,14 @@ package org.apache.s

[GitHub] spark pull request: [SPARK-10574] [ML] [MLlib] HashingTF supports ...

2016-04-21 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/12498#discussion_r60628509 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala --- @@ -66,8 +83,9 @@ class HashingTF(val numFeatures: Int) extends Serializ

[GitHub] spark pull request: [SPARK-14670][SQL] Allow updating SQLMetrics o...

2016-04-21 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/12427#issuecomment-213041496 What will happen if reusing a DataFrame? `localValue` in the driver will be a sum of all jobs if a DataFrame? --- If your project is set up for it, you can reply to th

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-213041583 **[Test build #56564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56564/consoleFull)** for PR 12081 at commit [`fa5f124`](https://gi

[GitHub] spark pull request: [SPARK-14680][SQL]Support all datatypes to use...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12440#issuecomment-213041578 **[Test build #56563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56563/consoleFull)** for PR 12440 at commit [`cf45b67`](https://gi

[GitHub] spark pull request: [SPARK-9656] [MLlib] [Python] Add missing meth...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9441#issuecomment-213041589 **[Test build #56565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56565/consoleFull)** for PR 9441 at commit [`0f82902`](https://gith

[GitHub] spark pull request: [SPARK-14680][SQL]Support all datatypes to use...

2016-04-21 Thread sameeragarwal
Github user sameeragarwal commented on a diff in the pull request: https://github.com/apache/spark/pull/12440#discussion_r60628217 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala --- @@ -224,6 +224,127 @@ class BenchmarkWholeStageC

[GitHub] spark pull request: [MINOR][ML][PYTHON][DOC] Remove use of JavaMLW...

2016-04-21 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/12542#issuecomment-213040834 CC: @yanboliang since you've worked on this code before --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14671][ML] Pipeline setStages should ha...

2016-04-21 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/12430#issuecomment-213040994 CC @thunterdb --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: [SPARK-14738][BUILD] Separate docker integrati...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12508#issuecomment-213040196 **[Test build #56562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56562/consoleFull)** for PR 12508 at commit [`5867866`](https://gi

[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-213039330 **[Test build #2847 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2847/consoleFull)** for PR 12455 at commit [`b96cde1`](https://g

[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-213039201 Thanks - let's merge this first once Jenkins pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14799][SQL] Remove MetastoreRelation de...

2016-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12566 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-10001][Core] Interrupt tasks in repl wi...

2016-04-21 Thread jodersky
Github user jodersky commented on a diff in the pull request: https://github.com/apache/spark/pull/12557#discussion_r60626992 --- Diff: repl/src/main/scala/org/apache/spark/repl/Signaling.scala --- @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] spark pull request: [SPARK-14789][SQL] Move FunctionResourceLoader...

2016-04-21 Thread andrewor14
Github user andrewor14 closed the pull request at: https://github.com/apache/spark/pull/12554 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-14670][SQL] Allow updating SQLMetrics o...

2016-04-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/12427#discussion_r60626984 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala --- @@ -38,7 +38,13 @@ case class SparkListenerSQLExecutionStart(

[GitHub] spark pull request: [SPARK-14799][SQL] Remove MetastoreRelation de...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12566#issuecomment-213038787 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14734][ML][MLLIB] Added asML, fromML me...

2016-04-21 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/12504#issuecomment-213038664 @dbtsai anything else? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-14790] Always run scalastyle on sbt com...

2016-04-21 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12555#issuecomment-213038221 Would it be possible to disable this via some other thing? Then we satisfy both worlds. I just took a look at the pr backlog and almost every pr failed style test once!

[GitHub] spark pull request: [Spark-14686] Allow setting local properties t...

2016-04-21 Thread mtustin-handy
Github user mtustin-handy commented on the pull request: https://github.com/apache/spark/pull/12456#issuecomment-213037791 @jerryshao Makes sense. Would having an explicitly inheritable facility, or a way to switch the spark context into inheritable mode (whatever that design looks li

[GitHub] spark pull request: [SPARK-14783] Preserve full exception stacktra...

2016-04-21 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/12548#issuecomment-213037748 nice --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

<    3   4   5   6   7   8   9   10   11   12   >