[GitHub] spark pull request #19841: [SPARK-22642][SQL] the createdTempDir will not be...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19841#discussion_r154490891 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -104,147 +105,153 @@ case class

[GitHub] spark pull request #19841: [SPARK-22642][SQL] the createdTempDir will not be...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19841#discussion_r154490878 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -104,147 +105,153 @@ case class

[GitHub] spark pull request #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast beh...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19858#discussion_r154490770 --- Diff: docs/sql-programming-guide.md --- @@ -1776,6 +1776,8 @@ options. Note that, for DecimalType(38,0)*, the table above intentionally

[GitHub] spark pull request #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/cor...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19863#discussion_r154490741 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/OrcTest.scala --- @@ -15,18 +15,17 @@ * limitations under the

[GitHub] spark pull request #19805: [SPARK-22649][PYTHON][SQL] Adding localCheckpoint...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r154490687 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -537,9 +537,48 @@ class Dataset[T] private[sql]( */

[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19865 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19865 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84386/ Test FAILed. ---

[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19865 **[Test build #84386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84386/testReport)** for PR 19865 at commit

[GitHub] spark issue #19865: [SPARK-22668][SQL] Exclude global variables from argumen...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19865 **[Test build #84386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84386/testReport)** for PR 19865 at commit

[GitHub] spark pull request #19865: [SPARK-22668][SQL] Exclude global variables from ...

2017-12-01 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/19865 [SPARK-22668][SQL] Exclude global variables from arguments of method split by CodegenContext.splitExpressions() ## What changes were proposed in this pull request? This PR fixes to make

[GitHub] spark issue #19783: [SPARK-21322][SQL] support histogram in filter cardinali...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19783 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84385/ Test PASSed. ---

[GitHub] spark issue #19783: [SPARK-21322][SQL] support histogram in filter cardinali...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19783 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19783: [SPARK-21322][SQL] support histogram in filter cardinali...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19783 **[Test build #84385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84385/testReport)** for PR 19783 at commit

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84384/ Test PASSed. ---

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19864 **[Test build #84384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84384/testReport)** for PR 19864 at commit

[GitHub] spark issue #10803: [SPARK-12875] [ML] Add Weight of Evidence and Informatio...

2017-12-01 Thread hhbyyh
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/10803 No it's not merged. Feel free to use the code as you wish. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19845: [SPARK-22651][PYTHON][ML] Prevent initiating mult...

2017-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19845 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19845: [SPARK-22651][PYTHON][ML] Prevent initiating multiple Hi...

2017-12-01 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19845 Merged to master. Thanks for reviewing this @viirya, @jiangxb1987, @dongjoon-hyun and @imatiach-msft. --- - To

[GitHub] spark issue #19783: [SPARK-21322][SQL] support histogram in filter cardinali...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19783 **[Test build #84385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84385/testReport)** for PR 19783 at commit

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19864 **[Test build #84384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84384/testReport)** for PR 19864 at commit

[GitHub] spark issue #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/core`

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84381/ Test PASSed. ---

[GitHub] spark issue #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/core`

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19863 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/core`

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19863 **[Test build #84381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84381/testReport)** for PR 19863 at commit

[GitHub] spark issue #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast behavior c...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19858 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast behavior c...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19858 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84383/ Test PASSed. ---

[GitHub] spark issue #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast behavior c...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19858 **[Test build #84383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84383/testReport)** for PR 19858 at commit

[GitHub] spark pull request #19841: [SPARK-22642][SQL] the createdTempDir will not be...

2017-12-01 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/19841#discussion_r154480032 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -104,147 +105,153 @@ case class

[GitHub] spark issue #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast behavior c...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19858 **[Test build #84383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84383/testReport)** for PR 19858 at commit

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-12-01 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/19627 @jkbradley I think it is better to review #19857 (fix python model specific optimization) and merge it first and then I rebase & update this PR. :) ---

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84380/ Test PASSed. ---

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19848 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19848 **[Test build #84380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84380/testReport)** for PR 19848 at commit

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19864 **[Test build #84382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84382/testReport)** for PR 19864 at commit

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84382/ Test FAILed. ---

[GitHub] spark issue #18424: [SPARK-17091] Add rule to convert IN predicate to equiva...

2017-12-01 Thread a10y
Github user a10y commented on the issue: https://github.com/apache/spark/pull/18424 @ptkool are you still tracking this at all? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19864 **[Test build #84382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84382/testReport)** for PR 19864 at commit

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize on-di...

2017-12-01 Thread CodingCat
Github user CodingCat commented on the issue: https://github.com/apache/spark/pull/19864 @cloud-fan @viirya @gatorsmile @felixcheung @hvanhovell @HyukjinKwon @dongjoon-hyun @liancheng --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19864: [SPARK-22673][SQL] InMemoryRelation should utiliz...

2017-12-01 Thread CodingCat
GitHub user CodingCat opened a pull request: https://github.com/apache/spark/pull/19864 [SPARK-22673][SQL] InMemoryRelation should utilize on-disk table stats whenever possible ## What changes were proposed in this pull request? The current implementation of

[GitHub] spark issue #19746: [SPARK-22346][ML] VectorSizeHint Transformer for using V...

2017-12-01 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19746 @WeichenXu123 From what I've seen, it's more common for people to use VectorAssembler to assemble a bunch of Numeric columns, rather than a bunch of Vector columns. I'd recommend we do things

[GitHub] spark issue #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/core`

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19863 **[Test build #84381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84381/testReport)** for PR 19863 at commit

[GitHub] spark pull request #19863: [SPARK-22672][TEST][SQL] Move OrcTest to `sql/cor...

2017-12-01 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/19863 [SPARK-22672][TEST][SQL] Move OrcTest to `sql/core` ## What changes were proposed in this pull request? To support ORC tests without Hive, we had better have `OrcTest` in `sql/core`

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

2017-12-01 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19527 Question about this PR description comment: > Note that keep can't be used at the same time with dropLast as true. Because they will conflict in encoded vector by producing a vector of zeros.

[GitHub] spark pull request #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast beh...

2017-12-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19858#discussion_r154453799 --- Diff: docs/sql-programming-guide.md --- @@ -1776,6 +1776,8 @@ options. Note that, for DecimalType(38,0)*, the table above intentionally

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-12-01 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r154452715 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala --- @@ -41,8 +41,12 @@ import org.apache.spark.sql.types.{DoubleType,

[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate queue for Streami...

2017-12-01 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/19838 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...

2017-12-01 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19838 Thanks! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84378/ Test PASSed. ---

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84378/testReport)** for PR 19651 at commit

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19848 **[Test build #84380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84380/testReport)** for PR 19848 at commit

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

2017-12-01 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/19045 So it seems like the YARN changes are only going to happen in Hadoop 3+ so this might make sense regardless of what happens in https://github.com/apache/spark/pull/19267 (since folks like K8 or

[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...

2017-12-01 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/19838 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19848 LGTM but I'll leave it here a bit for others to take a look. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19858: [SPARK-22489][DOC][FOLLOWUP] Update broadcast behavior c...

2017-12-01 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/19858 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2017-12-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19840 Instead of setting `PYSPARK_PYTHON=~/anaconda3/envs/py3/bin/python`, what happens if you set `PYSPARK_DRIVER_PYTHON=~/anaconda3/envs/py3/bin/python`? ---

[GitHub] spark issue #19831: [SPARK-22626][SQL] Wrong Hive table statistics may trigg...

2017-12-01 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/19831 Instead of manually setting up table statistics, I'm just trying to simulate the statistics for these tables by this way. If `totalSize (or rawDataSize) > 0` and `rowCount = 0`, at least one

[GitHub] spark issue #18113: [SPARK-20890][SQL] Added min and max typed aggregation f...

2017-12-01 Thread setjet
Github user setjet commented on the issue: https://github.com/apache/spark/pull/18113 @cloud-fan done, could you please have a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor R...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84379/ Test PASSed. ---

[GitHub] spark issue #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor R...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19758 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor R...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19758 **[Test build #84379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84379/testReport)** for PR 19758 at commit

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-12-01 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19627 Is this still WIP or ready? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-12-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19631 +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new O...

2017-12-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19651#discussion_r154408356 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to

[GitHub] spark pull request #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new O...

2017-12-01 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19651#discussion_r154407217 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to the

[GitHub] spark issue #19758: [SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor R...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19758 **[Test build #84379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84379/testReport)** for PR 19758 at commit

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-12-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19651 Thank you so much, @cloud-fan . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84378/testReport)** for PR 19651 at commit

[GitHub] spark pull request #19848: [SPARK-22162] Executors and the driver should use...

2017-12-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19848#discussion_r154403383 --- Diff: core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala --- @@ -70,7 +70,8 @@ object SparkHadoopMapRedUtil extends Logging {

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154403001 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,12 @@ object MimaExcludes { // Exclude rules for 2.3.x lazy val v23excludes =

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-01 Thread daroo
Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 @HyukjinKwon @zsxwing Any chance to go ahead with this? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19861 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19861 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84377/ Test PASSed. ---

[GitHub] spark issue #19862: [SPARK-22671][SQL] Make SortMergeJoin read less data whe...

2017-12-01 Thread gczsjdy
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/19862 cc @cloud-fan @viirya @ConeyLiu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19861 **[Test build #84377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84377/testReport)** for PR 19861 at commit

[GitHub] spark issue #19862: Make SortMergeJoin read less data when wholeStageCodegen...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19862 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19862: Make SortMergeJoin read less data when wholeStage...

2017-12-01 Thread gczsjdy
GitHub user gczsjdy opened a pull request: https://github.com/apache/spark/pull/19862 Make SortMergeJoin read less data when wholeStageCodegen is off ## What changes were proposed in this pull request? In SortMergeJoin(with wholeStageCodegen), an optimization already

[GitHub] spark issue #19860: [SPARK-22669][SQL] Avoid unnecessary function calls in c...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19860 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19860: [SPARK-22669][SQL] Avoid unnecessary function calls in c...

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84376/ Test PASSed. ---

[GitHub] spark issue #19860: [SPARK-22669][SQL] Avoid unnecessary function calls in c...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19860 **[Test build #84376 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84376/testReport)** for PR 19860 at commit

[GitHub] spark issue #19859: [SPARK-22634][BUILD] Update Bouncy Castle to 1.58

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19859 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19859: [SPARK-22634][BUILD] Update Bouncy Castle to 1.58

2017-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84375/ Test PASSed. ---

[GitHub] spark issue #19859: [SPARK-22634][BUILD] Update Bouncy Castle to 1.58

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19859 **[Test build #84375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84375/testReport)** for PR 19859 at commit

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-12-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19631 > If I understand what you're saying correctly, that should be considered a security issue in that server application regardless of this change. The server should not be exposing its environment

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154378257 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,12 @@ object MimaExcludes { // Exclude rules for 2.3.x lazy val v23excludes =

[GitHub] spark issue #19831: [SPARK-22626][SQL] Wrong Hive table statistics may trigg...

2017-12-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19831 Is it really an issue? If you manually set a wrong statistics, how would you expect the system to do? I think data source tables don't allow you set the statistics manually, so this problem is

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-12-01 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r154373635 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -542,7 +496,54 @@ private[spark] class SecurityManager( * Gets the

[GitHub] spark issue #19855: [SPARK-22662] [SQL] Failed to prune columns after rewrit...

2017-12-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19855 cc @jiangxb1987 who has more context. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2017-12-01 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18906 For https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala#L42, I don't think it's a public API. So, Scala

[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19861 **[Test build #84377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84377/testReport)** for PR 19861 at commit

[GitHub] spark pull request #19861: [SPARK-22387][SQL] Propagate session configs to d...

2017-12-01 Thread jiangxb1987
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/19861 [SPARK-22387][SQL] Propagate session configs to data source read/write options ## What changes were proposed in this pull request? Introduce a new interface `ConfigSupport` for

[GitHub] spark issue #19850: [SPARK-22653][CORE] executorAddress registered in Coarse...

2017-12-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19850 thanks @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2017-12-01 Thread ptkool
Github user ptkool commented on the issue: https://github.com/apache/spark/pull/18906 @HyukjinKwon As requested, here are the related Scala API changes:

[GitHub] spark issue #19860: [SPARK-22669][SQL] Avoid unnecessary function calls in c...

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19860 **[Test build #84376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84376/testReport)** for PR 19860 at commit

[GitHub] spark pull request #19860: [SPARK-22669][SQL] Avoid unnecessary function cal...

2017-12-01 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/19860 [SPARK-22669][SQL] Avoid unnecessary function calls in code generation ## What changes were proposed in this pull request? In many parts of the codebase for code generation, we are

[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...

2017-12-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18113#discussion_r154341528 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala --- @@ -99,3 +94,91 @@ class TypedAverage[IN](val f:

[GitHub] spark issue #19859: [SPARK-22634][BUILD] Update Bouncy Castle to 1.58

2017-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19859 **[Test build #84375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84375/testReport)** for PR 19859 at commit

[GitHub] spark pull request #19859: [SPARK-22634][BUILD] Update Bouncy Castle to 1.58

2017-12-01 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/19859 [SPARK-22634][BUILD] Update Bouncy Castle to 1.58 ## What changes were proposed in this pull request? Update Bouncy Castle to 1.58, and jets3t to 0.9.4 to (sort of) match. ## How

[GitHub] spark pull request #19846: [SPARK-22393][SPARK-SHELL] spark-shell can't find...

2017-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19846 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

  1   2   >