[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89267570 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala --- @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69053/ Test PASSed. ---

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15986 I managed to come up with a standalone end-to-end reproduction of the shuffle file leak, allowing me to validate this patch's fix. Run ``` ./bin/spark-shell

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15986 **[Test build #69053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69053/consoleFull)** for PR 15986 at commit

[GitHub] spark pull request #15977: [SPARK-18436][SQL] isin causing SQL syntax error ...

2016-11-22 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/15977#discussion_r89266916 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -114,10 +114,14 @@ object

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69051/ Test FAILed. ---

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15959 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69051/consoleFull)** for PR 15959 at commit

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15959 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89265835 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -310,6 +270,110 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark issue #15988: [SPARK-18519][SQL][BRANCH-2.0] map type can not be used ...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15988 **[Test build #69056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69056/consoleFull)** for PR 15988 at commit

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69057/consoleFull)** for PR 15959 at commit

[GitHub] spark issue #15988: [SPARK-18519][SQL][BRANCH-2.0] map type can not be used ...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15988 cc @hvanhovell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #15988: [SPARK-18519][SQL][BRANCH-2.0] map type can not b...

2016-11-22 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/15988 [SPARK-18519][SQL][BRANCH-2.0] map type can not be used in EqualTo ## What changes were proposed in this pull request? Technically map type is not orderable, but can be used in equality

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89264465 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -310,6 +270,110 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89263076 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89242805 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89249380 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89252556 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89262935 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -274,7 +274,7 @@ class DDLSuite extends QueryTest with

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89249078 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark pull request #15951: [SPARK-18510] Fix data corruption from inferred p...

2016-11-22 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/15951#discussion_r89248592 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -84,30 +88,106 @@ case class DataSource(

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15986 cc @tejasapatil --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #15694: [SPARK-18179][SQL] Throws analysis exception with...

2016-11-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15694 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15694: [SPARK-18179][SQL] Throws analysis exception with a prop...

2016-11-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15694 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15987 Why does it cast it to double? The fix looks pretty weird. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15985 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69048/ Test PASSed. ---

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15985 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15985 **[Test build #69048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69048/consoleFull)** for PR 15985 at commit

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69055/consoleFull)** for PR 15959 at commit

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15877 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69047/ Test PASSed. ---

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15877 **[Test build #69047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69047/consoleFull)** for PR 15877 at commit

[GitHub] spark issue #15956: [SPARK-18519][SQL] map type can not be used in EqualTo

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15956 Yea, I'll backport --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15983: [SPARK-18544] [SQL] Append with df.saveAsTable writes da...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15983 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69045/ Test PASSed. ---

[GitHub] spark issue #15983: [SPARK-18544] [SQL] Append with df.saveAsTable writes da...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15983 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15983: [SPARK-18544] [SQL] Append with df.saveAsTable writes da...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15983 **[Test build #69045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69045/consoleFull)** for PR 15983 at commit

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69054/consoleFull)** for PR 15959 at commit

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15877 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69049/ Test FAILed. ---

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15877 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15959 LGTM except one minor comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15877 **[Test build #69049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69049/consoleFull)** for PR 15877 at commit

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258891 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,176 @@ case class Statistics(

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69044/ Test PASSed. ---

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15480 **[Test build #69044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69044/consoleFull)** for PR 15480 at commit

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15987 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69050/ Test FAILed. ---

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15987 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15987 **[Test build #69050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69050/consoleFull)** for PR 15987 at commit

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15986 **[Test build #69053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69053/consoleFull)** for PR 15986 at commit

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69052/consoleFull)** for PR 15959 at commit

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258401 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics( }

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258367 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics( }

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258326 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics(

[GitHub] spark issue #15959: [SPARK-18522][SQL] Explicit contract for column stats se...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15959 **[Test build #69051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69051/consoleFull)** for PR 15959 at commit

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258177 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics( }

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258106 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics( }

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258088 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -319,7 +319,8 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89258070 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics(

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89257869 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics(

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89257822 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -339,7 +341,7 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89257721 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics(

[GitHub] spark pull request #15959: [SPARK-18522][SQL] Explicit contract for column s...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15959#discussion_r89257647 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -58,60 +61,170 @@ case class Statistics(

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69046/ Test FAILed. ---

[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15986 **[Test build #69046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69046/consoleFull)** for PR 15986 at commit

[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming

2016-11-22 Thread ChorPangChan
Github user ChorPangChan commented on the issue: https://github.com/apache/spark/pull/15974 I stay adding a new package to streaming is a better structure then modify the spark-core. can we make the decision for which implementation to use first --- If your project is set up for

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15985 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15985 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69043/ Test FAILed. ---

[GitHub] spark pull request #15976: [SPARK-18403][SQL] Fix unsafe data false sharing ...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15976#discussion_r89257213 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala --- @@ -262,7 +262,9 @@ class

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15985 **[Test build #69043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69043/consoleFull)** for PR 15985 at commit

[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming

2016-11-22 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15974 I think the base solutions are same, expect some other information which I am working to add. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89256962 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -274,4 +274,30 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming

2016-11-22 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15974 I think there is no need to open another reduplicate PR. Do your mind closing this PR, and let work on #15904 ? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89256608 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -339,7 +341,7 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #15904: [SPARK-18470][STREAMING][WIP] Provide Spark Streaming Mo...

2016-11-22 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15904 @ajbozarth Thank you for reminding me, i will take a look at it later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89256275 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -89,9 +89,11 @@ private[spark] class TaskSchedulerImpl( val

[GitHub] spark pull request #15983: [SPARK-18544] [SQL] Append with df.saveAsTable wr...

2016-11-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15983#discussion_r89256222 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -373,8 +373,19 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89256137 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -339,7 +341,7 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89255724 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -339,7 +341,7 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming

2016-11-22 Thread ChorPangChan
Github user ChorPangChan commented on the issue: https://github.com/apache/spark/pull/15974 It require manual merge if I do it against master. should I just PR with conflicts or rebase to master before PR? btw those conflicts are just versions from the pom file. --- If your

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15987 Hi, @hvanhovell . Could you review this PR? This is the first attempt to use `UnresolvedAttribute` and Analyzer rule. There are two debatable issues. - Catalyst Analyzer

[GitHub] spark issue #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails for no...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15987 **[Test build #69050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69050/consoleFull)** for PR 15987 at commit

[GitHub] spark pull request #15987: [SPARK-18515][SQL] AlterTableDropPartitions fails...

2016-11-22 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/15987 [SPARK-18515][SQL] AlterTableDropPartitions fails for non-string columns ## What changes were proposed in this pull request? While

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69040/ Test PASSed. ---

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15982 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15982 **[Test build #69040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69040/consoleFull)** for PR 15982 at commit

[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15971 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15971 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69042/ Test FAILed. ---

[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15971 **[Test build #69042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69042/consoleFull)** for PR 15971 at commit

[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming

2016-11-22 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/15974 This will never be merged into 1.6, so you'll have to send a new PR against master. Please close this one. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15984: [SPARK-18551] [Web UI] [Core] [WIP] Add functionality to...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15984 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15984: [SPARK-18551] [Web UI] [Core] [WIP] Add functionality to...

2016-11-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15984 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69041/ Test PASSed. ---

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89253541 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -525,7 +525,12 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #15984: [SPARK-18551] [Web UI] [Core] [WIP] Add functionality to...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15984 **[Test build #69041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69041/consoleFull)** for PR 15984 at commit

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15877 **[Test build #69049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69049/consoleFull)** for PR 15877 at commit

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-22 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/15877 cc @rxin @hvanhovell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15985: [SPARK-18545] [SQL] Verify number of hive client RPCs in...

2016-11-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15985 **[Test build #69048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69048/consoleFull)** for PR 15985 at commit

[GitHub] spark pull request #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskS...

2016-11-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/15986#discussion_r89252852 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -525,7 +525,12 @@ private[spark] class TaskSchedulerImpl(

  1   2   3   4   5   6   7   >