[GitHub] spark pull request #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16329#discussion_r93118975 --- Diff: examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java --- @@ -0,0 +1,154 @@ +/* + * Licensed to the A

[GitHub] spark pull request #15496: [SPARK-17950] [Python] Match SparseVector behavio...

2016-12-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15496#discussion_r93112496 --- Diff: python/pyspark/ml/linalg/__init__.py --- @@ -705,6 +705,23 @@ def __eq__(self, other): return Vectors._equals(self.indices, self.va

[GitHub] spark pull request #15496: [SPARK-17950] [Python] Match SparseVector behavio...

2016-12-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15496#discussion_r93116390 --- Diff: python/pyspark/ml/linalg/__init__.py --- @@ -705,6 +705,23 @@ def __eq__(self, other): return Vectors._equals(self.indices, self.va

[GitHub] spark pull request #15496: [SPARK-17950] [Python] Match SparseVector behavio...

2016-12-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15496#discussion_r93116698 --- Diff: python/pyspark/ml/linalg/__init__.py --- @@ -705,6 +705,23 @@ def __eq__(self, other): return Vectors._equals(self.indices, self.va

[GitHub] spark pull request #15496: [SPARK-17950] [Python] Match SparseVector behavio...

2016-12-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15496#discussion_r93113956 --- Diff: python/pyspark/ml/linalg/__init__.py --- @@ -705,6 +705,23 @@ def __eq__(self, other): return Vectors._equals(self.indices, self.va

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-19 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r93117073 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -206,6 +201,36 @@ class StreamExecution( st

[GitHub] spark pull request #10605: [SPARK-12295] [SQL] external spilling for window ...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/10605#discussion_r93116743 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -19,6 +19,8 @@ import java.util.Comp

[GitHub] spark issue #16240: [SPARK-16792][SQL] Dataset containing a Case Class with ...

2016-12-19 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/16240 /cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16320: [SPARK-18877][SQL] `CSVInferSchema.inferField` on Decima...

2016-12-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16320 Hi, @rxin and @falaki . If there are some committers to review this part, I think you are the best persons. Could you give some opinion about this when you have some time? --- If your

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-19 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r93114298 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -206,6 +201,36 @@ class StreamExecution( st

[GitHub] spark issue #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL progra...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70374/ Test PASSed. ---

[GitHub] spark issue #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL progra...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16329 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL progra...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16329 **[Test build #70374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70374/testReport)** for PR 16329 at commit [`d059d5d`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #15018: [SPARK-17455][MLlib] Improve PAVA implementation in Isot...

2016-12-19 Thread neggert
Github user neggert commented on the issue: https://github.com/apache/spark/pull/15018 @srowen Addressed your comments and fixed some style issues. Some updated timings: Alternating decreasing input val x = (1 to length).toArray.map(_.toDouble) val

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r93113059 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -206,6 +201,36 @@ class StreamExecution( s

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16291 I'm not sure whether that change would be a problem, but I wonder if a system where listeners can "opt in" to this behavior wouldn't be better. e.g., `EventLoggingListener` can slow down the bus, hav

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70377/testReport)** for PR 16322 at commit [`fb74f62`](https://github.com/apache/spark/commit/fb

[GitHub] spark pull request #16057: [SPARK-18624][SQL] Implicit cast ArrayType(Intern...

2016-12-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16057 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16322 cc @tdas @brkyvz @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wis

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-19 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r9324 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala --- @@ -240,7 +241,7 @@ trait StreamTest extends QueryTest with SharedSQLC

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/16291 > You are right, that's why the executor service is single threaded which guarantees ordered processing of the events per listener. But this is still substantially different from the ord

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

2016-12-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16057 LGTM - merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70376/testReport)** for PR 16322 at commit [`8fbb387`](https://github.com/apache/spark/commit/8f

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70369/ Test PASSed. ---

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16321 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16321 **[Test build #70369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70369/testReport)** for PR 16321 at commit [`471740b`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #16326: [SPARK-18915] [SQL] Automatic Table Repair when Creating...

2016-12-19 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/16326 Oh I see, you're saying if there are old files for the partition, the INSERT INTO will cause those to become visible. That is a little confusing. --- If your project is set up for it, you can reply t

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70375/testReport)** for PR 16322 at commit [`185f177`](https://github.com/apache/spark/commit/18

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16337 Can we create some higher level logical grouping and put these queries into multiple files? It's not as useful to dump 200 queries in one file. --- If your project is set up for it, you can reply to

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST][DO-NOT-MERGE] StateStoreSuite....

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16314 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST][DO-NOT-MERGE] StateStoreSuite....

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16314 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70367/ Test PASSed. ---

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST][DO-NOT-MERGE] StateStoreSuite....

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16314 **[Test build #70367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70367/testReport)** for PR 16314 at commit [`abff69c`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL progra...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16329 **[Test build #70374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70374/testReport)** for PR 16329 at commit [`d059d5d`](https://github.com/apache/spark/commit/d0

[GitHub] spark pull request #16332: [SPARK-18921][SQL] check database existence with ...

2016-12-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16332 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...

2016-12-19 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/16332 LGTM. Merging to master and branch 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add StripedLock for each table's rela...

2016-12-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16135 Merging to master/2.1. @xuanyuanking can you open a backport for 2.0, if we also need to merge this to that branche? --- If your project is set up for it, you can reply to this email and have yo

[GitHub] spark pull request #16135: [SPARK-18700][SQL] Add StripedLock for each table...

2016-12-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16288: [SPARK-18869][SQL] Add TreeNode.p that returns BaseType

2016-12-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16288 @srowen it looks like 2.0 has a different plan from master. I pushed a fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70366/ Test FAILed. ---

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70366/testReport)** for PR 16296 at commit [`8dafb9d`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add StripedLock for each table's rela...

2016-12-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16135 Go for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the f

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16337 Could we split the test suites based on the subquery types? - [With aliases](https://technet.microsoft.com/en-us/library/ms190410(v=sql.105).aspx). - With IN or NOT IN - [Sub

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r89886595 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -232,6 +225,13 @@ private[spark] class Executor( }

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16053 **[Test build #70373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70373/testReport)** for PR 16053 at commit [`ecd0731`](https://github.com/apache/spark/commit/ec

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r89889108 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -247,28 +248,29 @@ class CoarseGrainedSche

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r89872502 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -17,27 +17,179 @@ package org.apache.spark.scheduler

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r89872712 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -17,27 +17,179 @@ package org.apache.spark.scheduler

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r89895008 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -993,6 +993,12 @@ class DAGScheduler( JavaUtils.buffe

[GitHub] spark pull request #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN s...

2016-12-19 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16337#discussion_r93096132 --- Diff: sql/core/src/test/resources/sql-tests/inputs/subq01-in-subquery.sql --- @@ -0,0 +1,452 @@ +-- A test suite for IN predicate subquery ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70372/testReport)** for PR 16322 at commit [`ccb2a51`](https://github.com/apache/spark/commit/cc

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Consumes the returned...

2016-12-19 Thread dusenberrymw
Github user dusenberrymw commented on a diff in the pull request: https://github.com/apache/spark/pull/16263#discussion_r93093157 --- Diff: python/pyspark/rdd.py --- @@ -135,12 +135,12 @@ def _load_from_socket(port, serializer): break if not sock:

[GitHub] spark issue #16338: [SPARK-18837][WEBUI] Very long stage descriptions do not...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16338 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/16291 This change is ready for review @rxin, @zsxwing, @kayousterhout --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] spark issue #16338: [SPARK-18837][WEBUI] Very long stage descriptions do not...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16338 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70365/ Test FAILed. ---

[GitHub] spark issue #16338: [SPARK-18837][WEBUI] Very long stage descriptions do not...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16338 **[Test build #70365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70365/testReport)** for PR 16338 at commit [`c86dc72`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16282 **[Test build #70370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70370/testReport)** for PR 16282 at commit [`fd63513`](https://github.com/apache/spark/commit/fd

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16189 **[Test build #70371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70371/testReport)** for PR 16189 at commit [`9cff80f`](https://github.com/apache/spark/commit/9c

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16321 **[Test build #70369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70369/testReport)** for PR 16321 at commit [`471740b`](https://github.com/apache/spark/commit/47

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

2016-12-19 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16321 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

2016-12-19 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16282 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16189 **[Test build #70368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70368/testReport)** for PR 16189 at commit [`d87c8f6`](https://github.com/apache/spark/commit/d8

[GitHub] spark pull request #16053: [SPARK-17931] Eliminate unncessary task (de) seri...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16053#discussion_r93091673 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -17,27 +17,119 @@ package org.apache.spark.scheduler

[GitHub] spark issue #16339: [SPARK-18917][SQL] Add Skip Partition Check Flag to avoi...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16339 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #16339: [SPARK-18917][SQL] Add Skip Partition Check Flag to avoi...

2016-12-19 Thread alunarbeach
Github user alunarbeach commented on the issue: https://github.com/apache/spark/pull/16339 @dongjoon-hyun Please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16339: [SPARK-18917][SQL] Add Skip Partition Check Flag ...

2016-12-19 Thread alunarbeach
GitHub user alunarbeach opened a pull request: https://github.com/apache/spark/pull/16339 [SPARK-18917][SQL] Add Skip Partition Check Flag to avoid list all leaf files in append mode ## What changes were proposed in this pull request? Currently saving a dataframe in append mode

[GitHub] spark pull request #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN s...

2016-12-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16337#discussion_r93090592 --- Diff: sql/core/src/test/resources/sql-tests/inputs/subq01-in-subquery.sql --- @@ -0,0 +1,452 @@ +-- A test suite for IN predicate subquery

[GitHub] spark issue #16295: [SPARK-18356] [ML] KMeans should cache RDD before traini...

2016-12-19 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16295 This looks fine, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16053 Also I need to fix up the Mesos functionality here. I deliberately posted a broken Mesos version to verify my hypothesis that there are no end-to-end Mesos tests (this code would have failed

[GitHub] spark issue #16303: [SPARK-17807][core] Demote scalatest to "provided" in sp...

2016-12-19 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16303 Closing in favor of the other PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request #16303: [SPARK-17807][core] Demote scalatest to "provided...

2016-12-19 Thread vanzin
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/16303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15505: [SPARK-17931][CORE] taskScheduler has some unneeded seri...

2016-12-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/15505 @witgo @kayousterhout where do we stand on this and https://github.com/apache/spark/pull/16053? Both still viable alternatives? https://github.com/apache/spark/pull/16053 is still missing pe

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16053 Thanks for the review @squito. I got sidetracked from this at the end of last week and forgot to post the results of some benchmarks @shivaram and I did on a 20-machine m2.4xlarge EC2 machine

[GitHub] spark pull request #16189: [SPARK-18761][CORE] Introduce "task reaper" to ov...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16189#discussion_r93087921 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -432,6 +465,93 @@ private[spark] class Executor( } /**

[GitHub] spark pull request #16189: [SPARK-18761][CORE] Introduce "task reaper" to ov...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16189#discussion_r93087804 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -432,6 +465,93 @@ private[spark] class Executor( } /**

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST][DO-NOT-MERGE] StateStoreSuite....

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16314 **[Test build #70367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70367/testReport)** for PR 16314 at commit [`abff69c`](https://github.com/apache/spark/commit/ab

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST][DO-NOT-MERGE] StateStoreSuite....

2016-12-19 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/16314 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70366/testReport)** for PR 16296 at commit [`8dafb9d`](https://github.com/apache/spark/commit/8d

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16057 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70361/ Test PASSed. ---

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16057 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast ArrayType(InternalType)

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16057 **[Test build #70361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70361/testReport)** for PR 16057 at commit [`23089fa`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16336 Ur, that's true. The effective benefit might be tiny. I'll keep a few days and close this if there is no response from the others. Thank you for review. --- If your project is set up for

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16291 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70360/ Test PASSed. ---

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16291 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16291 **[Test build #70360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70360/testReport)** for PR 16291 at commit [`66e4f12`](https://github.com/apache/spark/commit/6

[GitHub] spark pull request #16053: [SPARK-17931] Eliminate unncessary task (de) seri...

2016-12-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16053#discussion_r93080712 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -17,27 +17,119 @@ package org.apache.spark.scheduler

[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70359/ Test PASSed. ---

[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16308 **[Test build #70359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70359/testReport)** for PR 16308 at commit [`6746265`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16337 In Spark, we can use the following way to send the whole query to the underlying data source. ```SQL CREATE TEMPORARY TABLE jdbcTable USING jdbc OPTIONS ( url "jdbc:postgre

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70358/ Test PASSed. ---

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r93079839 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -17,27 +17,179 @@ package org.apache.spark.scheduler

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70358/testReport)** for PR 16335 at commit [`54f0b77`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16337 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16337 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70362/ Test FAILed. ---

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16337 **[Test build #70362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70362/testReport)** for PR 16337 at commit [`c2ca009`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r93078511 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -139,29 +139,6 @@ class TaskSchedulerImplSuite extends SparkFunSu

[GitHub] spark pull request #15505: [SPARK-17931][CORE] taskScheduler has some unneed...

2016-12-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r93077759 --- Diff: core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala --- @@ -59,6 +62,12 @@ private[spark] class LocalEndpoint(

[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...

2016-12-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16332 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

<    1   2   3   4   5   >