[GitHub] spark issue #16342: [SPARK-18927][SS] MemorySink for StructuredStreaming can...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16342 **[Test build #70382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70382/testReport)** for PR 16342 at commit [`258da6f`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70381/ Test PASSed. ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70381/testReport)** for PR 16322 at commit [`299a13d`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Remove timeout for reading d...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16263 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Remove timeout for reading d...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16263 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70389/ Test PASSed. ---

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Remove timeout for reading d...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16263 **[Test build #70389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70389/testReport)** for PR 16263 at commit [`e02695e`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #16232: [SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correct...

2016-12-19 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/16232 @viirya That's not correct, the values does not have entry in the array. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70391/testReport)** for PR 16343 at commit [`04fa2f7`](https://github.com/apache/spark/commit/04

[GitHub] spark pull request #16340: [SPARK-18928] Check TaskContext.isInterrupted() i...

2016-12-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16340 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16340 Merging to master/2.1/2.0. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2016-12-19 Thread merlintang
Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile one more customer is running into this issue in the spark 1.6.x. I backport the code #16134 to here and test it manually. Please verify. --- If your project is set up for it, you ca

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16340 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16340 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70378/ Test PASSed. ---

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16340 **[Test build #70378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70378/testReport)** for PR 16340 at commit [`236efe5`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16232: [SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correct...

2016-12-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16232 @davies Yes, that is an extreme case. I use it to explain how we can cause the exception. Even we don't consider the extreme case if we don't pass the kind of map in, the original assert `ma

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16345: [SPARK-17755][Core]Use workerRef to send RegisterWorkerR...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16345 **[Test build #70390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70390/testReport)** for PR 16345 at commit [`b4b5552`](https://github.com/apache/spark/commit/b4

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16341 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70380/ Test FAILed. ---

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16341 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16341 **[Test build #70380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70380/testReport)** for PR 16341 at commit [`155f9ab`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #16345: [SPARK-17755][Core]Use workerRef to send RegisterWorkerR...

2016-12-19 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16345 cc @vanzin want to review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #16345: [SPARK-17755][Core]Use workerRef to send Register...

2016-12-19 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16345 [SPARK-17755][Core]Use workerRef to send RegisterWorkerResponse to avoid the race condition ## What changes were proposed in this pull request? The root cause of this issue is that Registe

[GitHub] spark pull request #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread actuaryzhang
GitHub user actuaryzhang opened a pull request: https://github.com/apache/spark/pull/16344 [SPARK-18929][ML] Add Tweedie distribution in GLM ## What changes were proposed in this pull request? I propose to add the full Tweedie family into the GeneralizedLinearRegression model. T

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Remove timeout for reading d...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16263 **[Test build #70389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70389/testReport)** for PR 16263 at commit [`e02695e`](https://github.com/apache/spark/commit/e0

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12775 **[Test build #70388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70388/testReport)** for PR 12775 at commit [`699730b`](https://github.com/apache/spark/commit/69

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-19 Thread kevinyu98
Github user kevinyu98 commented on the issue: https://github.com/apache/spark/pull/16337 We will separate this file to multiple files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/12775 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark issue #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsumer when ...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16282 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70386/ Test PASSed. ---

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Consumes the returned...

2016-12-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16263#discussion_r93143569 --- Diff: python/pyspark/sql/tests.py --- @@ -558,6 +558,18 @@ def test_create_dataframe_from_objects(self): self.assertEqual(df.dtypes, [("key",

[GitHub] spark issue #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsumer when ...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16282 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsumer when ...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16282 **[Test build #70386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70386/testReport)** for PR 16282 at commit [`410c223`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Consumes the returned...

2016-12-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16263#discussion_r93143410 --- Diff: python/pyspark/rdd.py --- @@ -135,12 +135,12 @@ def _load_from_socket(port, serializer): break if not sock: raise

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16340 LGTM - pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wish

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70387/testReport)** for PR 16343 at commit [`92144e4`](https://github.com/apache/spark/commit/92

[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.

2016-12-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16308 This is getting pretty close. Can you update the description to define more formally semantics for timezone? In addition, I think we also need to have a follow-up pr to focus on data sources (

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2016-12-19 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/12775 Yeah, that's ok, @kayousterhout --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70385/testReport)** for PR 16343 at commit [`dc6d8ed`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70385/ Test FAILed. ---

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/12775 LGTM -- are you ok with merging this @markhamstra, after re-testing it since it's been so long? Sorry for neglecting this @lirui-intel! --- If your project is set up for it, you can

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/12775 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70385/testReport)** for PR 16343 at commit [`dc6d8ed`](https://github.com/apache/spark/commit/dc

[GitHub] spark issue #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsumer when ...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16282 **[Test build #70386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70386/testReport)** for PR 16282 at commit [`410c223`](https://github.com/apache/spark/commit/41

[GitHub] spark issue #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsumer when ...

2016-12-19 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16282 cc @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the f

[GitHub] spark pull request #16261: [SPARK-18836] [CORE] Serialize one copy of task m...

2016-12-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16261 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.sock...

2016-12-19 Thread brkyvz
GitHub user brkyvz opened a pull request: https://github.com/apache/spark/pull/16343 [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket input stream ## What changes were proposed in this pull request? https://spark-tests.appspot.com/test-details?suite_name=org.apache.spark

[GitHub] spark issue #16261: [SPARK-18836] [CORE] Serialize one copy of task metrics ...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16261 Merged this to master. Thanks @shivaram! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70375/ Test PASSed. ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70375/testReport)** for PR 16322 at commit [`185f177`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #12436: [SPARK-14649][CORE] DagScheduler should not run duplicat...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/12436 @sitalkedia this has been inactive for a while and there were a few issues pointed out above that haven't yet been resolved. Do you have time to work on this? Otherwise, can you close the PR?

[GitHub] spark issue #16314: [SPARK-18900][FLAKY-TEST] StateStoreSuite.maintenance

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16314 **[Test build #70384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70384/testReport)** for PR 16314 at commit [`6775639`](https://github.com/apache/spark/commit/67

[GitHub] spark issue #16338: [SPARK-18837][WEBUI] Very long stage descriptions do not...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16338 **[Test build #70383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70383/testReport)** for PR 16338 at commit [`c86dc72`](https://github.com/apache/spark/commit/c8

[GitHub] spark issue #16261: [SPARK-18836] [CORE] Serialize one copy of task metrics ...

2016-12-19 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16261 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16338: [SPARK-18837][WEBUI] Very long stage descriptions do not...

2016-12-19 Thread sarutak
Github user sarutak commented on the issue: https://github.com/apache/spark/pull/16338 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16342: [SPARK-18927][SS] MemorySink for StructuredStreaming can...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16342 **[Test build #70382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70382/testReport)** for PR 16342 at commit [`258da6f`](https://github.com/apache/spark/commit/25

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70377/ Test FAILed. ---

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16341 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16341 **[Test build #70379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70379/testReport)** for PR 16341 at commit [`6e0cc04`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16341 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70379/ Test FAILed. ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70376/ Test FAILed. ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70376 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70376/testReport)** for PR 16322 at commit [`8fbb387`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #16342: [SPARK-18927][SS] MemorySink for StructuredStream...

2016-12-19 Thread brkyvz
GitHub user brkyvz opened a pull request: https://github.com/apache/spark/pull/16342 [SPARK-18927][SS] MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in SessionConf ## What changes were proposed in this pull request? Checkpoint Loca

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16341 **[Test build #70380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70380/testReport)** for PR 16341 at commit [`155f9ab`](https://github.com/apache/spark/commit/15

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70381/testReport)** for PR 16322 at commit [`299a13d`](https://github.com/apache/spark/commit/29

[GitHub] spark issue #16341: [SQL] [WIP] Switch internal catalog types to use URI ins...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16341 **[Test build #70379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70379/testReport)** for PR 16341 at commit [`6e0cc04`](https://github.com/apache/spark/commit/6e

[GitHub] spark pull request #16341: [SQL] [WIP] Switch internal catalog types to use ...

2016-12-19 Thread ericl
GitHub user ericl opened a pull request: https://github.com/apache/spark/pull/16341 [SQL] [WIP] Switch internal catalog types to use URI instead of string for locationUri ## What changes were proposed in this pull request? This should help prevent accidental incorrect conve

[GitHub] spark issue #16232: [SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correct...

2016-12-19 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/16232 @viirya A map with multiple values for same key, is only used for hashed relation, not aggregation, will also not passed into UnsafeKVExternalSorter. So I think this is not actually a bug. --- If y

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70373/ Test PASSed. ---

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16053 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16053 **[Test build #70373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70373/testReport)** for PR 16053 at commit [`ecd0731`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Consumes the returned...

2016-12-19 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/16263#discussion_r93127142 --- Diff: python/pyspark/sql/tests.py --- @@ -558,6 +558,18 @@ def test_create_dataframe_from_objects(self): self.assertEqual(df.dtypes, [("key",

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Consumes the returned...

2016-12-19 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/16263#discussion_r93126957 --- Diff: python/pyspark/rdd.py --- @@ -135,12 +135,12 @@ def _load_from_socket(port, serializer): break if not sock: raise

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16340 For a bit of context on the `UnsafeSorter` changes, note that there are currently 5 implementations of UnsafeSorterIterator but three of those are just chaining / merging wrappers around the two i

[GitHub] spark pull request #16340: [SPARK-18928] Check TaskContext.isInterrupted() i...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16340#discussion_r93125739 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java --- @@ -94,6 +97,14 @@ public boolean hasNext() {

[GitHub] spark pull request #16340: [SPARK-18928] Check TaskContext.isInterrupted() i...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16340#discussion_r93125701 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java --- @@ -51,6 +53,7 @@ private byte[] arr = n

[GitHub] spark issue #16340: [SPARK-18928] Check TaskContext.isInterrupted() in FileS...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16340 **[Test build #70378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70378/testReport)** for PR 16340 at commit [`236efe5`](https://github.com/apache/spark/commit/23

[GitHub] spark pull request #16340: [SPARK-18928] Check TaskContext.isInterrupted() i...

2016-12-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16340#discussion_r93125645 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala --- @@ -301,6 +301,7 @@ private[jdbc] class JDBCRDD(

[GitHub] spark pull request #16340: [SPARK-18928] Check TaskContext.isInterrupted() i...

2016-12-19 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/16340 [SPARK-18928] Check TaskContext.isInterrupted() in FileScanRDD, JDBCRDD & UnsafeSorter ## What changes were proposed in this pull request? In order to respond to task cancellation, Spark

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16189 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70371/ Test PASSed. ---

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16189 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16189 **[Test build #70371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70371/testReport)** for PR 16189 at commit [`9cff80f`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16189 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70368/ Test PASSed. ---

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16189 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16189: [SPARK-18761][CORE] Introduce "task reaper" to oversee t...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16189 **[Test build #70368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70368/testReport)** for PR 16189 at commit [`d87c8f6`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16282 **[Test build #70370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70370/testReport)** for PR 16282 at commit [`fd63513`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16282 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16282 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70370/ Test PASSed. ---

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r93121938 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -206,6 +201,36 @@ class StreamExecution( s

[GitHub] spark issue #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL progra...

2016-12-19 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/16329 This is great! Thanks for taking the time to write up such complete examples. I think this was a big gap in the existing docs. One other ask. The screen-shot is great, but I'd like to see which

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70372/ Test PASSed. ---

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70372/testReport)** for PR 16322 at commit [`ccb2a51`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16329#discussion_r93119785 --- Diff: examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala --- @@ -0,0 +1,82 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16329#discussion_r93121147 --- Diff: examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala --- @@ -0,0 +1,82 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request #16329: [SPARK-16046][DOCS] Aggregations in the Spark SQL...

2016-12-19 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16329#discussion_r93118905 --- Diff: examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java --- @@ -0,0 +1,154 @@ +/* + * Licensed to the A

<    1   2   3   4   5   >