[GitHub] spark pull request #19130: [SPARK-21917][CORE][YARN] Supporting adding http(...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19130#discussion_r138692611 --- Diff: docs/running-on-yarn.md --- @@ -212,6 +212,14 @@ To use a custom metrics.properties for the application master and executors, upd

[GitHub] spark issue #19136: [SPARK-15689][SQL] data source v2 read path

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19136 **[Test build #81723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81723/testReport)** for PR 19136 at commit [`abcc606`](https://github.com/apache/spark/commit/a

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/19222 [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks to choose several types of memory block ## What changes were proposed in this pull request? This PR allows us to use one of several t

[GitHub] spark issue #19136: [SPARK-15689][SQL] data source v2 read path

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19136 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81723/ Test PASSed. ---

[GitHub] spark issue #19136: [SPARK-15689][SQL] data source v2 read path

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19136 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #81731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81731/testReport)** for PR 19222 at commit [`b7ffa10`](https://github.com/apache/spark/commit/b7

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19204 **[Test build #81732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81732/testReport)** for PR 19204 at commit [`e684dbb`](https://github.com/apache/spark/commit/e6

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19204 **[Test build #81732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81732/testReport)** for PR 19204 at commit [`e684dbb`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81732/ Test PASSed. ---

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19204 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #16121: [SPARK-16589][PYTHON] Chained cartesian produces incorre...

2017-09-13 Thread stuarteberg
Github user stuarteberg commented on the issue: https://github.com/apache/spark/pull/16121 This PR seems to have introduced a bug, which I have reported here: https://issues.apache.org/jira/browse/SPARK-21985 Any thoughts, @aray? Can the check in question simply be removed

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/19215 cc @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19136: [SPARK-15689][SQL] data source v2 read path

2017-09-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/19136#discussion_r138709319 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/Statistics.java --- @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19215 **[Test build #81733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81733/testReport)** for PR 19215 at commit [`be83f80`](https://github.com/apache/spark/commit/be

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19215 **[Test build #81733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81733/testReport)** for PR 19215 at commit [`be83f80`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19215 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19215 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81733/ Test FAILed. ---

[GitHub] spark pull request #19186: [SPARK-21972][ML] Add param handlePersistence

2017-09-13 Thread smurching
Github user smurching commented on a diff in the pull request: https://github.com/apache/spark/pull/19186#discussion_r138712707 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -483,24 +488,24 @@ class LogisticRegression @Since("1.2

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19221 **[Test build #81730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81730/testReport)** for PR 19221 at commit [`0b8d47a`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19221 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19221 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81730/ Test FAILed. ---

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19181 **[Test build #81734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81734/testReport)** for PR 19181 at commit [`2fd8e91`](https://github.com/apache/spark/commit/2f

[GitHub] spark pull request #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json su...

2017-09-13 Thread goldmedal
GitHub user goldmedal opened a pull request: https://github.com/apache/spark/pull/19223 [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR ## What changes were proposed in this pull request? In previous work SPARK-21513, we h

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19223 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19221 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19221 **[Test build #81735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81735/testReport)** for PR 19221 at commit [`0b8d47a`](https://github.com/apache/spark/commit/0b

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #81731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81731/testReport)** for PR 19222 at commit [`b7ffa10`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81731/ Test FAILed. ---

[GitHub] spark issue #16121: [SPARK-16589][PYTHON] Chained cartesian produces incorre...

2017-09-13 Thread aray
Github user aray commented on the issue: https://github.com/apache/spark/pull/16121 I'll take a look, sorry about that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-m

[GitHub] spark issue #19207: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19207 **[Test build #81736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81736/testReport)** for PR 19207 at commit [`de21671`](https://github.com/apache/spark/commit/de

[GitHub] spark issue #19211: [SPARK-18838][core] Add separate listener queues to Live...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19211 **[Test build #81728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81728/testReport)** for PR 19211 at commit [`ad6ff49`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #19211: [SPARK-18838][core] Add separate listener queues to Live...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19211 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19211: [SPARK-18838][core] Add separate listener queues to Live...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19211 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81728/ Test PASSed. ---

[GitHub] spark issue #19208: [SPARK-21087] [ML] CrossValidator, TrainValidationSplit ...

2017-09-13 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19208 CC @hhbyyh and @MLnick Does this look reasonable to you? And @hhbyyh would you want to split off a new JIRA for your original solution of adding an option to dump models to disk? Then we

[GitHub] spark issue #19207: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19207 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19207: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19207 **[Test build #81736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81736/testReport)** for PR 19207 at commit [`de21671`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #19207: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19207 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81736/ Test FAILed. ---

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138733553 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -85,7 +85,7 @@ private final LinkedLis

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138733992 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -162,14 +162,20 @@ private int getUsableCapac

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19181 cc @juliuszsompolski can you take a look, or perhaps take this for a test drive? --- - To unsubscribe, e-mail: reviews-unsubs

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138735379 --- Diff: core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java --- @@ -503,6 +511,47 @@ public void testGetIte

[GitHub] spark issue #19212: [SPARK-21988] Add default stats to StreamingExecutionRel...

2017-09-13 Thread joseph-torres
Github user joseph-torres commented on the issue: https://github.com/apache/spark/pull/19212 @zsxwing for review --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138737410 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -75,67 +76,131 @@ public static boolean unaligned() { return

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138737542 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java --- @@ -46,6 +47,42 @@ public static int roundNumberOfBytesToN

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138738642 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138739155 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,61 @@ import org.apache.spa

[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...

2017-09-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19206 @Chaos-Ju Sean means the bug you reference in the title has nothing to do with the code. --- - To unsubscribe, e-mail: reviews-un

[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...

2017-09-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19206 Also, doesn't this mean that your YARN service does not know about the federated HDFS service? Same thing can happen if you instead have an HA setup but YARN doesn't know about it. It will f

[GitHub] spark pull request #19141: [SPARK-21384] [YARN] Spark + YARN fails with Loca...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19141#discussion_r138742624 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -565,7 +565,6 @@ private[spark] class Client(

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138744155 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java --- @@ -18,6 +18,7 @@ package org.apache.spark.unsafe.types;

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138737907 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apa

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138744855 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/hash/Murmur3_x86_32.java --- @@ -59,6 +60,18 @@ public static int hashUnsafeWords(Object b

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138744794 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java --- @@ -46,6 +47,42 @@ public static int roundNumberOfBytesTo

[GitHub] spark issue #19215: [MINOR][SQL] Only populate type metadata for required ty...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19215 **[Test build #81737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81737/testReport)** for PR 19215 at commit [`7098295`](https://github.com/apache/spark/commit/70

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-09-13 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r138745656 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java --- @@ -46,6 +47,42 @@ public static int roundNumberOfBytesTo

[GitHub] spark pull request #19224: [SPARK-20990] Support multiple multiline JSON in ...

2017-09-13 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/19224 [SPARK-20990] Support multiple multiline JSON in the same file ## What changes were proposed in this pull request? The PR improves the JSON parsing so that now all the JSON documents in a

[GitHub] spark issue #19224: [SPARK-20990] Support multiple multiline JSON in the sam...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19224 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19223 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138751518 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -85,7 +85,7 @@ private final LinkedList sp

[GitHub] spark issue #19212: [SPARK-21988] Add default stats to StreamingExecutionRel...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19212 **[Test build #81738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81738/testReport)** for PR 19212 at commit [`81b1af9`](https://github.com/apache/spark/commit/81

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138751839 --- Diff: core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java --- @@ -503,6 +511,47 @@ public void testGetIterato

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19221 **[Test build #81735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81735/testReport)** for PR 19221 at commit [`0b8d47a`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19223 **[Test build #81739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81739/testReport)** for PR 19223 at commit [`29e7323`](https://github.com/apache/spark/commit/29

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19223 **[Test build #81739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81739/testReport)** for PR 19223 at commit [`29e7323`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19223 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81739/ Test FAILed. ---

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19221 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81735/ Test PASSed. ---

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19221 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19223: [SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support c...

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19223 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19188: [SPARK-21973][SQL] Add an new option to filter qu...

2017-09-13 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19188#discussion_r138752752 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmarkArguments.scala --- @@ -32,6 +36,10 @@ class TPCDSQueryBenchma

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19221 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.ap

[GitHub] spark issue #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiveFile.s...

2017-09-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19221 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19221: [SPARK-4131] Merge HiveTmpFile.scala to SaveAsHiv...

2017-09-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19221 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19188: [SPARK-21973][SQL] Add an new option to filter queries i...

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19188 **[Test build #81740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81740/testReport)** for PR 19188 at commit [`c957e47`](https://github.com/apache/spark/commit/c9

[GitHub] spark issue #19212: [SPARK-21988] Add default stats to StreamingExecutionRel...

2017-09-13 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19212 LGTM pending tests --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark pull request #19196: [SPARK-21977] SinglePartition optimizations break...

2017-09-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/19196#discussion_r138756939 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala --- @@ -117,8 +119,33 @@ class IncrementalExecution(

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19181 **[Test build #81734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81734/testReport)** for PR 19181 at commit [`2fd8e91`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19181 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81734/ Test PASSed. ---

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138756535 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138754738 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138754518 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -416,63 +424,52 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138760714 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -612,25 +623,26 @@ class CoarseGrainedSchedulerB

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138760403 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -314,6 +314,7 @@ private[spark] class TaskSchedulerImpl( def r

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138754158 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -416,63 +424,52 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138757437 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138753970 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -108,6 +110,10 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138761658 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala --- @@ -32,8 +32,13 @@ private[spark] object BlockManagerMessages {

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138755966 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19181 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138761778 --- Diff: docs/configuration.md --- @@ -1705,6 +1705,26 @@ Apart from these, the following properties are also available, and may be useful descript

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138760879 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1231,6 +1231,7 @@ private[spark] class BlockManager( def replicateBl

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138756115 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138758727 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138761246 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -120,6 +124,10 @@ class BlockManagerMasterEndpoint(

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138760176 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138760634 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -88,6 +89,12 @@ class CoarseGrainedSchedulerBack

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138758096 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138755830 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138761473 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala --- @@ -116,4 +121,11 @@ private[spark] object BlockManagerMessages {

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2017-09-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r138757166 --- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (A

<    1   2   3   4   5   6   >