[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165482549 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -420,63 +432,53 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165488466 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(

[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20479 **[Test build #86940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86940/testReport)** for PR 20479 at commit [`7fdfc35`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20479 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86940/ Test PASSed. ---

[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20479 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20470 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20470 **[Test build #86945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86945/testReport)** for PR 20470 at commit [`b962167`](https://github.com/apache/spark/commit/b9

[GitHub] spark pull request #20327: [SPARK-12963][CORE] NM host for driver end points

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20327#discussion_r165496876 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -744,7 +744,9 @@ object SparkSubmit extends CommandLineUtils with Logging {

[GitHub] spark issue #20167: [SPARK-16501] [MESOS] Allow providing Mesos principal & ...

2018-02-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20167 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark pull request #20167: [SPARK-16501] [MESOS] Allow providing Mesos princ...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20167#discussion_r165497310 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -71,40 +74,64 @@ trait MesosSched

[GitHub] spark issue #20167: [SPARK-16501] [MESOS] Allow providing Mesos principal & ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20167 **[Test build #86946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86946/testReport)** for PR 20167 at commit [`adc59ea`](https://github.com/apache/spark/commit/ad

[GitHub] spark issue #20167: [SPARK-16501] [MESOS] Allow providing Mesos principal & ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20167 **[Test build #86946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86946/testReport)** for PR 20167 at commit [`adc59ea`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #20167: [SPARK-16501] [MESOS] Allow providing Mesos principal & ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20167 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86946/ Test FAILed. ---

[GitHub] spark issue #20167: [SPARK-16501] [MESOS] Allow providing Mesos principal & ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20167 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20470 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86945/ Test PASSed. ---

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20470 **[Test build #86945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86945/testReport)** for PR 20470 at commit [`b962167`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20470 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20476: [SPARK-23301][SQL] data source column pruning should wor...

2018-02-01 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20476 Since you are being more and more familar with our codes, I believe you can offer us more useful inputs. Let me merge this PR for fixing the bugs. Then, we can have more detailed discus

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20455 **[Test build #86941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86941/testReport)** for PR 20455 at commit [`6d5f7ec`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20455 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86941/ Test PASSed. ---

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20455 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165504275 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class AbstractApplicationResour

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165504748 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class AbstractApplicationResour

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/20481 [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with the completed timestamp before cleaning up them ## What changes were proposed in this pull request? Sort jobs/stages/tasks/queries w

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

2018-02-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20272 So how are you expected to contact the thrift server after it's up, and even figure out where it started? Isn't the container unreachable from the outside world unless you do some port mapping on the

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20481 cc @vanzin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/489/ Test

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20481 **[Test build #86947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86947/testReport)** for PR 20481 at commit [`761f1ee`](https://github.com/apache/spark/commit/76

[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...

2018-02-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/20408 UI looks good to me, and taking a quick look at the code it looks fine, sorry haven't had time to do in depth review,but if @squito is good with it I'm fine with it. --- --

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20480 cc @jiangxb1987 @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20460 @jiangxb1987 it seems `spark.cores.max` only applies to standalone and Mesos coarse-grained modes. In k8s mode, `spark.cores.max` is not applicable. I think the check really should not be here, b

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20460 Yea, we should either move the check to elsewhere or modify it, to avoid potential failure. --- - To unsubscribe, e-mail: re

[GitHub] spark issue #20474: [SPARK-23235][Core] Add executor Threaddump to api

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20474 **[Test build #86948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86948/testReport)** for PR 20474 at commit [`8b91ef3`](https://github.com/apache/spark/commit/8b

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165509830 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class AbstractApplicationR

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165510071 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class AbstractApplicationR

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165510869 --- Diff: core/src/main/scala/org/apache/spark/status/KVUtils.scala --- @@ -69,14 +69,17 @@ private[spark] object KVUtils extends Logging { db

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20460 @jiangxb1987 fixed the check in https://github.com/apache/spark/pull/20460/commits/44e489eca6c22254fa57ff6f4750e3af85c43e34. ---

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165512101 --- Diff: core/src/main/scala/org/apache/spark/status/KVUtils.scala --- @@ -69,14 +69,17 @@ private[spark] object KVUtils extends Logging { db

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20460 **[Test build #86949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86949/testReport)** for PR 20460 at commit [`44e489e`](https://github.com/apache/spark/commit/44

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165512677 --- Diff: core/src/main/scala/org/apache/spark/status/KVUtils.scala --- @@ -69,14 +69,17 @@ private[spark] object KVUtils extends Logging { db

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20460 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/490/ Test

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20460 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20460 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/ --- ---

[GitHub] spark pull request #20334: How to check registered table name.

2018-02-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20334 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20262: Branch 2.2

2018-02-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20262 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20460 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/ --- -

[GitHub] spark issue #20462: [SPARK-23020][core] Fix another race in the in-process l...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20462 **[Test build #86942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86942/testReport)** for PR 20462 at commit [`82c276f`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #20462: [SPARK-23020][core] Fix another race in the in-process l...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20462 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86942/ Test PASSed. ---

[GitHub] spark issue #20462: [SPARK-23020][core] Fix another race in the in-process l...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20462 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165515603 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -69,6 +70,10 @@ private[spark] class Client( p

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165516482 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1104,14 +1117,39 @@ private[spark] class Client(

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165515886 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -656,7 +664,9 @@ private[spark] class Client( // C

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165518845 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1104,14 +1117,39 @@ private[spark] class Client(

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165516090 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -784,6 +794,9 @@ private[spark] class Client( val

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165516542 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1104,14 +1117,39 @@ private[spark] class Client(

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165518653 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -1104,14 +1117,39 @@ private[spark] class Client(

[GitHub] spark pull request #19616: [SPARK-22404][YARN] Provide an option to use unma...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19616#discussion_r165516803 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -51,33 +52,16 @@ import org.apache.spark.util._

[GitHub] spark issue #20470: [SPARK-23296][YARN] Include stacktrace in YARN-app diagn...

2018-02-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20470 Merging to master / 2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #20470: [SPARK-23296][YARN] Include stacktrace in YARN-ap...

2018-02-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20470 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20445 **[Test build #86950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86950/testReport)** for PR 20445 at commit [`478ad17`](https://github.com/apache/spark/commit/47

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/491/ Test

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/492/ Test

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20445 **[Test build #86951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86951/testReport)** for PR 20445 at commit [`6389d80`](https://github.com/apache/spark/commit/63

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86950/ Test FAILed. ---

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20445 **[Test build #86950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86950/testReport)** for PR 20445 at commit [`478ad17`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20445 This PR is currently blocked by a DataSourceV2ScanExec bug, which is being fixed in this PR https://github.com/apache/spark/pull/20387 ---

[GitHub] spark pull request #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSo...

2018-02-01 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20445#discussion_r165522181 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala --- @@ -89,7 +96,7 @@ case class MemoryStream[A : Encoder](id:

[GitHub] spark pull request #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSo...

2018-02-01 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20445#discussion_r165222931 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -270,16 +270,17 @@ class MicroBatchExecutio

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20445 **[Test build #86951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86951/testReport)** for PR 20445 at commit [`6389d80`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #20445: [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20445 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86951/ Test FAILed. ---

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20480 **[Test build #86943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86943/testReport)** for PR 20480 at commit [`df96f0c`](https://github.com/apache/spark/commit/d

[GitHub] spark pull request #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyA...

2018-02-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20473#discussion_r165527157 --- Diff: python/run-tests.py --- @@ -151,6 +151,38 @@ def parse_opts(): return opts +def _check_dependencies(python_exec, modul

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20480 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86943/ Test PASSed. ---

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/493/ Test

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20481 **[Test build #86952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86952/testReport)** for PR 20481 at commit [`f0de4be`](https://github.com/apache/spark/commit/f0

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20481 @vanzin just updated it. I don't fix the task order as I think it's already using `stage` index and I cannot iterate tasks using two indices. --- --

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-02-01 Thread bersprockets
Github user bersprockets commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r165528385 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class PythonWorkerFactory(p

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/494/ Test

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

2018-02-01 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20479#discussion_r165528709 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala --- @@ -655,4 +655,35 @@ class OrcQuerySuite exten

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20481 > I cannot iterate tasks using two indices. You can actually; indices can have a parent index, and there's actually a bunch of examples in `TaskDataWrapper`. Use them like this.

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20480 **[Test build #86944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86944/testReport)** for PR 20480 at commit [`afe40e5`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86944/ Test PASSed. ---

[GitHub] spark pull request #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/querie...

2018-02-01 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20481#discussion_r165529443 --- Diff: core/src/main/scala/org/apache/spark/status/storeTypes.scala --- @@ -73,6 +73,8 @@ private[spark] class JobDataWrapper( @JsonIgnore @KVIndex

[GitHub] spark issue #20480: [Spark-23306] Fix the oom caused by contention

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20480 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20424: [Spark-23240][python] Better error message when extraneo...

2018-02-01 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20424 So, I think I am fine. I believe it's already a corner case and I think we don't have to put a lot of efforts on this ... to be honest. I only wanted to double check if there is a slight step ba

[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...

2018-02-01 Thread ozzieba
Github user ozzieba commented on the issue: https://github.com/apache/spark/pull/20272 @vanzin In general Kubernetes makes this super easy: - The most basic workflow is to use the driver pod name (which is in the output of Spark Submit, or can be found with `kubectl get pods`), and

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20481 > This is fine, but if you want you can probably replace the filters in the listener by setting this to -1 for running jobs / stages / others, and starting iteration at "0". Not sure if I g

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/495/ Test

[GitHub] spark issue #20481: [SPARK-23307][WEBUI]Sort jobs/stages/tasks/queries with ...

2018-02-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20481 **[Test build #86953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86953/testReport)** for PR 20481 at commit [`0424c1d`](https://github.com/apache/spark/commit/04

[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

2018-02-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20479 Thank you for review and approval! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

2018-02-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20479#discussion_r165537214 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala --- @@ -655,4 +655,35 @@ class OrcQuerySuite ext

<    1   2   3   4   5   6   7   >