[GitHub] spark pull request #22630: [SPARK-25497][SQL] Limit operation within whole s...

2018-10-08 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22630#discussion_r223474906 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala --- @@ -345,6 +345,61 @@ trait CodegenSupport extends SparkPlan

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21710 **[Test build #97125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97125/testReport)** for PR 21710 at commit [`0393076`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Add base class for token rene...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r223472701 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/AbstractCredentialRenewer.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to the Apa

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97125/ Test PASSed. ---

[GitHub] spark pull request #22653: [SPARK-25659][PYTHON][TEST] Test type inference s...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22653#discussion_r223475373 --- Diff: python/pyspark/sql/tests.py --- @@ -1149,6 +1149,75 @@ def test_infer_schema(self): result = self.spark.sql("SELECT l[0].a from te

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97127/testReport)** for PR 22623 at commit [`20c4e01`](https://github.com/apache/spark/commit/20

[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22664#discussion_r223476989 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala --- @@ -34,10 +34,15 @@ import org.apache.sp

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97127/ Test FAILed. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97127/testReport)** for PR 22623 at commit [`20c4e01`](https://github.com/apache/spark/commit/2

[GitHub] spark pull request #22630: [SPARK-25497][SQL] Limit operation within whole s...

2018-10-08 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22630#discussion_r223478828 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -452,46 +452,68 @@ case class RangeExec(range: org.apac

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223479414 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,211 @@ head(sql("select * from aggregates")) -# Using F

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223479786 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode = "complete", checkpointLocation =

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #97121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97121/testReport)** for PR 21688 at commit [`a4ed19d`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97121/ Test FAILed. ---

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223480732 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,161 @

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223480986 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51 @@

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223481016 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,211 @@ head(sql("select * from aggregates")) -# Usin

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223481427 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,161 @

[GitHub] spark issue #22643: [SPARK-25630][TEST] Reduce test time of HadoopFsRelation...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22643 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comm

[GitHub] spark issue #22673: [SPARK-20144] Allow reading files in order with spark.sq...

2018-10-08 Thread darabos
Github user darabos commented on the issue: https://github.com/apache/spark/pull/22673 > @darabos . Thank you for trying to make a contribution. However, we had better discuss on that JIRA first before making a PR. Especially, for SPARK-20144 which is discussed already, it does.

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21809 **[Test build #97120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97120/testReport)** for PR 21809 at commit [`0be099a`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21809 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-10-08 Thread michal-databricks
Github user michal-databricks commented on the issue: https://github.com/apache/spark/pull/21688 @pgandhi999 thanks for the changes. I will review them tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97120/ Test FAILed. ---

[GitHub] spark pull request #22643: [SPARK-25630][TEST] Reduce test time of HadoopFsR...

2018-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22643 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22667: [SPARK-25673][BUILD] Remove Travis CI which enables Java...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22667 Thank you for taking care of this, @HyukjinKwon . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For ad

[GitHub] spark pull request #22612: [SPARK-24958] Add executors' process tree total m...

2018-10-08 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/22612#discussion_r223467439 --- Diff: core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala --- @@ -0,0 +1,272 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223483718 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51 @@

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 **[Test build #97128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97128/testReport)** for PR 21669 at commit [`e303048`](https://github.com/apache/spark/commit/e3

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97129/testReport)** for PR 22627 at commit [`222bfc6`](https://github.com/apache/spark/commit/22

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22614 **[Test build #97119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97119/testReport)** for PR 22614 at commit [`544b2ad`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97119/ Test PASSed. ---

[GitHub] spark pull request #22612: [SPARK-24958] Add executors' process tree total m...

2018-10-08 Thread rezasafi
Github user rezasafi commented on a diff in the pull request: https://github.com/apache/spark/pull/22612#discussion_r223486041 --- Diff: core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala --- @@ -0,0 +1,272 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3803/ ---

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21809 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22029 @mgaido91 . The failure is relevant to this PR. Could you take a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@sp

[GitHub] spark pull request #22029: [SPARK-24395][SQL] IN operator should return NULL...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22029#discussion_r223488428 --- Diff: sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql --- @@ -14,8 +14,25 @@ CREATE

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21809 **[Test build #97130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97130/testReport)** for PR 21809 at commit [`0be099a`](https://github.com/apache/spark/commit/0b

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97129/testReport)** for PR 22627 at commit [`222bfc6`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97129/ Test PASSed. ---

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3803/ --- --

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3803/

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3804/

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223490214 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51 @@

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22614 As @tejasapatil suggested above, this fallback is not suggested to be on in a prod setting. It could also impact the system health (e.g., the other concurrent queries that are querying the same H

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491087 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -# Usin

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491101 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -# Usin

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223491099 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490811 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -# Usin

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490770 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -# Usin

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223491915 --- Diff: docs/running-on-kubernetes.md --- @@ -799,4 +815,168 @@ specific to Spark on Kubernetes. This sets the major Python version of the docker

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223491937 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile var exitedC

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22614 Also, https://github.com/apache/spark/pull/22614#discussion_r223172392 proposes a better and safer solution by introducing `exponential backoff with retries`. @kmanamcheri Could you update your P

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223492379 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51 @@

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223493848 --- Diff: docs/running-on-kubernetes.md --- @@ -185,6 +185,22 @@ To use a secret through an environment variable use the following options to the --c

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Add base class for token rene...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r223493773 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/AbstractCredentialRenewer.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22627 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apach

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97131/testReport)** for PR 22627 at commit [`9d60534`](https://github.com/apache/spark/commit/9d

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22614 > a better and safer solution by introducing exponential backoff with retries I'm confused about that suggestion. What is being retried? The user query? The HMS call? What is changing in bet

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3805/

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit [`96104bd`](https://github.com/apache/spark/commit/96

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223497773 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile var exitedC

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97131/testReport)** for PR 22627 at commit [`9d60534`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97131/ Test PASSed. ---

[GitHub] spark pull request #22614: [SPARK-25561][SQL] Implement a new config to cont...

2018-10-08 Thread kmanamcheri
Github user kmanamcheri commented on a diff in the pull request: https://github.com/apache/spark/pull/22614#discussion_r223498018 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala --- @@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends Shi

[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/22305 I think there is a typo in your example in the description ``` @pandas_udf('double', PandasUDFType.GROUPED_AGG) def avg(v): return v.mean() return avg ``` I think y

[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...

2018-10-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/22305 @BryanCutler Yes that was a typo :) Thanks! I am also +1 to support numpy data structure in addition to Pandas. So happy to discuss here or separately. ---

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread kmanamcheri
Github user kmanamcheri commented on the issue: https://github.com/apache/spark/pull/22614 @gatorsmile, @tejasapatil was reviewing the code before I added the new config option. I have asked him to review the new code. Lets see what his thoughts are on that. I have also asked him clar

[GitHub] spark issue #22635: [SPARK-25591][PySpark][SQL] Avoid overwriting deserializ...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22635 How about pandas UDF? How about using RDD APIs? Do we face the same issues? --- - To unsubscribe, e-mail: reviews-unsubscr...@

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22627 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97133/testReport)** for PR 22623 at commit [`9e60602`](https://github.com/apache/spark/commit/9e

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504565 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile var

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504756 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile var

[GitHub] spark issue #22560: [SPARK-25547][Spark Core] Pluggable JDBC connection fact...

2018-10-08 Thread fsauer65
Github user fsauer65 commented on the issue: https://github.com/apache/spark/pull/22560 @gatorsmile did you get a chance to think about this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For add

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223506840 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223505747 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223507242 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala --- @@ -63,7 +65,7 @@ private[spark] object PythonEvalType { */

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223507984 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -70,23 +70,25 @@ trait TestPrematureExit { mainObject: Command

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit [`96104bd`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97132/ Test FAILed. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22654: [SPARK-25660][SQL] Fix for the backward slash as CSV fie...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22654 @gatorsmile Please, take a look at this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22429 @zsxwing Please, have a look at the PR one more time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addi

[GitHub] spark issue #22656: [SPARK-25669][SQL] Check CSV header only when it exists

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22656 @HyukjinKwon Could you look at the PR, please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22615: [SPARK-25016][BUILD][CORE] Remove support for Had...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22615#discussion_r223509836 --- Diff: dev/create-release/release-build.sh --- @@ -191,9 +191,17 @@ if [[ "$1" == "package" ]]; then make_binary_release() { NAME=$1

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97134/testReport)** for PR 22623 at commit [`71e3e30`](https://github.com/apache/spark/commit/71

[GitHub] spark issue #22628: [SPARK-25641] Change the spark.shuffle.server.chunkFetch...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22628 Is it supposed to the flaky ChunkFetchIntegrationSuite? http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.network.ChunkFetchIntegrationSuite&test_name=fetchFileChunk ---

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn...

2018-10-08 Thread NiharS
Github user NiharS commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223512824 --- Diff: core/src/main/scala/org/apache/spark/internal/Logging.scala --- @@ -192,7 +211,15 @@ private[spark] object Logging { defaultSparkLog4jC

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22482 **[Test build #97135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97135/testReport)** for PR 22482 at commit [`94e9859`](https://github.com/apache/spark/commit/94

[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22173#discussion_r223513440 --- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java --- @@ -77,17 +82,54 @@ private static final MessageEnc

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 I guess we've just pinged @shaneknapp ! But I figured the jobs would simply fail and could be removed at leisure. Yes, this mechanism is a little weird but may be the simplest thing here. I

<    1   2   3   4   5   6   >