[GitHub] spark pull request #21397: [SPARK-24334] Fix race condition in ArrowPythonRu...

2018-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21397#discussion_r191040434 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala --- @@ -94,8 +88,18 @@ class ArrowPythonRunner(

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-05-25 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r191040210 --- Diff: python/pyspark/sql/tests.py --- @@ -4931,6 +4931,63 @@ def foo3(key, pdf): expected4 = udf3.func((), pdf)

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-25 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21319 I'll open a PR to demonstrate what I'm proposing. It wouldn't change the plan, it would use a reader and push-down to report stats correctly as a temporary fix. I'm -1 on this PR. I'll have

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91176/ Test PASSed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91176/testReport)** for PR 20697 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91175/ Test PASSed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91175/testReport)** for PR 20697 at commit

[GitHub] spark pull request #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files...

2018-05-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21426#discussion_r191039095 --- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala --- @@ -153,4 +154,30 @@ object PythonRunner { .map { p =>

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91178/ Test PASSed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91178/testReport)** for PR 21416 at commit

[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...

2018-05-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21435#discussion_r191038840 --- Diff: python/pyspark/sql/functions.py --- @@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()): @since(2.3) def

[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...

2018-05-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21435#discussion_r191038814 --- Diff: python/pyspark/sql/functions.py --- @@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()): @since(2.3) def

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91172/ Test PASSed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91172/testReport)** for PR 20697 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21428 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91174/ Test PASSed. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91174 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91174/testReport)** for PR 21428 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91173/ Test FAILed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91173/testReport)** for PR 20697 at commit

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-05-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r191037717 --- Diff: python/pyspark/worker.py --- @@ -111,9 +114,16 @@ def wrapped(key_series, value_series): "Number of columns of the returned

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21416 I just found that PySpark's `isin` is capable of similar feature. One issue is Scala/Java API now doesn't match with PySpark API. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91170/ Test PASSed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91170 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91170/testReport)** for PR 20697 at commit

[GitHub] spark issue #21425: [SPARK-24381][TESTING] Add unit tests for NOT IN subquer...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21425 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91171/ Test PASSed. ---

[GitHub] spark issue #21425: [SPARK-24381][TESTING] Add unit tests for NOT IN subquer...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21425 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21425: [SPARK-24381][TESTING] Add unit tests for NOT IN subquer...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21425 **[Test build #91171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91171/testReport)** for PR 21425 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91168/ Test PASSed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91168/testReport)** for PR 20697 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 Hm, I'm glad I tried to paste output of running against master (well, strictly my rebase of this PR, but functionally the same thing). It's throwing some error that I need to debug:

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3606/

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3605/

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3604/

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3476/ ---

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21436 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91177/ Test FAILed. ---

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21436 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191034109 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala --- @@

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21436 **[Test build #91177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91177/testReport)** for PR 21436 at commit

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21436 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21436 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3603/

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191034013 --- Diff: resource-managers/kubernetes/integration-tests/src/test/resources/log4j.properties --- @@ -0,0 +1,31 @@ +# +# Licensed to the Apache

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191033974 --- Diff: resource-managers/kubernetes/integration-tests/pom.xml --- @@ -0,0 +1,230 @@ + + +http://maven.apache.org/POM/4.0.0;

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191033921 --- Diff: resource-managers/kubernetes/integration-tests/pom.xml --- @@ -0,0 +1,230 @@ + + +http://maven.apache.org/POM/4.0.0;

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191033936 --- Diff: resource-managers/kubernetes/integration-tests/pom.xml --- @@ -0,0 +1,230 @@ + + +http://maven.apache.org/POM/4.0.0;

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191033878 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Logging.scala --- @@ -0,0 +1,35 @@

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3476/ ---

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91179/ Test FAILed. ---

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91179/testReport)** for PR 21437 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3602/

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91179/testReport)** for PR 21437 at commit

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-25 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21437 @JoshRosen @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 Ok, I didn't do anything wrong. There is a bug in the Github web UI that the PR's web view doesn't get updated when I rebase it. I had to change what Github thought the branch I was merging into

[GitHub] spark pull request #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocal...

2018-05-25 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/21437 [SPARK-24397][PYSPARK] Added TaskContext.getLocalProperties in Python ## What changes were proposed in this pull request? Added. ## How was this patch tested? New test

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3475/ ---

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21436 **[Test build #91177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91177/testReport)** for PR 21436 at commit

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isinSet in DataFrame API for S...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91178/testReport)** for PR 21416 at commit

[GitHub] spark pull request #21436: [SPARK-24250][SQL][Follow-up] support accessing S...

2018-05-25 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/21436 [SPARK-24250][SQL][Follow-up] support accessing SQLConf inside tasks ## What changes were proposed in this pull request? We should not stop users from calling `getActiveSession` and

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3475/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3601/

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 Oops, I really did something wrong there. Please ignore while I fix. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3474/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 FYI - I just rebased to an updated master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91176/testReport)** for PR 20697 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3474/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3600/

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91175/testReport)** for PR 20697 at commit

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91174/testReport)** for PR 21428 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 Thanks for the pointer about @ifilonenko's comment. I removed the Kerberos/secrets related code. I'll post the output momentarily. ---

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-25 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21428 addressed comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191031379 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -0,0

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3473/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3599/

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3473/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20697 I won't really have time to look more closely at this until next week, at least, but I did notice an old comment from @ifilonenko that you did not address. If you could also post the output

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3598/

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91173/testReport)** for PR 20697 at commit

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 I saw that @mccheah put a comment in the e2e-prow.sh that we don't need it here, so I took that as an answer to my previous question. I think I got rid of the cloning and building Spark

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91172/testReport)** for PR 20697 at commit

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191028169 --- Diff: resource-managers/kubernetes/integration-tests/e2e/e2e-prow.sh --- @@ -0,0 +1,69 @@ +#!/bin/bash + +# Licensed to the Apache Software

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-25 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191028204 --- Diff: resource-managers/kubernetes/integration-tests/include/util.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 I will take a look at the docs too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-25 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 You're right, the prow-based codepath still does the cloning. Sorry! I did remove the cloning and building option in the entry point

[GitHub] spark pull request #21428: [SPARK-24235][SS] Implement continuous shuffle wr...

2018-05-25 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21428#discussion_r191027151 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowWriter.scala --- @@ -0,0 +1,54 @@ +/* +

[GitHub] spark pull request #21428: [SPARK-24235][SS] Implement continuous shuffle wr...

2018-05-25 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21428#discussion_r191027072 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/ContinuousShuffleWriter.scala --- @@ -0,0 +1,27 @@

  1   2   3   4   5   >