[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3689/ Tes

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21427 I guess sending configurations is not that difficult. We can write configs (as `Map[String, String]` for further configurations in the future?) before `PythonUDFRunner.writeUDFs(dataOut, funcs, ar

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3555/ ---

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 Yup, my impression was that there could be a corner case too but I wasn't sure how much the corner case makes sense, and haven't checked it closelt yet. I believe elaborating the case might be h

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3555/ --- --

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3688/ Tes

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91274/ Test FAILed. ---

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91274/testReport)** for PR 21437 at commit [`9d95c12`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21413 **[Test build #91282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91282/testReport)** for PR 21413 at commit [`d8f3906`](https://github.com/apache/spark/commit/d8

[GitHub] spark pull request #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Pyt...

2018-05-29 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/21413#discussion_r191611779 --- Diff: python/pyspark/ml/regression.py --- @@ -619,6 +627,22 @@ def getSubsamplingRate(self): """ return self.getOrDefault(sel

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91281/testReport)** for PR 21366 at commit [`5b9c00f`](https://github.com/apache/spark/commit/5b

[GitHub] spark pull request #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocal...

2018-05-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21437#discussion_r191611421 --- Diff: python/pyspark/taskcontext.py --- @@ -88,3 +89,9 @@ def taskAttemptId(self): TaskAttemptID. """ return self._ta

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91273/ Test FAILed. ---

[GitHub] spark pull request #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmar...

2018-05-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21288#discussion_r191610297 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala --- @@ -131,211 +132,214 @@ object FilterPushdo

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 **[Test build #91273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91273/testReport)** for PR 21366 at commit [`b30ed39`](https://github.com/apache/spark/commit/b

[GitHub] spark pull request #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Pyt...

2018-05-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21413#discussion_r191609540 --- Diff: python/pyspark/ml/regression.py --- @@ -619,6 +627,22 @@ def getSubsamplingRate(self): """ return self.getOrDefault(s

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91271/ Test PASSed. ---

[GitHub] spark issue #21453: Test branch to see how Scala 2.11.12 performs

2018-05-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21453 Jenkins, add to whitelist. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark issue #21453: Test branch to see how Scala 2.11.12 performs

2018-05-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21453 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21451 **[Test build #91271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91271/testReport)** for PR 21451 at commit [`68c5d5f`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21454 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191607288 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,10 +219,15 @@ object ReorderAssociativeOpera

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21427 I'm sorry for the late review, but I think the current fix is still behavior change.. --- - To unsubscribe, e-mail: reviews-unsub

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21413 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21413 **[Test build #91280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91280/testReport)** for PR 21413 at commit [`714ab33`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91280/ Test PASSed. ---

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3687/ Tes

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21413 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Python GBT...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21413 **[Test build #91280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91280/testReport)** for PR 21413 at commit [`714ab33`](https://github.com/apache/spark/commit/71

[GitHub] spark pull request #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Pyt...

2018-05-29 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/21413#discussion_r191602398 --- Diff: python/pyspark/ml/regression.py --- @@ -619,6 +627,22 @@ def getSubsamplingRate(self): """ return self.getOrDefault(sel

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread PenguinToast
Github user PenguinToast commented on the issue: https://github.com/apache/spark/pull/21454 Retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21454 **[Test build #91279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91279/testReport)** for PR 21454 at commit [`badbf0e`](https://github.com/apache/spark/commit/ba

[GitHub] spark issue #21346: [SPARK-6237][NETWORK] Network-layer changes to allow str...

2018-05-29 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21346 So, one thing that I was thinking about is whether it would be worth it to make error handling a little better here. I think this is no worse than the current status quo, and looking at the related P

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3686/ Tes

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91277/testReport)** for PR 21437 at commit [`2ea9cbc`](https://github.com/apache/spark/commit/2e

[GitHub] spark issue #21428: [SPARK-24235][SS] Implement continuous shuffle writer fo...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21428 **[Test build #91278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91278/testReport)** for PR 21428 at commit [`65837ac`](https://github.com/apache/spark/commit/65

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21454 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91270/ Test FAILed. ---

[GitHub] spark pull request #21428: [SPARK-24235][SS] Implement continuous shuffle wr...

2018-05-29 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21428#discussion_r191596882 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/shuffle/ContinuousShuffleSuite.scala --- @@ -40,22 +60,129 @@ class Continu

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21454 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocal...

2018-05-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21437#discussion_r191596607 --- Diff: python/pyspark/taskcontext.py --- @@ -88,3 +89,9 @@ def taskAttemptId(self): TaskAttemptID. """ return self._ta

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21454 **[Test build #91270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91270/testReport)** for PR 21454 at commit [`f198f28`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-05-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r191596459 --- Diff: python/pyspark/worker.py --- @@ -111,9 +114,16 @@ def wrapped(key_series, value_series): "Number of columns of the returned pan

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191594326 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191594348 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191593987 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191591921 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191591799 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191593927 --- Diff: python/pyspark/sql/dataframe.py --- @@ -351,8 +352,62 @@ def show(self, n=20, truncate=True, vertical=False): else: pr

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191591455 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -291,37 +289,57 @@ class Dataset[T] private[sql]( } }

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-05-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r191595442 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -231,16 +234,17 @@ class Dataset[T] private[sql]( } /** -

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191595951 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,10 +219,15 @@ object ReorderAssociativeOperator

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-05-29 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21450 This doesn't seem to be addressing the issue reported in the bug. The exact same error happens with your patch: ``` $ ./bin/run-example Exception in thread "main" java.lang.IllegalAr

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21450 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21450 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91267/ Test PASSed. ---

[GitHub] spark pull request #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocal...

2018-05-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21437#discussion_r191589537 --- Diff: python/pyspark/taskcontext.py --- @@ -88,3 +89,9 @@ def taskAttemptId(self): TaskAttemptID. """ return s

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21450 **[Test build #91267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91267/testReport)** for PR 21450 at commit [`a69850b`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21457: [SPARK-24414][ui] Calculate the correct number of tasks ...

2018-05-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21457 +1 pending sparkQa, changes look good, and I manually verified against both the jira use cases. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91275/ Test PASSed. ---

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21427 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21427 **[Test build #91275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91275/testReport)** for PR 21427 at commit [`e322e1a`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-29 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191585661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,10 +219,15 @@ object ReorderAssociativeOp

[GitHub] spark issue #21454: [SPARK-24337][Core] Improve error messages for Spark con...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21454 IIUC this PR print the config key in the error message if the config value(either default or get from the configMap) can't be cast properly. Personally I think it add some value to include this

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-29 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191585050 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,10 +219,15 @@ object ReorderAssociativeOp

[GitHub] spark pull request #21454: [SPARK-24337][Core] Improve error messages for Sp...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21454#discussion_r191584812 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -448,6 +473,20 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Lo

[GitHub] spark pull request #21454: [SPARK-24337][Core] Improve error messages for Sp...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21454#discussion_r191582665 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -394,23 +407,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with L

[GitHub] spark pull request #21454: [SPARK-24337][Core] Improve error messages for Sp...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21454#discussion_r191582611 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -394,23 +407,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with L

[GitHub] spark pull request #21454: [SPARK-24337][Core] Improve error messages for Sp...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21454#discussion_r191582499 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -394,23 +407,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with L

[GitHub] spark issue #21457: [SPARK-24414][ui] Calculate the correct number of tasks ...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21457 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3685/ Tes

[GitHub] spark issue #21457: [SPARK-24414][ui] Calculate the correct number of tasks ...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21457 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21413: [SPARK-23161][PYSPARK][ML]Add missing APIs to Pyt...

2018-05-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21413#discussion_r191581932 --- Diff: python/pyspark/ml/regression.py --- @@ -619,6 +627,22 @@ def getSubsamplingRate(self): """ return self.getOrDefault(s

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91269/ Test PASSed. ---

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21409 **[Test build #91269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91269/testReport)** for PR 21409 at commit [`e90fa00`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21457: [SPARK-24414][ui] Calculate the correct number of tasks ...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21457 **[Test build #91276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91276/testReport)** for PR 21457 at commit [`40b6cb7`](https://github.com/apache/spark/commit/40

[GitHub] spark pull request #21457: [SPARK-24414][ui] Calculate the correct number of...

2018-05-29 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/21457 [SPARK-24414][ui] Calculate the correct number of tasks for a stage. This change takes into account all non-pending tasks when calculating the number of tasks to be shown. This also means that wh

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21427 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3684/ Tes

[GitHub] spark issue #21453: Test branch to see how Scala 2.11.12 performs

2018-05-29 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21453 Here is the issue in Scala side. https://github.com/scala/bug/issues/10913 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21427 **[Test build #91275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91275/testReport)** for PR 21427 at commit [`e322e1a`](https://github.com/apache/spark/commit/e3

[GitHub] spark issue #21453: Test branch to see how Scala 2.11.12 performs

2018-05-29 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21453 I'm also looking at this issue. The challenge is that one of the hacks we use to initialize the Spark before REPL sees any files was removed in Scala 2.11.12. https://github.com/apache/s

[GitHub] spark issue #21449: [SPARK-24385][SQL] Resolve self-join condition ambiguity...

2018-05-29 Thread daniel-shields
Github user daniel-shields commented on the issue: https://github.com/apache/spark/pull/21449 This case can also occur when the datasets are different but share a common lineage. Consider the following: `df = spark.range(10) df1 = df.groupby('id').count() df2 = df.groupby('

[GitHub] spark issue #21390: [SPARK-24340][Core] Clean up non-shuffle disk block mana...

2018-05-29 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21390 Are there any other concerns over this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3683/ Tes

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3549/ ---

[GitHub] spark issue #21455: [SPARK-24093][DStream][Minor]Make some fields of KafkaSt...

2018-05-29 Thread merlintang
Github user merlintang commented on the issue: https://github.com/apache/spark/pull/21455 @jerryshao can you review this minor update ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additio

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-29 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191567638 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala ---

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-05-29 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r191568423 --- Diff: resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh --- @@ -0,0 +1,91 @@ +#!/usr/bin/env bash + +#

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91274/testReport)** for PR 21437 at commit [`9d95c12`](https://github.com/apache/spark/commit/9d

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21366 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3549/ --- --

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3682/ Tes

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21366 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21449: [SPARK-24385][SQL] Resolve self-join condition ambiguity...

2018-05-29 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21449 @daniel-shields in that case you have 2 different datasets `df1` and `df2`. So they are 2 distinct attributes and the check `a.sameRef(b)` would return false. This is applied only in case you have

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-05-29 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21403 @juliuszsompolski yes, you're right, sorry, SPARK-24395 uses literal and not subqueries, sorry. --- - To unsubscribe, e-mail: r

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-05-29 Thread gaborgsomogyi
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/21450 cc @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...

<    1   2   3   4   5   6   >