Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/639/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21601
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/640/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21601
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199826429
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -744,11 +747,42 @@ case class
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21433
**[Test build #4203 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4203/testReport)**
for PR 21433 at commit
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21596#discussion_r199803209
--- Diff: pom.xml ---
@@ -158,8 +158,8 @@
2.11.12
2.11
1.9.13
-2.6.7
-
2.6.7.1
+2.9.6
+
2.9.6
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21657
@HyukjinKwon kindly ping
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21705
**[Test build #92564 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92564/testReport)**
for PR 21705 at commit
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21705#discussion_r199817343
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala
---
@@ -40,16 +40,24 @@ class ExecutorSideSQLConfSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/21440
@tgravescs @felixcheung @zsxwing maybe one of you could take a look? I got
a lgtm from marcelo but he's out for a few weeks, would prefer to get another
approval, plus I'll need review on
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21699
**[Test build #92561 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92561/testReport)**
for PR 21699 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21699
**[Test build #92568 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92568/testReport)**
for PR 21699 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20446
@WeichenXu123 looks like there was one more outstanding comment, about
using `.show()`?
---
-
To unsubscribe, e-mail:
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/21669#discussion_r199803715
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -211,6 +211,51 @@ private[spark] object Config
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/638/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21693
**[Test build #4204 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4204/testReport)**
for PR 21693 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21705
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/21669#discussion_r199806583
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala
---
@@ -81,4 +83,35 @@ private[spark] object
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21705
**[Test build #92565 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92565/testReport)**
for PR 21705 at commit
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21705#discussion_r199820857
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala
---
@@ -40,16 +40,24 @@ class ExecutorSideSQLConfSuite
GitHub user dmateusp opened a pull request:
https://github.com/apache/spark/pull/21706
[SPARK-24702] Fix Unable to cast to calendar interval in spark sql
## What changes were proposed in this pull request?
Making the `calendarinterval` a parse-able DataType keyword to allow
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21704
**[Test build #92562 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92562/testReport)**
for PR 21704 at commit
Github user hthuynh2 commented on the issue:
https://github.com/apache/spark/pull/21653
@jiangxb1987 yes, you are correct that it is actually ignored. I think it
doesn't worth to add a new TaskState because we might need to add changes in
many places but does not add much benefit.
Github user dmateusp commented on the issue:
https://github.com/apache/spark/pull/21706
Realized I need to add `calendarinterval` as a function as well to
reproduce the behavior of `int`, `date`, `string` etc..
example:
```scala
scala> spark.sql("select
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199857371
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21702
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dmateusp commented on the issue:
https://github.com/apache/spark/pull/21706
Just added it to the FunctionRegistry:
```scala
scala> spark.sql("DESC function calendarinterval").show(truncate=false)
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199878616
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21633
**[Test build #92563 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92563/testReport)**
for PR 21633 at commit
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21705#discussion_r199858281
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala
---
@@ -40,16 +40,24 @@ class ExecutorSideSQLConfSuite
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21667
**[Test build #92569 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92569/testReport)**
for PR 21667 at commit
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199861581
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
---
@@ -152,6 +152,12 @@ trait FileFormat {
}
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199861251
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
---
@@ -42,65 +38,9 @@ object DataSourceUtils {
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21699
cc @aray
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21702
**[Test build #92570 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92570/testReport)**
for PR 21702 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199867594
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
---
@@ -156,28 +156,6 @@ class HiveOrcSourceSuite extends
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21542
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/643/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21542
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199877744
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92564/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21653
IIUC this speculative task is not really killed right ? It is actually
ignored. Does that worth it to add a new TaskState for this case ?
---
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199857808
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21667
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21667
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/642/
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199860902
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -306,6 +306,7 @@ case class FileSourceScanExec(
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199863675
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
---
@@ -156,28 +156,6 @@ class HiveOrcSourceSuite extends
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21667#discussion_r199863349
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
---
@@ -148,6 +144,28 @@ class JsonFileFormat
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/21667
@HyukjinKwon @maropu I have updated the code. It is now using whitelist.
@cloud-fan Thanks for the review and +1
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21657
Eh .. actually can we revive
https://github.com/apache/spark/commit/64fad0b519cf35b8c0a0dec18dd3df9488a5ed25#diff-d19881aceddcaa5c60620fdcda99b4c4L79
? This sounds safer to me.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21542
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92571/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21542
**[Test build #92571 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92571/testReport)**
for PR 21542 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21542
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21542
**[Test build #92571 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92571/testReport)**
for PR 21542 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199879761
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21705
**[Test build #92564 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92564/testReport)**
for PR 21705 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21686
Seems fine to me otherwise.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199883857
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -744,11 +747,42 @@ case class StructsToJson(
Github user rednaxelafx commented on the issue:
https://github.com/apache/spark/pull/21699
This mostly looks good, but I'd like to ask a few things first:
1. The new overloaded `pivot()` that takes `Column` only exist for
`pivot(Column, Seq[Any])`. Were you planning to add a
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199907218
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21601
**[Test build #92566 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92566/testReport)**
for PR 21601 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21601
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21601
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92566/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21699
**[Test build #92568 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92568/testReport)**
for PR 21699 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21699
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92568/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21699
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21699
> Were you planning to add a new overload for each existing String
version, e.g. pivot(Column) and pivot(Column, java.util.List[Any])?
The methods have been added already. @rednaxelafx
GitHub user tgravescs opened a pull request:
https://github.com/apache/spark/pull/21707
Update for spark 2.2.2 release
Release process for Spark 2.2.2 say to update the test.
You can merge this pull request into a Git repository by running:
$ git pull
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21707
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/646/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21707
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21707
**[Test build #92576 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92576/testReport)**
for PR 21707 at commit
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21659#discussion_r199916087
--- Diff: python/docs/Makefile ---
@@ -1,19 +1,44 @@
# Makefile for Sphinx documentation
#
+ifndef SPHINXBUILD
+ifndef SPHINXPYTHON
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/21659
When you say orthogonally do you mean with another PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92573 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92573/testReport)**
for PR 21588 at commit
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21459
Thanks. Merged into master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/645/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21649
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92575 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92575/testReport)**
for PR 21686 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199890648
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df =
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199891105
--- Diff: python/pyspark/sql/functions.py ---
@@ -2235,6 +2240,28 @@ def to_json(col, options={}):
return Column(jc)
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199890912
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199891460
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df =
Github user mengxr commented on the issue:
https://github.com/apache/spark/pull/21659
@HyukjinKwon I reported the issue to Sphinx at
https://github.com/sphinx-doc/sphinx/issues/5142. We are not sure if Python 2
is the root cause. I don't have strong preferences on the approaches
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/644/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21433
**[Test build #4203 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4203/testReport)**
for PR 21433 at commit
Github user michaelmior commented on the issue:
https://github.com/apache/spark/pull/21433
@srowen Yes, I don't expect it will be widely used but I've personally
found it helpful in some performance debugging and it's a fairly low impact
change. I was just hoping to avoid having to
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21649
**[Test build #92574 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92574/testReport)**
for PR 21649 at commit
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21459
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21693
**[Test build #4204 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4204/testReport)**
for PR 21693 at commit
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199894916
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92565/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21705
**[Test build #92565 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92565/testReport)**
for PR 21705 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21659
@mengxr, thanks for reporting it. This will probably related with
`functools.wraps`'s behaviour between Python 2 and 3. Python 3 copies its
signature and Python 2 doesn't. Let me make a
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199899708
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType,
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199899927
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df = spark.createDataFrame(data,
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/21690
@yuanboliu From reading KafkaConsumer code, and from testing, I don't see
where consumer.position() alone would un-pause topicpartitions. See below.
Can you give a counter-example?
I
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199856590
--- Diff: python/pyspark/sql/functions.py ---
@@ -2235,6 +2240,28 @@ def to_json(col, options={}):
return Column(jc)
301 - 400 of 430 matches
Mail list logo
