Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20531
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/20588
[SPARK-23352][PYTHON][BRANCH-2.3] Explicitly specify supported types in
Pandas UDFs
This PR backports https://github.com/apache/spark/pull/20531 to branch-2.3.
You can merge this pull request
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/20567
LGTM. I'd leave it to @HyukjinKwon and @gatorsmile whether we should merge
this into branch-2.3 or not.
---
-
To unsubscribe,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87326/
Test PASSed.
---
GitHub user zhuge134 opened a pull request:
https://github.com/apache/spark/pull/20586
Branch 2.1
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch
GitHub user zhuge134 opened a pull request:
https://github.com/apache/spark/pull/20587
Branch 2.2
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-2.2
Alternatively you can review and apply these changes as
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20531
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20531
Thank you for reviewing this, @icexelloss, @ueshin and @BryanCutler.
---
-
To unsubscribe, e-mail:
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19431
@tdas could you take a look at it?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167545871
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with
Github user a10y commented on the issue:
https://github.com/apache/spark/pull/20580
If you did add a test it should probably generate the Parquet file
programmatically rather than checking it in. Some examples in
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20586
@zhuge134, seems mistakenly open. Could you close this please?
---
-
To unsubscribe, e-mail:
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/20585
[SPARK-22002][SQL][FOLLOWUP][TEST] Add a test to check if the original
schema doesn't have metadata.
## What changes were proposed in this pull request?
This is a follow-up pr of #19231
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20531
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20587
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/20490
Been having talks with colleagues last week and want to check something.
How exactly do Spark executors abort speculative jobs without waiting for
them get into the ready-to-commit
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/20587
@zhuge134, looks mistaken open. Could you please double-check and close
this PR?
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20567
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20567
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/808/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20567
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/809/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20588
**[Test build #87333 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87333/testReport)**
for PR 20588 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20585
**[Test build #87331 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87331/testReport)**
for PR 20585 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/20585
cc @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87326 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87326/testReport)**
for PR 20583 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/807/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20588
cc @ueshin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/10527
@vectorijk Is this PR dead ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20546#discussion_r167514294
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1715,7 +1715,7 @@ class SparkContext(config: SparkConf) extends Logging
{
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20586
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20531
**[Test build #87328 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87328/testReport)**
for PR 20531 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20587
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20531
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87328/
Test PASSed.
---
Github user stoader commented on the issue:
https://github.com/apache/spark/pull/19775
@smurakozi thank you for reviewing. The PR has been updated based on your
comments.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20586
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20531
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87329/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20531
**[Test build #87329 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87329/testReport)**
for PR 20531 at commit
Github user a10y commented on the issue:
https://github.com/apache/spark/pull/20580
As far as we can tell this is an accidental breaking change, as dropping
support for this in vectorized Parquet reader was never called out. We have
Parquet datasets with binary columns with logical
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20567
**[Test build #87332 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87332/testReport)**
for PR 20567 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20582
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20582
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87330/
Test PASSed.
---
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/20588
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20531
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jamesthomp commented on the issue:
https://github.com/apache/spark/pull/20580
@kiszk - I've changed the implementation to no longer use
`column.isArray()` and instead just inline the decimal type check (so no
changes needed to the public api). I don't think you could
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20295#discussion_r167501868
--- Diff: python/pyspark/sql/types.py ---
@@ -1694,6 +1694,13 @@ def from_arrow_schema(arrow_schema):
for field in arrow_schema])
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/20400
@HyukjinKwon Thanks a lot for your help!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87331/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20585
**[Test build #87331 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87331/testReport)**
for PR 20585 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20582
**[Test build #87330 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87330/testReport)**
for PR 20582 at commit
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20521
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/810/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20585
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20581
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20581
Merged to master/2.3/2.2
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20585
**[Test build #87334 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87334/testReport)**
for PR 20585 at commit
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20396#discussion_r167591614
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
---
@@ -421,13 +456,220 @@ private[evaluation] object
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87333/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20567
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87332/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20567
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20521
thanks, merging to master/2.3!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20582
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20490
AFAIK, there is no KILL mechanism for speculative tasks, which means, if
multiple task attempts are running at the same time and one of them finishes
earlier, the others will keep running until
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/20567
Sorry I am late to the party.
https://github.com/apache/spark/pull/20567#issuecomment-364639922 Does look
like a bug to me. However, I am a bit concerned that such magic behavior would
be not
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20578#discussion_r167563685
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -158,18 +159,30 @@ class FPGrowth @Since("2.2.0") (
}
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20516
Now this no longer changes `spark.testing`, which was the main point here.
I think this should just be closed; the purpose is changing and unclear here.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20588
**[Test build #87333 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87333/testReport)**
for PR 20588 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20584
> I am suspicious about relationship between afterEach() and
addTaskCompletionListener (call close()). But, not sure. Let us try this
approach first.
This is one of my speculations.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20396#discussion_r167581903
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
---
@@ -421,13 +456,220 @@ private[evaluation] object
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20396#discussion_r167582857
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
---
@@ -421,13 +456,220 @@ private[evaluation] object
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20396#discussion_r167590799
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
---
@@ -421,13 +456,220 @@ private[evaluation] object
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20567
**[Test build #87332 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87332/testReport)**
for PR 20567 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20584
I think I rushed to take a look at the first time. Thanks for fixing this.
---
-
To unsubscribe, e-mail:
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20585#discussion_r167571089
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1168,4 +1168,26 @@ class JDBCSuite extends SparkFunSuite
val
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20582
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167567793
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with
Github user jamesthomp commented on the issue:
https://github.com/apache/spark/pull/20580
I'll see if I can generate a parquet file with the right schema to add for
a test, but probably cannot look at this till tomorrow.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20580
It was an accident, thanks for the fix!
Can we add a test? It's always good to have a test for a bug fix, even the
bug was introduced recently.
---
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20585#discussion_r167566823
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1168,4 +1168,26 @@ class JDBCSuite extends SparkFunSuite
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20555
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user rdblue reopened a pull request:
https://github.com/apache/spark/pull/20490
[SPARK-23323][SQL]: Support commit coordinator for DataSourceV2 writes
## What changes were proposed in this pull request?
DataSourceV2 batch writes should use the output commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20555
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/812/
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20580#discussion_r167656976
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
---
@@ -444,7 +444,8 @@ private void
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20579
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20580
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87335/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/20579#discussion_r167664375
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -72,6 +72,29 @@ class FileBasedDataSourceSuite extends
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20546
**[Test build #87336 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87336/testReport)**
for PR 20546 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20589
**[Test build #87343 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87343/testReport)**
for PR 20589 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20546
**[Test build #87345 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87345/testReport)**
for PR 20546 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20589
**[Test build #87344 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87344/testReport)**
for PR 20589 at commit
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20570#discussion_r167668944
--- Diff: core/src/main/resources/org/apache/spark/ui/static/webui.js ---
@@ -80,4 +80,6 @@ $(function() {
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20580
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20567
Regarding the error message, this is a good example to show how to provide
a user-friendly message. To the external end users, they might not be aware of
the internal implementation. They might
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20585
**[Test build #87334 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87334/testReport)**
for PR 20585 at commit
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20588#discussion_r167631828
--- Diff: python/pyspark/sql/tests.py ---
@@ -4325,15 +4333,16 @@ def data(self):
.withColumn("vs", array([lit(i) for i in range(20,
1 - 100 of 481 matches
Mail list logo
