Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21546#discussion_r212163122
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -183,34 +178,106 @@ private[sql] object
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21546#discussion_r212162307
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -183,34 +178,106 @@ private[sql] object
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212162206
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22180
**[Test build #95132 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95132/testReport)**
for PR 22180 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22180
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22180
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/22180#discussion_r212161599
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -368,7 +369,11 @@ private[spark] class
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21546#discussion_r212161411
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3268,13 +3268,49 @@ class Dataset[T] private[sql](
}
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21546#discussion_r212158131
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -111,65 +113,58 @@ private[sql] object
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/22187
Thanks, updated both titles
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212160161
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/22188
OK, I reran the tests for the lower column count cases, and the runs with
the patch consistently show a tiny (1-3%) improvement compared to the master
branch. So even the lower column count
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20345
**[Test build #95131 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95131/testReport)**
for PR 20345 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20345
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20345
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22163
**[Test build #95130 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95130/testReport)**
for PR 22163 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22163
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20345
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22163
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user 10110346 commented on the issue:
https://github.com/apache/spark/pull/22163
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212154100
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22112
**[Test build #95129 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95129/testReport)**
for PR 22112 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22112
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22112
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22112
**[Test build #95128 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95128/testReport)**
for PR 22112 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22112
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22112
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21332
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95122/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21332
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21332
**[Test build #95122 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95122/testReport)**
for PR 21332 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22112
BTW, I think a cleaner fix is to make shuffle files reliable(e.g. put them
on HDFS), so that Spark will never retry a task from a finished shuffle map
stage. Then all the problems go away, the
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22185
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22185
LGTM, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22193
**[Test build #95127 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95127/testReport)**
for PR 22193 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22193
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95127/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22193
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21756
From what I talked to @shrutig offline, the use-case her team has is when
`UserGroupInformation.getCurrentUser.getAuthenticationMethod ==
AuthenticationMethod.KERBEROS`, they want to set the remote
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22193
**[Test build #95127 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95127/testReport)**
for PR 22193 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22193
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22193
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/22193
[SPARK-25186][SQL] Remove v2 save mode.
## What changes were proposed in this pull request?
This removes `SaveMode` from the v2 write API. Overwrite is temporarily
implemented by deleting
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22187
Yeah, I agreed with @rednaxelafx to directly ship the StructType objects
looks like a better solution. +1 for that.
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22063
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95121/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22063
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22063
**[Test build #95121 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95121/testReport)**
for PR 22063 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22192
Please remove all of the PR template text from the description.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22192
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22192
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22192
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22192
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user NiharS opened a pull request:
https://github.com/apache/spark/pull/22192
[SPARK-24918] Executor Plugin API
A continuation of @squito's executor plugin task. By his request I took his
code and added testing and moved the plugin initialization to a separate thread.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22185
+1, LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22188
That does seem counter intuitive, but no idea what could explain that since
the new code seems like a straight-forward better version.
---
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/22190
This is related to #21308, which adds `DeleteSupport`. Both
`BatchOverwriteSupport` and `DeleteSupport` use the same input to remove data
(`Filter[]`) and can reject deletes that don't align with
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21749
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/22188
Thanks @vanzin. In my benchmark tests, the tiny degradation (0.5%) in the
lower column count cases is pretty consistent, which concerns me a little. I am
going to re-run those tests in a
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21749
@vanzin Thanks for the reviewing. I'll merge it as it, and have another PR
trying to merge 2.11 and 2.12 branches. I had a positive conversation with
Scala community to add a proper hook for us, and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22187
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95119/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22187
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22187
**[Test build #95119 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95119/testReport)**
for PR 22187 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21749
The patch LGTM. I'd add a comment where Spark-specific code was inserted,
so it's easy to find later. But ok as is too.
Not sure why it seems like a big deal to have the Scala libraries add
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22191
**[Test build #95126 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95126/testReport)**
for PR 22191 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22177
You're sorting on stage ID but your description says task ID. Just a small
inconsistency.
This is also against branch-2.3, it should be against master.
---
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22177#discussion_r212134537
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala ---
@@ -337,7 +337,9 @@ private[ui] class JobPage(parent: JobsTab, store:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22177#discussion_r212134229
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala ---
@@ -18,18 +18,18 @@
package org.apache.spark.ui.jobs
import
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22188
LGTM. Will leave here for a bit to see if anyone else comments...
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22191
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95125/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22191
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22191
**[Test build #95125 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95125/testReport)**
for PR 22191 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22191
**[Test build #95125 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95125/testReport)**
for PR 22191 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22191
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22191
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/22191
[SPARK-25204][SS] Fix race in rate source test.
## What changes were proposed in this pull request?
Fix a race in the rate source tests. We need a better way of testing
restart
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22188
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95118/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22188
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22188
**[Test build #95118 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95118/testReport)**
for PR 22188 at commit
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22146#discussion_r212126595
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala
---
@@ -51,7 +57,13 @@
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22187
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95117/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22187
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22187
**[Test build #95117 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95117/testReport)**
for PR 22187 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22190
**[Test build #95124 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95124/testReport)**
for PR 22190 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95124/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22157
**[Test build #4285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4285/testReport)**
for PR 22157 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22190
**[Test build #95124 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95124/testReport)**
for PR 22190 at commit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22190#discussion_r212121224
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/MicroBatchWriteSupport.scala
---
@@ -18,27 +18,38 @@
package
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22190#discussion_r212120878
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/BatchPartitionOverwriteSupport.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95123/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22190
**[Test build #95123 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95123/testReport)**
for PR 22190 at commit
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22190#discussion_r212120411
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/BatchPartitionOverwriteSupport.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22190#discussion_r212119716
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/BatchOverwriteSupport.java
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22190#discussion_r212118021
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -279,10 +277,7 @@ private[kafka010] class
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22190
**[Test build #95123 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95123/testReport)**
for PR 22190 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22190
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/22190
SPARK-25188: Add WriteConfig to v2 write API.
## What changes were proposed in this pull request?
This updates the v2 write path to a similar structure as the v2 read path.
Individual
Github user rednaxelafx commented on the issue:
https://github.com/apache/spark/pull/22187
So the new solution now is to directly ship the `StructType` object as a
reference object. Why not? ;-)
I'm +1 on shipping the object directly instead of generating code to
recreate it on
101 - 200 of 600 matches
Mail list logo
