Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23253#discussion_r240048780
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/BadRecordException.scala
---
@@ -20,6 +20,16 @@ package
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23260
If you're on YARN, this feels like something you would manage via YARN and
its cluster management options. Is there a specific use case here, that this
has to happen in Spark?
---
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23072
@dongjoon-hyun @felixcheung how about now?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23253
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99888/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23253
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23253#discussion_r240041107
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -347,17 +347,28 @@ class JacksonParser(
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/23241
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23253
**[Test build #99888 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99888/testReport)**
for PR 23253 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23253
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23253
**[Test build #99888 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99888/testReport)**
for PR 23253 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23253
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23248#discussion_r240041688
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
---
@@ -131,8 +131,20 @@ object ExtractPythonUDFs extends
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/23241
Thanks a lot @srowen .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23266#discussion_r240053406
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/Table.java
---
@@ -18,9 +18,6 @@
package org.apache.spark.sql.sources.v2;
Github user seancxmao commented on a diff in the pull request:
https://github.com/apache/spark/pull/23258#discussion_r240038550
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -182,10 +182,13 @@ class SQLMetricsSuite extends
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23201#discussion_r240038837
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala
---
@@ -121,7 +122,26 @@ private[sql] class
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23263
My first impression is that it's a big change, which is reason for caution
here.
Visualizing a workflow is nice, but Spark's Pipelines are typically pretty
straightforward and linear. I
GitHub user davidvrba opened a pull request:
https://github.com/apache/spark/pull/23267
[SPARK-25401] [SQL] Reorder join predicates to match child outputOrdering
## What changes were proposed in this pull request?
In case of SortMergeJoin if tables are bucketed with keys
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user wangjiaochun commented on the issue:
https://github.com/apache/spark/pull/23225
Okey.@dongjoon-hyun
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23072
It looks enough to me, @srowen .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23266#discussion_r240074711
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/SupportsBatchRead.java
---
@@ -20,14 +20,27 @@
import
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/23248#discussion_r240079120
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
---
@@ -131,8 +131,20 @@ object ExtractPythonUDFs
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16812
I still don't think need this since the workaround is easy. If other
committers find it worth, I won't object.
If there are no interests fro this PR afterwards, I would just close this.
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
Hi @tgravescs , I tried it, but found it's difficult to produce
KILLED_BY_RESOURCEMANAGER exit status. I followed
[YARN-73](https://issues.apache.org/jira/browse/YARN-73)
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23267
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23225
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/23260
@srowen
For now executor log url is **static** in Spark, which forces Node Manager
to be alive even after application is finished, in order to provide executor
log in SHS.
This
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23263
> Visualizing a workflow is nice, but Spark's Pipelines are typically
pretty straightforward and linear. I could imagine producing a nicer
visualization than what you get from reading the Spark
Github user fjh100456 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22707#discussion_r240068636
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -774,4 +774,23 @@ class InsertSuite extends QueryTest with
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23225
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23225
**[Test build #99890 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99890/testReport)**
for PR 23225 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23267
**[Test build #99889 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99889/testReport)**
for PR 23267 at commit
Github user 10110346 commented on the issue:
https://github.com/apache/spark/pull/23228
I have updated, thanks all.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22707#discussion_r240073151
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -774,4 +774,23 @@ class InsertSuite extends QueryTest with
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23266#discussion_r240073831
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/SupportsBatchRead.java
---
@@ -20,14 +20,27 @@
import
Github user fjh100456 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22707#discussion_r240077883
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -774,4 +774,23 @@ class InsertSuite extends QueryTest with
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/23248
LGTM to the surgical fix for backporting.
We need to fix this rule with the other rules for avoiding making such a
strong and hidden assumption.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23267
**[Test build #99889 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99889/testReport)**
for PR 23267 at commit
Github user fjh100456 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22707#discussion_r240067762
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -774,4 +774,23 @@ class InsertSuite extends QueryTest with
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/23228
Please update the title `[MINOR][DOC] Update the condition description of
serialized shuffle`
---
-
To unsubscribe, e-mail:
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23260
Ok, got it. @vanzin or @squito or others would be better able to evaluate.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99889/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23267
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user fjh100456 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22707#discussion_r240070378
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
---
@@ -227,18 +227,22 @@ case class
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/23241
Merged to master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23255
**[Test build #99891 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99891/testReport)**
for PR 23255 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23255
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23255
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23204
@cloud-fan If we decide to partial revert SPARK-21052 and no need for
#23214, I will close it.
---
-
To unsubscribe,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23204
If we can quickly finish #23214 (within several days), let's go for it. But
if we can't, I'd suggest we do the partial revert first to fix the perf
regression, and add back the metrics later.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23228
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23228
**[Test build #99892 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99892/testReport)**
for PR 23228 at commit
Github user cloud-fan closed the pull request at:
https://github.com/apache/spark/pull/23265
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23201#discussion_r240090192
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala
---
@@ -121,7 +122,26 @@ private[sql] class
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23258#discussion_r240090371
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -182,10 +182,13 @@ class SQLMetricsSuite extends
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23228
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23228
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21877
**[Test build #99893 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99893/testReport)**
for PR 21877 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23248
If it's fine for 2.4, I think it's also fine for master as a temporary fix?
We can create another ticket to clean up the subquery optimization hack. IIUC
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21877
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21877
**[Test build #99893 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99893/testReport)**
for PR 21877 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21877
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99893/
Test FAILed.
---
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22305#discussion_r240092271
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/WindowInPandasExec.scala
---
@@ -144,24 +282,107 @@ case class WindowInPandasExec(
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23211#discussion_r240092936
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
---
@@ -267,6 +267,17 @@ object ScalarSubquery {
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23204
**[Test build #99894 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99894/testReport)**
for PR 23204 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23204
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23204
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23225
**[Test build #99890 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99890/testReport)**
for PR 23225 at commit
Github user JkSelf commented on the issue:
https://github.com/apache/spark/pull/23204
@cloud-fan @dongjoon-hyun update the patch, please help review if you have
time. Thanks.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99890/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23225
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
As @cloud-fan said `the hash join metrics is wrongly implemented`, we will
partial revert # SPARK-21052, no longer need this patch, close it ~
---
Github user LuciferYang closed the pull request at:
https://github.com/apache/spark/pull/23214
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23204
ok~ already close #23214
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22290
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23204
can we follow
https://github.com/apache/spark/pull/23204#issuecomment-445510026 and create a
new ticket?
---
-
To
GitHub user sadhen opened a pull request:
https://github.com/apache/spark/pull/23268
[Hive][Minor] Refactor on HiveShim and Add Unit Tests
## What changes were proposed in this pull request?
Refactor on HiveShim, and add Unit Tests.
## How was this patch tested?
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23255
**[Test build #99891 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99891/testReport)**
for PR 23255 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23255
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99891/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23255
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23268
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23268
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23268#discussion_r240097105
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala
---
@@ -53,19 +53,12 @@ private[hive] object HiveShim {
* This
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23225
Thank you, @wangjiaochun .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23211#discussion_r240097255
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -649,13 +664,16 @@ object CollapseProject extends
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23268
**[Test build #99895 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99895/testReport)**
for PR 23268 at commit
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23211#discussion_r240097479
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -984,6 +1002,28 @@ object PushDownPredicate extends
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/23225
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/23211
to make the PR smaller, can we add an individual rule
`PushdownLeftSemiOrAntiJoin` first?
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23268#discussion_r240097931
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala
---
@@ -53,19 +53,12 @@ private[hive] object HiveShim {
* This
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23268
Let's close this one.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/23268
@sadhen What is the motivation of this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user sadhen commented on a diff in the pull request:
https://github.com/apache/spark/pull/23268#discussion_r240098620
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala
---
@@ -53,19 +53,12 @@ private[hive] object HiveShim {
* This function
Github user sadhen commented on a diff in the pull request:
https://github.com/apache/spark/pull/23268#discussion_r240098806
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala
---
@@ -53,19 +53,12 @@ private[hive] object HiveShim {
* This function
1 - 100 of 177 matches
Mail list logo
