Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239995006
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,13 +38,21 @@ case class CollectLimitExec(limit: Int, child
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239748512
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleWriteProcessor.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239744840
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleWriteProcessor.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239744767
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -78,6 +80,7 @@ object SQLMetrics {
private val
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239743452
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleWriteProcessor.scala ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
the code looks much cleaner now!
```
Sorry for the original rush and code, I should and will pay more attention
on coding clean and more discussion on optional implementation
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239698500
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -78,6 +80,7 @@ object SQLMetrics {
private val
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239698273
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -333,8 +343,19 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239698174
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleWriterProcessor.scala ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
Can we put the above in a closure and pass it into shuffle dependency? Then
in SQL we just put the above in SQL using custom metrics.
```
Yea, the commit of a780b70 achieve
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239548704
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239312090
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
```
can you separate the prs to rename read side metric and the write side
change?
```
No problem, next commit will revert the changes of rename read side
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239311564
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239311141
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239311018
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239069014
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239067552
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -163,6 +171,8 @@ object SQLMetrics
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239054315
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -299,12 +312,25 @@ class SQLMetricsSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239050549
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239049398
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -163,6 +171,8 @@ object SQLMetrics
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239049121
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -78,6 +78,7 @@ object SQLMetrics {
private val
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239049030
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239048356
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238732441
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -92,6 +92,12 @@ private[spark] class ShuffleMapTask
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
Thanks for your reply Wenchen, there's a sketch doc assigned in
JIRA:https://docs.google.com/document/d/1DX0gLkpk_NCE5MwI1_m4gnA2rLdjDkynZ02u2VWDR-8/edit
```
IMO shuffle write
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
cc @cloud-fan @gatorsmile @rxin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@SparkQA test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@SparkQA retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
@AmplabJenkins
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23207
test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/23207
[SPARK-26193][SQL] Implement shuffle write metrics in SQL
## What changes were proposed in this pull request?
1. Implement `SQLShuffleWriteMetricsReporter` on the SQL side
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23175
Thanks @cloud-fan @rxin.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237346452
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237346431
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -194,4 +202,16 @@ object SQLMetrics
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/23175
[SPARK-26142]followup: Move sql shuffle read metrics relatives to
SQLShuffleMetricsReporter
## What changes were proposed in this pull request?
Follow up for https://github.com
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
@rxin Thanks for guidance, I'll address these comments in a follow up PR
soon.
---
-
To unsubscribe, e-mail: reviews
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
Thanks @cloud-fan @gatorsmile @rxin !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236982643
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala ---
@@ -154,7 +156,10 @@ class ShuffledRowRDD
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
python UT failed cause jvm crush.
retest this pleas.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236926403
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala ---
@@ -154,7 +156,10 @@ class ShuffledRowRDD
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23153
Thanks for the fix from Wenchen,
```
the suites should also construct the dummy python udf from both side.
```
I fix the suite locally, they can be simply modified like
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23153#discussion_r236743539
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -155,19 +155,20 @@ object EliminateOuterJoin extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23153#discussion_r236647128
--- Diff: python/pyspark/sql/tests/test_udf.py ---
@@ -209,6 +209,18 @@ def test_udf_in_join_condition(self):
with self.sql_conf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236720141
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala ---
@@ -154,7 +156,14 @@ class ShuffledRowRDD
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236646423
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala ---
@@ -154,7 +156,14 @@ class ShuffledRowRDD
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236251210
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -82,6 +82,14 @@ object SQLMetrics
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r236036258
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/23128
@gatorsmile Thanks Xiao! Conflicts resolve done, as Reynold comments in
https://github.com/apache/spark/pull/23105#discussion_r235950427, when the
ShuffleMetricsReporter move
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236032855
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/23128
[SPARK-26139][SQL] Support passing shuffle metrics to exchange operator
## What changes were proposed in this pull request?
Implement `SQLShuffleMetricsReporter` on the sql side
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21363
@MaxGekk Sorry for the late, something inserted in the my scheduler, I plan
to start this PR in this weekend, if its too late please just take it, sorry
for the late again
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22989
Thanks @HyukjinKwon @viirya @felixcheung @srowen for your review and advise!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r233432630
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala ---
@@ -331,7 +333,7 @@ class KMeansSuite extends SparkFunSuite
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r233432568
--- Diff: dev/checkstyle.xml ---
@@ -180,5 +180,10
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22962
@HyukjinKwon No problem, I'll give a follow up PR to address all your
comments and rewrite the UT in to a separate class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r233283645
--- Diff: python/pyspark/taskcontext.py ---
@@ -147,8 +147,8 @@ def __init__(self):
@classmethod
def _getOrCreate(cls
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r233275410
--- Diff: python/pyspark/taskcontext.py ---
@@ -147,8 +147,8 @@ def __init__(self):
@classmethod
def _getOrCreate(cls
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r233055939
--- Diff: python/pyspark/taskcontext.py ---
@@ -147,8 +147,8 @@ def __init__(self):
@classmethod
def _getOrCreate(cls
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r232986340
--- Diff: python/pyspark/tests.py ---
@@ -618,10 +618,13 @@ def test_barrier_with_python_worker_reuse(self):
"""
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22962
Thanks @gatorsmile @HyukjinKwon @cloud-fan !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232984147
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala ---
@@ -283,7 +283,9 @@ class VectorIndexerSuite extends MLTest
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232983941
--- Diff: dev/checkstyle-suppressions.xml ---
@@ -46,4 +46,12 @@
files="sql/catalyst/src/main/java/org/apache/spark/sql/stre
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232955017
--- Diff: dev/checkstyle-suppressions.xml ---
@@ -46,4 +46,12 @@
files="sql/catalyst/src/main/java/org/apache/spark/sql/stre
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22989
@srowen Great thanks for your guidance, address all your suggestion in
ff234d3 and update the record table in
https://github.com/apache/spark/pull/22989#issuecomment-437939830
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232722383
--- Diff: dev/checkstyle.xml ---
@@ -64,6 +64,11 @@
+
+
--- End diff --
Thanks, done
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232722184
--- Diff: scalastyle-config.xml ---
@@ -240,6 +240,18 @@ This file is divided into 3 sections:
]]>
+
--- End d
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232721829
--- Diff: dev/checkstyle.xml ---
@@ -64,6 +64,11 @@
+
+
+
--- End diff
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22989#discussion_r232721412
--- Diff: scalastyle-config.xml ---
@@ -240,6 +240,18 @@ This file is divided into 3 sections:
]]>
+
+throw
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22989
cc all reviewer, as @srowen's suggestion, add a rule to ban all of new
Error cases.
List currently `throw new XXXError` in Spark source below and record fix up
or exclude for review
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22955
Thanks @mgaido91 @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r232634698
--- Diff: python/pyspark/tests.py ---
@@ -618,10 +618,13 @@ def test_barrier_with_python_worker_reuse(self):
"""
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22989
Sorry for late reply, great thanks for all reviewer's advise, will address
them soon.
---
-
To unsubscribe, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22962
@HyukjinKwon Thanks for your review, comment address and PR
description/title changed done.
---
-
To unsubscribe, e-mail
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r232528333
--- Diff: python/pyspark/taskcontext.py ---
@@ -144,10 +144,19 @@ def __init__(self):
"""Construct a BarrierTaskContext,
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r232527808
--- Diff: python/pyspark/tests.py ---
@@ -614,6 +614,18 @@ def context_barrier(x):
times =
rdd.barrier().mapPartitions(f).map
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232489060
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -0,0 +1,171
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232488956
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -50,20 +50,11
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21363
@HyukjinKwon Great thanks for ping me, I'll try to work on this and cc all
reviewer in this PR.
---
-
To unsubscribe, e
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22989
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232163956
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -0,0 +1,128
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232163715
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -0,0 +1,128
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232163787
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -0,0 +1,128
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22955#discussion_r232163738
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
---
@@ -0,0 +1,128
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22989
[SPARK-25986][Build] Banning throw new OutOfMemoryErrors
## What changes were proposed in this pull request?
Add scala and java lint check rules to ban the usage of `throw new
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22955
Thanks for the reply, unnecessary end-to-end tests removed in
https://github.com/apache/spark/pull/22326/commits/2b6977de4a3b3489b9c2172a6a8a39831bf1d048,
others maybe should be kept? Cause
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22955
cc @cloud-fan @mgaido91
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22962
cc @cloud-fan @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22962
[SPARK-25921][PySpark] Fix BarrierTaskContext while python worker reuse
## What changes were proposed in this pull request?
While python worker reuse, BarrierTaskContext._getOrCreate
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22165
gental ping @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/22165
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/22955
[SPARK-25949][SQL] Add test for PullOutPythonUDFInJoinCondition
## What changes were proposed in this pull request?
As comment in
https://github.com/apache/spark/pull/22326
1 - 100 of 777 matches
Mail list logo
