Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16350
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user xuanyuanking closed the pull request at:
https://github.com/apache/spark/pull/16350
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592806
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -97,7 +99,15 @@ object FileSourceStrategy
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592805
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -97,7 +99,15 @@ object FileSourceStrategy
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592818
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -126,4 +136,52 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592816
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -126,4 +136,52 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592821
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -126,4 +136,52 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592865
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -126,4 +136,52 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592876
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -212,6 +212,11 @@ object SQLConf {
.booleanConf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592883
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -661,6 +666,8 @@ private[sql] class SQLConf extends Serializable
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r84592888
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
---
@@ -571,6 +571,37 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r85656729
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -126,4 +136,52 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/14957#discussion_r85656841
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
---
@@ -442,6 +443,79 @@ class
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman Thanks for let me know. I'll try your patch and check #14957 take
over or not.
I also think we need getting feedback from @liancheng , from our last
discussion, liancheng m
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/16135
SPARK-18700: add ReadWriteLock for each table's relation in cache
## What changes were proposed in this pull request?
As the scenario describe in
[SPARK-18700][
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
@rxin @liancheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
@ericl Thanks for your review.
> Is it sufficient to lock around the catalog.filterPartitions(Nil)?
Yes, this patch port from 1.6.2 and I missed the diff here. Fixed in next
pa
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
hi @ericl
This commit do the 3 things below, thanks for your check:
1. Delete the unnecessary lock use and simplify the lock operation
2. Add UT test in
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849544
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -33,6 +35,7 @@ import
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849543
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -97,6 +97,12 @@ object HiveCatalogMetrics extends Source
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849557
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -53,6 +56,18 @@ private[hive] class HiveMetastoreCatalog
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849565
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala
---
@@ -352,4 +353,28 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849561
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -53,6 +56,18 @@ private[hive] class HiveMetastoreCatalog
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849563
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -53,6 +56,18 @@ private[hive] class HiveMetastoreCatalog
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91849598
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala
---
@@ -352,4 +353,28 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91879183
--- Diff:
core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala ---
@@ -105,6 +111,7 @@ object HiveCatalogMetrics extends Source
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91904834
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -53,6 +53,18 @@ private[hive] class HiveMetastoreCatalog
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91904915
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala
---
@@ -352,4 +353,34 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/16135#discussion_r91904868
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -209,72 +221,79 @@ private[hive] class
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
Thanks for ericl's review!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
en
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
cc @rxin thanks for check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/16350
[SPARK-18700][SQL][BACKPORT-2.0] Add StripedLock for each table's relation
in cache
## What changes were proposed in this pull request?
Backport of #16135 to branc
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16135
@hvanhovell Sure, I open a new BACKPORT-2.0.
There's a little diff in branch-2.0, the ut test of this patch based on the
`HiveCatalogMetrics` which not added in 2.0, so I added the
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/16350
Delete the UT and metrics done. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/20150
[SPARK-22956][SS] Bug fix for 2 streams union failover scenario
## What changes were proposed in this pull request?
This problem reported by @yanlin-Lynn @ivoson and @LiangchangZ
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20150
cc @zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20150
cc @gatorsmile @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20150
Hi Shixiong, thanks a lot for your reply.
The full stack below can reproduce by running the added UT based on
original code base.
```
Assert on query failed: : Query [id = 3421db21
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20244
reopen this...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20244#discussion_r161144809
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2417,93 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20244#discussion_r161141499
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -96,6 +98,22 @@ class MyRDD(
override def toString
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20244#discussion_r161141879
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2417,93 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20244
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20244
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20244
@ivoson Tengfei, please post the full stack trace of the
`ClassCastException`.
---
-
To unsubscribe, e-mail: reviews
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20150#discussion_r161426632
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -318,6 +318,84 @@ class KafkaSourceSuite
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20150#discussion_r161426641
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -122,6 +122,11 @@ case class MemoryStream[A : Encoder
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20150#discussion_r161426622
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -318,6 +318,84 @@ class KafkaSourceSuite
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20150
Thanks for your review! Shixiong
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r183198368
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1266,6 +1266,9 @@ class DAGScheduler
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20930
@Ngone51
You can check the screenshot in detail, stage 2's shuffleID is 1, but stage
3 failed by missing an output for shuffle '0'! So here the stage 2's skip cause
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20930
@Ngone51 Ah, maybe I know how the description misleading you, the in the
description 5, 'this stage' refers to 'Stage 2' in screenshot, thanks for your
check, I modifie
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20930
Stage 0\1\2\3 same with 20\21\22\23 in this screenshot, stage2's shuf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21116#discussion_r183224838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala
---
@@ -0,0 +1,126
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20946#discussion_r183447988
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
---
@@ -125,6 +125,19 @@ class OffsetSeqLogSuite
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20946#discussion_r183447816
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
---
@@ -39,7 +39,9 @@ case class OffsetSeq(offsets: Seq
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21136#discussion_r183604217
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala
---
@@ -771,7 +778,16 @@ class
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21136
+1 for this.
We find this by CP app use filter with functions, this can be supported by
current implement.
cc @jose-torres @zsxwing @tdas
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r183770468
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -258,14 +258,8 @@ private[spark] object AccumulatorContext
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r183772627
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -209,10 +209,8 @@ class AccumulatorSuite extends SparkFunSuite with
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r184109204
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1266,6 +1266,9 @@ class DAGScheduler
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r184260210
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2399,84 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r184260597
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2399,84 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r184274946
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2399,84 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r184276403
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2399,84 @@ class DAGSchedulerSuite extends
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/20930
> Have you applied this patch: #17955 ?
No, this happened on Spark 2.1. Thanks xingbo & wenchen, I'll port back
this patch to our internal Spark 2.1.
> That
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21177#discussion_r184724132
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmark.scala
---
@@ -87,10 +90,20 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21177#discussion_r184725980
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmark.scala
---
@@ -78,7 +81,7 @@ object TPCDSQueryBenchmark
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184882396
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -20,12 +20,12 @@ package org.apache.spark.io
import
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184882338
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -20,12 +20,12 @@ package org.apache.spark.io
import
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21194#discussion_r185252544
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -101,25 +101,10 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21194#discussion_r185252360
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -101,25 +101,10 @@ object
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21194#discussion_r185851172
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProviderSuite.scala
---
@@ -173,55 +173,154 @@ class
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21188#discussion_r185852663
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamProvider.scala
---
@@ -107,14 +107,25 @@ object
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21188
@maasg as comment in #21194, I just consider we should not change the
behavior while `seconds > rampUpTimeSeconds`. Maybe it more important than
smo
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21199#discussion_r186765402
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala
---
@@ -0,0 +1,304
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21199#discussion_r186764630
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala
---
@@ -0,0 +1,304
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/21293
[SPARK-24237][SS] Continuous shuffle dependency and map output tracker
## What changes were proposed in this pull request?
As our disscussion in [jira
comment](https
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21293
cc @jose-torres @zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187596748
--- Diff: core/src/main/scala/org/apache/spark/Dependency.scala ---
@@ -65,15 +65,17 @@ abstract class NarrowDependency[T](_rdd: RDD[T])
extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187597922
--- Diff: core/src/main/scala/org/apache/spark/Dependency.scala ---
@@ -88,14 +90,53 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187598100
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -233,6 +239,28 @@ private[spark] class MapOutputTrackerMasterEndpoint
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187598365
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -227,6 +228,7 @@ object SparkEnv extends Logging
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187598787
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ContinuousShuffleMapTask.scala
---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r187599741
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -769,6 +796,43 @@ private[spark] class MapOutputTrackerWorker(conf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r187763285
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -209,10 +209,8 @@ class AccumulatorSuite extends SparkFunSuite with
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r187763308
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r187823469
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21114
cc @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21114#discussion_r188152980
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r188269208
--- Diff: core/src/main/scala/org/apache/spark/Dependency.scala ---
@@ -65,15 +65,17 @@ abstract class NarrowDependency[T](_rdd: RDD[T])
extends
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r188270290
--- Diff: core/src/main/scala/org/apache/spark/Dependency.scala ---
@@ -88,14 +90,53 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r188273722
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -769,6 +796,43 @@ private[spark] class MapOutputTrackerWorker(conf
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21293#discussion_r188277683
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ContinuousShuffleMapTask.scala
---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21293
@jose-torres Great thanks for you advise and guidance for us! I found the
main difference between us is whether we can reuse current implementation of
scheduler and shuffle. I marked in your
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/21332
[SPARK-24236][SS] Continuous replacement for ShuffleExchangeExec
## What changes were proposed in this pull request?
1. New RDD named ContinuousShuffleRowRDD
2. New case class
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21332
cc @jose-torres
As we discussion in #21293, the main difference between us is whether we
can reuse current implementation of scheduler and shuffle, but in this part
about the
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188601016
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/UnsafeRowReceiver.scala
---
@@ -0,0 +1,56
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21337#discussion_r188604001
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/ContinuousShuffleReadRDD.scala
---
@@ -0,0 +1,64
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21114
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
1 - 100 of 777 matches
Mail list logo
