Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21036#discussion_r180650637
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -55,7 +56,8 @@ private[spark] class HadoopPartition(rddId: Int, override
val
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21036#discussion_r180650706
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -86,8 +88,7 @@ private[spark] class HadoopPartition(rddId: Int, override
val
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
Hi, @squito . Thank for review and comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20998#discussion_r180614873
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -880,6 +880,59 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
Hi, @mridulm, thank for your comment. Actually, I have the same worry with
you. May be we can make this change as a second choice for `hasAttemptOnHost `,
in case of there's really no other hosts
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20996#discussion_r180347526
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -370,12 +370,10 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
Hi, @felixcheung , thank for trigger a task and your comments.
> shouldn't this be up to the scheduler backend?
Actually, it is `TaskSchedulerImpl` who holds a thread to check whet
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
ping @pwendell @kayousterhout . pls help review, thanks :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20998
[SPARK-23888][CORE] speculative task should not run on a given host where
another attempt is already running on
## What changes were proposed in this pull request?
There's a bug
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20987
Yeahï¼agree.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20996
[SPARK-23884][CORE] hasLaunchedTask should be true when launchedAnyTask be
true
## What changes were proposed in this pull request?
`hasLaunchedTask` should be `true` when
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20987
Things I'm concerned about is that does there exists a situation like 'a
task gets killed after it gets a FetchFailure, but re-run (not by resubmit)
later and gets a FetchFailure too without
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20987#discussion_r179647613
--- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala
---
@@ -173,8 +173,26 @@ class ExecutorSuite extends SparkFunSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20987#discussion_r179647240
--- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala
---
@@ -330,6 +362,15 @@ class FetchFailureHidingRDD(
case t
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20977#discussion_r179490408
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ---
@@ -166,7 +166,7 @@ private class AsyncEventQueue(val name: String, conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20977#discussion_r179338562
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ---
@@ -166,7 +166,7 @@ private class AsyncEventQueue(val name: String, conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20770#discussion_r173116730
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -140,8 +141,8 @@ class DAGScheduler(
private[scheduler] def
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20685#discussion_r172731294
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -583,8 +587,8 @@ object ShuffleBlockFetcherIterator
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172395871
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172043652
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172037418
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172019362
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172019312
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172018886
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java ---
@@ -17,47 +17,168 @@
package
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172019234
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java
---
@@ -36,22 +42,35 @@ public MemoryBlock allocate(long
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172019050
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172013155
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java
---
@@ -36,22 +42,35 @@ public MemoryBlock allocate(long
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172009747
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java ---
@@ -17,47 +17,168 @@
package
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172009500
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java ---
@@ -45,38 +45,143 @@
*/
public static final int
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172013541
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172010392
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r172010352
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20576#discussion_r171451103
--- Diff:
core/src/test/scala/org/apache/spark/shuffle/sort/SortShuffleManagerSuite.scala
---
@@ -85,6 +85,14 @@ class SortShuffleManagerSuite extends
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20667
Hi, @jiangxb1987 , thanks for your kindly explanation.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20667
Hi, @caneGuy , sorry for my previous comment as I mixed up ```BlockId```
with ```BlockManagerId```, and leave some wrong comments. And thanks for your
reply.
Back to now, I have the same
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r171128021
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20676
Hi, @ConeyLiu. I don't think it's about compatibility. Because both
```StaticMemoryManager``` and ```UnifiedMemoryManager``` call the same function
for release unroll memory, which
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r171124468
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r170956019
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r170954231
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r170961404
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20667#discussion_r170424196
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
---
@@ -132,10 +133,15 @@ private[spark] object BlockManagerId
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163807859
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -702,6 +634,93 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162962203
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -233,17 +235,13 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162931279
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -233,17 +235,13 @@ private[spark] class MemoryStore
Github user Ngone51 closed the pull request at:
https://github.com/apache/spark/pull/20053
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user Ngone51 closed the pull request at:
https://github.com/apache/spark/pull/20056
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162848405
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162841437
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162837169
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -346,85 +348,24 @@ private[spark] class MemoryStore(
} else
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162835021
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162834864
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162834903
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162549759
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -261,37 +263,93 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162548350
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,33 @@ private[spark] class MemoryStore
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162548052
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -233,17 +235,13 @@ private[spark] class MemoryStore
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
@squito yea, clean up will be fine. Maybe, I make things more complicated.
Thank for comment.
---
-
To unsubscribe, e-mail
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
@squito by now, the variable `droppedEventsCounter ` is unused. So, I want
to reuse it.
---
-
To unsubscribe, e-mail: reviews
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160709123
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -842,6 +842,7 @@ class VersionsSuite extends SparkFunSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20199
Ok, will try.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160404821
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20199
cc @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160344054
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20199
[Spark-22967][Hive]Fix VersionSuite's unit tests by change Windows path
into URI path
## What changes were proposed in this pull request?
Two unit test will fail due to Windows format
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
@vanzin @squito Can you have a look at this pr ? Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r159081427
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r159076507
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
@maropu Actually, I didn't modify this unit test ever. And my unit test
locate in SparkListenerSuite haven'been started according to the "Console
O
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
It seems something wrong while testing 'ReplayListenerSuite':
```
[info] ReplayListenerSuite:
[info] - Simple replay (97 milliseconds)
[info] - Replay compressed inprogress log file
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20056#discussion_r158751285
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -25,14 +25,13 @@ import scala.collection.JavaConverters._
import
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
please retest it. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
@gatorsmile @HyukjinKwon @dongjoon-hyun Can any one of you have a look at
this pr, Thanks!
---
-
To unsubscribe, e-mail
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20056
cc: @maropu
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r158574199
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -125,13 +128,39 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r158573853
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r158573360
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r158572288
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20053#discussion_r158572161
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala ---
@@ -112,6 +112,7 @@ private class AsyncEventQueue(val name: String, conf
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20056
[SPARK-22878] [CORE] Count totalDroppedEvents for LiveListenerBus
## What changes were proposed in this pull request?
Count total dropped events from all queues' numDroppedEvents
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20053
Init lastReportTimestamp with system current time when start() calledâ¦
⦠in AsyncEventQueue
## What changes were proposed in this pull request
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20039#discussion_r158424923
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20033
[SPARK-22847] [CORE] Remove duplicate code in AppStatusListener while
assigning schedulingPool for stage
## What changes were proposed in this pull request?
In AppStatusListener's
101 - 182 of 182 matches
Mail list logo
