Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2416#discussion_r17921484
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -66,23 +66,19 @@ class ExternalAppendOnlyMap[K, V, C](
mergeCombiners: (C, C) => C,
serializer: Serializer = SparkEnv.get.serializer,
blockManager: BlockManager = SparkEnv.get.blockManager)
- extends Iterable[(K, C)] with Serializable with Logging {
+ extends Iterable[(K, C)]
+ with Serializable
+ with Logging
+ with Spillable[SizeTrackingAppendOnlyMap[K, C]] {
private var currentMap = new SizeTrackingAppendOnlyMap[K, C]
private val spilledMaps = new ArrayBuffer[DiskMapIterator]
private val sparkConf = SparkEnv.get.conf
private val diskBlockManager = blockManager.diskBlockManager
- private val shuffleMemoryManager = SparkEnv.get.shuffleMemoryManager
// Number of pairs inserted since last spill; note that we count them
even if a value is merged
// with a previous key in case we're doing something like groupBy where
the result grows
- private var elementsRead = 0L
-
- // Number of in-memory pairs inserted before tracking the map's shuffle
memory usage
- private val trackMemoryThreshold = 1000
-
- // How much of the shared memory pool this collection has claimed
- private var myMemoryThreshold = 0L
+ protected[this] var elementsRead = 0L
--- End diff --
What does `protected[this]` mean? It seems that we can just use `protected`
here to keep the signatures simple.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
