LuciferYang commented on code in PR #43448:
URL: https://github.com/apache/spark/pull/43448#discussion_r1411568249
##########
common/sketch/src/test/scala/org/apache/spark/util/sketch/CountMinSketchSuite.scala:
##########
@@ -56,7 +56,7 @@ class CountMinSketchSuite extends AnyFunSuite { //
scalastyle:ignore funsuite
val exactFreq = {
val sampledItems = sampledItemIndices.map(allItems)
- sampledItems.groupBy(identity).mapValues(_.length.toLong)
+ sampledItems.groupBy(identity).view.mapValues(_.length.toLong)
Review Comment:
@cloud-fan
Thank you very much for pointing this out, it's a great observation.
Let me explain first, in Scala 2.13, changing to use `.transform` would
change the semantics,
```
sampledItems.groupBy(identity).mapValues(_.length.toLong)
```
and
```
sampledItems.groupBy(identity).view.mapValues(_.length.toLong)
```
both return a `MapView`.
While
```
sampledItems.groupBy(identity).transform((_, v) => v.length.toLong)
```
returns a `Map`.
The main difference between `MapView` and `Map` lies in their computation
methods and memory usage:
- `MapView` is a kind of lazy computation collection view, it does not
immediately calculate the result, but calculates when needed. This means that
if you create a `MapView` but don't actually use it, the related calculations
will not occur. This can improve performance when dealing with large data sets,
as it can avoid unnecessary calculations. However, each time `MapView` is
accessed, the calculation will be performed again, which may lead to
performance degradation, especially in cases where multiple accesses are
required.
- On the other hand, `Map` is an immediate computation collection, it
calculates all the results at creation and stores the results in memory. This
means that whether or not you actually use this `Map`, the related calculations
will occur. This may lead to higher memory usage, but in cases where the same
`Map` needs to be accessed multiple times, it can provide better performance
because the results have been calculated and stored in memory, no need to
recalculate each time it is accessed.
I will keep an eye on this case and adjust the cases where `Map` is suitable
for use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
