caican00 commented on PR #37876:
URL: https://github.com/apache/spark/pull/37876#issuecomment-1283585465
> > > I haven't thought of a better way yet
> >
> >
> > Thanks. I'll share it with you if I can think of a better way
>
> @LuciferYang I have a question , why `ArrayBasedMapData#toScalaMap` method
return immutable.Map type rather than mutable.Map type? I write a simple case
to compare `building mutable.Map` and `building immutable.Map` and `building
mutable.Map` is about twice as fast as `building immutable.Map`
>
> test code:
>
> ```
> case class Test(id: Int, name: String)
>
> // build immutable.Map
> val builder = immutable.Map.newBuilder[Test, String]
> var index: Int = 0
> val start = System.currentTimeMillis()
> while (index < 10000000) {
> builder += (Test(index, s"$index _ $index"),
index.toString).asInstanceOf[(Test, String)]
> index += 1
> }
> builder.result()
> val end = System.currentTimeMillis()
> // scalastyle:off println
> println(s"immutable map spent time: ${(end - start)/1000}s")
> // scalastyle:off println
>
>
> // build mutable.Map
> val map = mutable.Map[Test, String]()
> index = 0
> val start1 = System.currentTimeMillis()
> while (index < 10000000) {
> map.put(Test(index, s"$index _ $index"), index.toString)
> index += 1
> }
> val end1 = System.currentTimeMillis()
> // scalastyle:off println
> println(s"mutable map spent time: ${(end1 - start1)/1000}s")
> // scalastyle:off println
> ```
>
> test result:
>
> ```
> 5000000 elements:
> immutable map spent time: 8s
> mutable map spent time: 4s
>
> 10000000 elements:
> immutable map spent time: 21s
> mutable map spent time: 9s
> ```
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
