WuZeyi created MAPREDUCE-7223:
---------------------------------
Summary: Quicksort GetInt performance Issue in Terasort
Key: MAPREDUCE-7223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7223
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 3.1.2, 2.7.6
Reporter: WuZeyi
Fix For: 2.7.6
I find a hot spot of 'java.nio.Bits.getIntL' in the Terasort case of Hadoop.
It is done by shifting four bytes in the byte array each time to get an int.
This 'getIntL' operation is repeatedly called in the quick-sort of KVbuffer
which has a complexity of NlogN, and it causes the hot spot.
The element that is gotten in the quick-sort may be gotten frequently, which
means it has to be shifted again and again.
After replacing 'java.nio.Bits.getIntL' with 'unsafe.getInt', the performance
of quick-sort can be improved by 30%。Terasort can be improved by 10%
!terasort.png!!makeint.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
