[
https://issues.apache.org/jira/browse/MAPREDUCE-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WuZeyi updated MAPREDUCE-7223:
------------------------------
Description:
I find a hot spot of 'java.nio.Bits.getIntL' in the Terasort case of Hadoop.
It is done by shifting four bytes in the byte array each time to get an int.
This 'getIntL' operation is repeatedly called in the quick-sort of KVbuffer
which has a complexity of NlogN, and it causes the hot spot.
The element that is gotten in the quick-sort may be gotten frequently, which
means it has to be shifted again and again.
After replacing 'java.nio.Bits.getIntL' with 'unsafe.getInt', the performance
of quick-sort can be improved by 30%。Terasort can be improved by 10%
Quick-sort performance: The time of quick-sort using unsafe is 16515s,and using
byteBuffer is 21643s.The performance can improve by 31%
unsafe(s) byteBuffer(s) byteBuffer/unsafe
AVG 16515 21643 1.310481735
was:
I find a hot spot of 'java.nio.Bits.getIntL' in the Terasort case of Hadoop.
It is done by shifting four bytes in the byte array each time to get an int.
This 'getIntL' operation is repeatedly called in the quick-sort of KVbuffer
which has a complexity of NlogN, and it causes the hot spot.
The element that is gotten in the quick-sort may be gotten frequently, which
means it has to be shifted again and again.
After replacing 'java.nio.Bits.getIntL' with 'unsafe.getInt', the performance
of quick-sort can be improved by 30%。Terasort can be improved by 10%
!terasort.png!!makeint.png!
> Quicksort GetInt performance Issue in Terasort
> ----------------------------------------------
>
> Key: MAPREDUCE-7223
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7223
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 2.7.6, 3.1.2
> Reporter: WuZeyi
> Priority: Major
> Labels: patch, performance
> Fix For: 2.7.6
>
>
> I find a hot spot of 'java.nio.Bits.getIntL' in the Terasort case of Hadoop.
> It is done by shifting four bytes in the byte array each time to get an int.
> This 'getIntL' operation is repeatedly called in the quick-sort of KVbuffer
> which has a complexity of NlogN, and it causes the hot spot.
> The element that is gotten in the quick-sort may be gotten frequently, which
> means it has to be shifted again and again.
> After replacing 'java.nio.Bits.getIntL' with 'unsafe.getInt', the
> performance of quick-sort can be improved by 30%。Terasort can be improved by
> 10%
>
> Quick-sort performance: The time of quick-sort using unsafe is 16515s,and
> using byteBuffer is 21643s.The performance can improve by 31%
> unsafe(s) byteBuffer(s) byteBuffer/unsafe
> AVG 16515 21643 1.310481735
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
