[
https://issues.apache.org/jira/browse/HBASE-21551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21551:
-----------------------------
Description:
We open the RegionServerScanner with STREAM as following:
{code}
RegionScannerImpl#initializeScanners
|---> HStore#getScanner
|----------> StoreScanner()
|------->
StoreFileScanner#getScannersForStoreFiles
|------>
HStoreFile#getStreamScanner #1
{code}
In #1, we put the StoreFileReader into a concurrent hash map streamReaders,
but not remove the StreamReader from streamReaders until closing the store
file.
So if we scan with stream with so many times, the streamReaders hash map will
be exploded. we can see the heap dump in the attached heap-dump.jpg.
I found this bug, because when i benchmark the scan performance by using YCSB
in a cluster (heap size of RS is 50g), the Rs will easy to happen a long time
full gc ( ~ 110 sec)....
was:
We open the RegionServerScanner with STREAM as following:
{code}
RegionScannerImpl#initializeScanners
|---> HStore#getScanner
|----------> StoreScanner()
|------->
StoreFileScanner#getScannersForStoreFiles
|------>
HStoreFile#getStreamScanner #1
{code}
In #1, we put the StoreFileReader into a concurrent hash map streamReaders,
but not remove the StreamReader from streamReaders until closing the store
file.
So if we scan with stream with so many times, the streamReaders hash map will
be exploded. we can see the heap dump in the attached heap-dump.jpg.
> Memory leak when use scan with STREAM at server side
> ----------------------------------------------------
>
> Key: HBASE-21551
> URL: https://issues.apache.org/jira/browse/HBASE-21551
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Blocker
> Fix For: 3.0.0, 2.2.0, 2.1.2, 2.0.4
>
> Attachments: heap-dump.jpg
>
>
> We open the RegionServerScanner with STREAM as following:
> {code}
> RegionScannerImpl#initializeScanners
> |---> HStore#getScanner
> |----------> StoreScanner()
> |------->
> StoreFileScanner#getScannersForStoreFiles
> |------>
> HStoreFile#getStreamScanner #1
> {code}
> In #1, we put the StoreFileReader into a concurrent hash map streamReaders,
> but not remove the StreamReader from streamReaders until closing the store
> file.
> So if we scan with stream with so many times, the streamReaders hash map
> will be exploded. we can see the heap dump in the attached heap-dump.jpg.
> I found this bug, because when i benchmark the scan performance by using YCSB
> in a cluster (heap size of RS is 50g), the Rs will easy to happen a long
> time full gc ( ~ 110 sec)....
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
