[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16032:
Fix Version/s: (was: 1.3.1)
1.3.0
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch, HBASE-16032_v4.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16032:
Fix Version/s: (was: 1.2.3)
1.2.2
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.2, 1.1.6, 1.3.1, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch, HBASE-16032_v4.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
No problem, thanks for the confirmation [~enis]
Pushed change into all branches (master, branch-1, branch-1.1, branch-1.2,
branch-1.3 and 0.98). Closing issue.
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.1.6, 1.3.1, 0.98.21, 1.2.3
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch, HBASE-16032_v4.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Attachment: HBASE-16032_v4.patch
Update patch resolving the issue Enis pointed out, rollback to use the
try-catch way in StoreScanner constructor to avoid missing the StoreFile update
in resetKVHeap when flush/bulkload happens
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.1.6, 1.3.1, 0.98.21, 1.2.3
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch, HBASE-16032_v4.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16032:
Fix Version/s: (was: 1.2.2)
1.2.3
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.1.6, 1.3.1, 0.98.21, 1.2.3
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Attachment: HBASE-16032_v3.patch
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.2, 1.1.6, 1.3.1, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch,
> HBASE-16032_v3.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16032:
Fix Version/s: (was: 1.3.0)
1.3.1
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.2, 1.1.6, 1.3.1, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in {{HRegion#get}}
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will
> cause scanner==null then memory leak, so we also need to handle this part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Description:
We observed frequent fullGC of RS in our production environment, and after
analyzing the heapdump, we found large memory occupancy by
HStore#changedReaderObservers, the map is surprisingly containing 7500w
objects...
After some debugging, I located some possible memory leak in StoreScanner
constructor:
{code}
public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
NavigableSet columns,
long readPt)
throws IOException {
this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
if (columns != null && scan.isRaw()) {
throw new DoNotRetryIOException("Cannot specify any column for a raw
scan");
}
matcher = new ScanQueryMatcher(scan, scanInfo, columns,
ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
oldestUnexpiredTS, now, store.getCoprocessorHost());
this.store.addChangedReaderObserver(this);
// Pass columns to try to filter out unnecessary StoreFiles.
List scanners = getScannersNoCompaction();
...
seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
&& lazySeekEnabledGlobally, parallelSeekEnabled);
...
resetKVHeap(scanners, store.getComparator());
}
{code}
If there's any Exception thrown after
{{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
null and there's no chance to remove the scanner from changedReaderObservers,
like in {{HRegion#get}}
{code}
RegionScanner scanner = null;
try {
scanner = getScanner(scan);
scanner.next(results);
} finally {
if (scanner != null)
scanner.close();
}
{code}
What's more, all exception thrown in the {{HRegion#getScanner}} path will cause
scanner==null then memory leak, so we also need to handle this part.
was:
We observed frequent fullGC of RS in our production environment, and after
analyzing the heapdump, we found large memory occupancy by
HStore#changedReaderObservers, the map is surprisingly containing 7500w
objects...
After some debugging, I located some possible memory leak in StoreScanner
constructor:
{code}
public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
NavigableSet columns,
long readPt)
throws IOException {
this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
if (columns != null && scan.isRaw()) {
throw new DoNotRetryIOException("Cannot specify any column for a raw
scan");
}
matcher = new ScanQueryMatcher(scan, scanInfo, columns,
ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
oldestUnexpiredTS, now, store.getCoprocessorHost());
this.store.addChangedReaderObserver(this);
// Pass columns to try to filter out unnecessary StoreFiles.
List scanners = getScannersNoCompaction();
...
seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
&& lazySeekEnabledGlobally, parallelSeekEnabled);
...
resetKVHeap(scanners, store.getComparator());
}
{code}
If there's any Exception thrown after
{{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
null and there's no chance to remove the scanner from changedReaderObservers,
like in HRegion#get
{code}
RegionScanner scanner = null;
try {
scanner = getScanner(scan);
scanner.next(results);
} finally {
if (scanner != null)
scanner.close();
}
{code}
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now,
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Attachment: HBASE-16032_v2.patch
Update the patch, covering missed exception-thrown handling on the
{{HRegion#getScanner}} path
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.1.5, 0.98.20
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6, 0.98.21
>
> Attachments: HBASE-16032.patch, HBASE-16032_v2.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in HRegion#get
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Fix Version/s: 0.98.21
1.1.6
1.2.2
1.3.0
2.0.0
Affects Version/s: 1.2.1
1.1.5
0.98.20
Status: Patch Available (was: Open)
Submit patch for HadoopQA
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.20, 1.1.5, 1.2.1
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6, 0.98.21
>
> Attachments: HBASE-16032.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in HRegion#get
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16032:
--
Attachment: HBASE-16032.patch
To solve the problem, we should catch exceptions after
{{addChangedReaderObserver}} and remove the StoreScanner from
{{changedReaderObservers}} if there's any.
Uploading a straight-forward patch following this design.
> Possible memory leak in StoreScanner
>
>
> Key: HBASE-16032
> URL: https://issues.apache.org/jira/browse/HBASE-16032
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-16032.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after
> analyzing the heapdump, we found large memory occupancy by
> HStore#changedReaderObservers, the map is surprisingly containing 7500w
> objects...
> After some debugging, I located some possible memory leak in StoreScanner
> constructor:
> {code}
> public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final
> NavigableSet columns,
> long readPt)
> throws IOException {
> this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
> if (columns != null && scan.isRaw()) {
> throw new DoNotRetryIOException("Cannot specify any column for a raw
> scan");
> }
> matcher = new ScanQueryMatcher(scan, scanInfo, columns,
> ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
> oldestUnexpiredTS, now, store.getCoprocessorHost());
> this.store.addChangedReaderObserver(this);
> // Pass columns to try to filter out unnecessary StoreFiles.
> List scanners = getScannersNoCompaction();
> ...
> seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
> && lazySeekEnabledGlobally, parallelSeekEnabled);
> ...
> resetKVHeap(scanners, store.getComparator());
> }
> {code}
> If there's any Exception thrown after
> {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be
> null and there's no chance to remove the scanner from changedReaderObservers,
> like in HRegion#get
> {code}
> RegionScanner scanner = null;
> try {
> scanner = getScanner(scan);
> scanner.next(results);
> } finally {
> if (scanner != null)
> scanner.close();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
