[
https://issues.apache.org/jira/browse/OAK-7052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289166#comment-16289166
]
Vikas Saurabh commented on OAK-7052:
------------------------------------
Just to illustrate: here's some data from a test setup where this issue led to
OOM during purge of deleted index blobs:
{noformat}
repository/index/deleted-blobs$ wc -l blobs-151*
418397 blobs-1512664032264.txt
627148 blobs-1512698405656.txt
1121837 blobs-1512987450004.txt
308292 blobs-1513130410963.txt
2475674 total
repository/index/deleted-blobs$ ls -l blobs-151*
-rw-r--r-- 1 root root 3369065620 Dec 8 01:59 blobs-1512664032264.txt
-rw-r--r-- 1 root root 4532250073 Dec 9 01:59 blobs-1512698405656.txt
-rw-r--r-- 1 root root 6370201955 Dec 13 01:59 blobs-1512987450004.txt
-rw-r--r-- 1 root root 1916223582 Dec 13 11:52 blobs-1513130410963.txt
{noformat}
||File|| file size || num_entries || size/num_entries ||
|blobs-1512664032264.txt|3369065620|418397|8052.31782254653|
|blobs-1512698405656.txt|4532250073|627148|7226.7631771129|
|blobs-1512987450004.txt|6370201955|1121837|5678.36678144864|
|blobs-1513130410963.txt|1916223582|308292|6215.61241290724|
While the patch above would fix OOM, I think there's another interesting thing
- there seem to be a LOT of inlined blobs. For active deletion, that means that
these blob id containing files would essentially be storing all deleted inlined
blobs.
[~chetanm], [~tmueller], [~amjain], maybe, active deletion logic should ignore
deletion of very small blobs!? (should we tie it with blob store's inline
threshold.... I don't feel that way though)
> Active deletion purge can OOM if number of blobs listed in a file become too
> large
> ----------------------------------------------------------------------------------
>
> Key: OAK-7052
> URL: https://issues.apache.org/jira/browse/OAK-7052
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
> Reporter: Vikas Saurabh
> Assignee: Vikas Saurabh
> Fix For: 1.8
>
>
> Actively deletion, while purging, reads blob ids from a file using
> {{FileUtils#readLines}}. That method returns a list, while we only require to
> iterate over the file contents line-by-line.
> We should probably switch to {{FileUtils#lineIterator}} to clearly declare
> our intent to read line by line.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)