[ 
https://issues.apache.org/jira/browse/OAK-7052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289166#comment-16289166
 ] 

Vikas Saurabh commented on OAK-7052:
------------------------------------

Just to illustrate: here's some data from a test setup where this issue led to 
OOM during purge of deleted index blobs:
{noformat}
repository/index/deleted-blobs$ wc -l blobs-151*
     418397 blobs-1512664032264.txt
     627148 blobs-1512698405656.txt
    1121837 blobs-1512987450004.txt
     308292 blobs-1513130410963.txt
    2475674 total

repository/index/deleted-blobs$ ls -l blobs-151*
-rw-r--r-- 1 root root 3369065620 Dec  8 01:59 blobs-1512664032264.txt
-rw-r--r-- 1 root root 4532250073 Dec  9 01:59 blobs-1512698405656.txt
-rw-r--r-- 1 root root 6370201955 Dec 13 01:59 blobs-1512987450004.txt
-rw-r--r-- 1 root root 1916223582 Dec 13 11:52 blobs-1513130410963.txt
{noformat}


||File|| file size || num_entries || size/num_entries ||
|blobs-1512664032264.txt|3369065620|418397|8052.31782254653|
|blobs-1512698405656.txt|4532250073|627148|7226.7631771129|
|blobs-1512987450004.txt|6370201955|1121837|5678.36678144864|
|blobs-1513130410963.txt|1916223582|308292|6215.61241290724|

While the patch above would fix OOM, I think there's another interesting thing 
- there seem to be a LOT of inlined blobs. For active deletion, that means that 
these blob id containing files would essentially be storing all deleted inlined 
blobs.

[~chetanm], [~tmueller], [~amjain], maybe, active deletion logic should ignore 
deletion of very small blobs!? (should we tie it with blob store's inline 
threshold.... I don't feel that way though)

> Active deletion purge can OOM if number of blobs listed in a file become too 
> large
> ----------------------------------------------------------------------------------
>
>                 Key: OAK-7052
>                 URL: https://issues.apache.org/jira/browse/OAK-7052
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: lucene
>            Reporter: Vikas Saurabh
>            Assignee: Vikas Saurabh
>             Fix For: 1.8
>
>
> Actively deletion, while purging, reads blob ids from a file using 
> {{FileUtils#readLines}}. That method returns a list, while we only require to 
> iterate over the file contents line-by-line.
> We should probably switch to {{FileUtils#lineIterator}} to clearly declare 
> our intent to read line by line.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to