[
https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072317#comment-16072317
]
Chetan Mehrotra commented on OAK-5192:
--------------------------------------
{quote}
Attached binSize100.txt, binSize16384.txt, binSizeTotal.txt and
nonBinSizeTotal.txt with the sizes for binaries <= 100, <=16384 and <=
Long.MaxValue and the sizes for non binaries per revision in diff.txt.zip,
respectively.
Total sum of sizes are: 44785 bytes, 5887029, bytes, 237895837 bytes and 245128
bytes listed in the same order as above
{quote}
Had a look at stats again.
{noformat}
$ grep ',' binSize16384.txt | cut -f 1 -d L | awk '$1<4000' | awk '{n += $1};
END{print n}'
380980
{noformat}
So checking the size of blobs < 4kb (typical inline size) via following command
shows that actual index size in segment nodestore is 380KB.
So from number it appears that impact of Lucene growth on SegmentNodeStore with
a suggested setup involving maxInlineSize of no more than 4kb should not be
very high. The earlier analysis was done on a setup where inline size was set
to 16KB which caused a higher impact.
We should still check out ways to reduce the growth in DataStore side but from
SegmentNodeStore we should be ok with current setup
/cc [~teofili] [~mduerig] [~tmueller]
> Reduce Lucene related growth of repository size
> -----------------------------------------------
>
> Key: OAK-5192
> URL: https://issues.apache.org/jira/browse/OAK-5192
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, segment-tar
> Reporter: Michael Dürig
> Assignee: Chetan Mehrotra
> Labels: perfomance, scalability
> Fix For: 1.8, 1.7.8
>
> Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt,
> binSizeTotal.txt, diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch
>
>
> I observed Lucene indexing contributing to up to 99% of repository growth.
> While the size of the index itself is well inside reasonable bounds, the
> overall turnover of data being written and removed again can be as much as
> 99%.
> In the case of the TarMK this negatively impacts overall system performance
> due to fast growing number of tar files / segments, bad locality of
> reference, cache misses/thrashing when looking up segments and vastly
> prolonged garbage collection cycles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
