[
https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002715#comment-16002715
]
Michael Dürig commented on OAK-5192:
------------------------------------
The script for generating above diff:
{code}
import $ivy.`michid:script-oak:latest.integration`
import michid.script.oak._
import michid.script.oak.nodestore.Projection
import org.apache.jackrabbit.oak.api.{PropertyState, Type}
import org.apache.jackrabbit.oak.segment.SegmentBlob
import michid.script.oak.nodestore.Changes._
import michid.script.oak.nodestore.ItemStates
implicit val fs = fileStoreAnalyser(builder = dummyBlobStoreBuilder)
val changes = fs.changes(Projection("root/oak:index/lucene")).
map{case(diffs, date) => (date.getTime, diffs)}.
filter{ case(_, diffs) => !diffs.isEmpty }
def blobId(p: PropertyState) = p.analyse.
values.filter(_.tyqe == Type.BINARY).
map(_.value.asInstanceOf[SegmentBlob].getBlobId).
toList
val diff = changes.map{ case(timestamp, diffs) =>
(timestamp, diffs.map {
case PropertyAdded(path, after) =>
(PropertyAdded(path, after), blobId(after),
ItemStates.propertySize(false)(after))
case PropertyRemoved(path, before) =>
(PropertyRemoved(path, before), blobId(before),
ItemStates.propertySize(false)(before))
case x => x
})}
browse(diff, viewer="vim", colors = pprint.Colors.BlackWhite)
{code}
> Reduce Lucene related growth of repository size
> -----------------------------------------------
>
> Key: OAK-5192
> URL: https://issues.apache.org/jira/browse/OAK-5192
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, segment-tar
> Reporter: Michael Dürig
> Assignee: Tommaso Teofili
> Labels: perfomance, scalability
> Fix For: 1.8, 1.7.3
>
> Attachments: added-bytes-zoom.png, diff.txt.zip
>
>
> I observed Lucene indexing contributing to up to 99% of repository growth.
> While the size of the index itself is well inside reasonable bounds, the
> overall turnover of data being written and removed again can be as much as
> 99%.
> In the case of the TarMK this negatively impacts overall system performance
> due to fast growing number of tar files / segments, bad locality of
> reference, cache misses/thrashing when looking up segments and vastly
> prolonged garbage collection cycles.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
