Re: [SegmentStore] Blobs under 16 KB always inlined in tar files?

2018-02-21 Thread Michael Dürig
On 15.02.18 22:06, Alexander Klimetschek wrote: I would agree on first sight. However, there might be good reasons for the current design and these concerns would not be true in practice. The same setting is essentially used for both STRING and BINARY properties - maybe it makes a lot of se

Re: [SegmentStore] Blobs under 16 KB always inlined in tar files?

2018-02-16 Thread Alexander Klimetschek
Thanks! The real world example of " In one setup out of 370 GB segmentstore size 290GB is due to inlined binary" shows that Ian's hunch was pretty spot on. It's clear we don't want these binaries waste memory being memory mapped. Same would apply to longer strings IMO, say rich text html or mar

Re: [SegmentStore] Blobs under 16 KB always inlined in tar files?

2018-02-15 Thread Chetan Mehrotra
See OAK-6911 Chetan Mehrotra On Fri, Feb 16, 2018 at 2:36 AM, Alexander Klimetschek wrote: > Hi, > > it seems the segment store will inline any binary blob up to ~16KB in the tar > files and not store them in the BlobStore [1]. The 16 KB limit > (Segment.MEDIUM_LIMIT) is hardcoded and not conf

[SegmentStore] Blobs under 16 KB always inlined in tar files?

2018-02-15 Thread Alexander Klimetschek
Hi, it seems the segment store will inline any binary blob up to ~16KB in the tar files and not store them in the BlobStore [1]. The 16 KB limit (Segment.MEDIUM_LIMIT) is hardcoded and not configurable. I can see this in action when debugging and when looking at an S3 datastore of a full Oak s