Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Alex Parvulescu]

Hi,

thanks for the feedback!
I quickly tested the proposed solution (on both emails :) to disable
mmapping for that check and it seems to very nicely alleviate the problem.

Bonus points: oak-segment-tar doesn't seem affected by this (all commands
impls move to the segment-tar module), except a single command that is
related to datastore consistency: 'datastorecheck'.

I'll followup with an oak issue to disable mmaps for the version check.

thanks,
alex




On Thu, Mar 23, 2017 at 4:28 PM, Francesco Mari 
wrote:

> Hi,
>
> 2017-03-23 16:13 GMT+01:00 Alex Parvulescu :
> > Hi,
> >
> > To add what I have found so far. This seems related to OAK-4274, but I
> > think there might be a twist in there somewhere.
> > I'm pretty sure that this method is the one introducing the extra full
> > mapping of the repository: FileStoreHelper.checkFileStoreVersionOrFail
> [0].
> > Disabling this method takes the 2x mapping away completely.
>
> I think your analysis matches with the 2x increase in memory mappings.
> It might be that the memory mappings created while opening the
> FileStore in read-only mode are never garbage collected, even if the
> FileStore is correctly closed when checkFileStoreVersionOrFail()
> returns. To avoid this, we might want to open the FileStore in
> checkFileStoreVersionOrFail() with memory mapping disabled.
>
> > The reason I'm saying it is relate to OAK-4274 is because I looked at a
> > heap dump to verify what is keeping the references to the readonly store
> > and there are no live ones, the refs should be GC'ed, but for some reason
> > they are not.
> >
> > I'm still poking around, did not create an oak issue yet. Still pending
> is
> > to verify if this affects other areas than oak-run.
> >
> > Feedback is more than welcome!
> >
> > best,
> > alex
> >
> > [0]
> > https://github.com/apache/jackrabbit-oak/blob/1.6/oak-
> run/src/main/java/org/apache/jackrabbit/oak/plugins/
> segment/FileStoreHelper.java#L209
>

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Francesco Mari]

Hi,

2017-03-23 16:13 GMT+01:00 Alex Parvulescu :
> Hi,
>
> To add what I have found so far. This seems related to OAK-4274, but I
> think there might be a twist in there somewhere.
> I'm pretty sure that this method is the one introducing the extra full
> mapping of the repository: FileStoreHelper.checkFileStoreVersionOrFail [0].
> Disabling this method takes the 2x mapping away completely.

I think your analysis matches with the 2x increase in memory mappings.
It might be that the memory mappings created while opening the
FileStore in read-only mode are never garbage collected, even if the
FileStore is correctly closed when checkFileStoreVersionOrFail()
returns. To avoid this, we might want to open the FileStore in
checkFileStoreVersionOrFail() with memory mapping disabled.

> The reason I'm saying it is relate to OAK-4274 is because I looked at a
> heap dump to verify what is keeping the references to the readonly store
> and there are no live ones, the refs should be GC'ed, but for some reason
> they are not.
>
> I'm still poking around, did not create an oak issue yet. Still pending is
> to verify if this affects other areas than oak-run.
>
> Feedback is more than welcome!
>
> best,
> alex
>
> [0]
> https://github.com/apache/jackrabbit-oak/blob/1.6/oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/segment/FileStoreHelper.java#L209

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Michael Dürig]

Hi,


I'm pretty sure that this method is the one introducing the extra
full mapping of the repository:
FileStoreHelper.checkFileStoreVersionOrFail


We should probably run this check with memory mapping disabled anyway. 
Nothing to gain here but probably this would fix the double mapping and 
sideline OAK-4274, which effectively is 
http://bugs.java.com/view_bug.do?bug_id=4724038.


Michael


On 23.03.17 16:13, Alex Parvulescu wrote:

Hi,

To add what I have found so far. This seems related to OAK-4274, but
I think there might be a twist in there somewhere. I'm pretty sure
that this method is the one introducing the extra full mapping of the
repository: FileStoreHelper.checkFileStoreVersionOrFail [0].
Disabling this method takes the 2x mapping away completely.

The reason I'm saying it is relate to OAK-4274 is because I looked at
a heap dump to verify what is keeping the references to the readonly
store and there are no live ones, the refs should be GC'ed, but for
some reason they are not.

I'm still poking around, did not create an oak issue yet. Still
pending is to verify if this affects other areas than oak-run.

Feedback is more than welcome!

best, alex

[0]
https://github.com/apache/jackrabbit-oak/blob/1.6/oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/segment/FileStoreHelper.java#L209







On Thu, Mar 23, 2017 at 12:10 PM, Ian Boston  wrote:


Hi, Based on the page fault behaviour, I think the areas mapped and
reported by pmap are being actively accessed by the JVM. The number
of page faults for Oak 1.4.11 is well over 2x the number of page
faults for Oak 1.0.29 on the same VM, with the same DB when
performing an oak-run offline compaction. This is on the same VM
with the same repository in the same state. The Tar files are not
the same, but 1 copy of the tar files is 32GB in both instances,
1.4.11 maps 64GB as mentioned before.

I dont know if its the behaviour seen in OAK-4274. I have seen
similar in the past. I was not confident that a GC cycle did unmap,
but it would be logical. Best Regards Ian

On 23 March 2017 at 09:07, Francesco Mari
 wrote:


You might be hitting OAK-4274, which I discovered quite some time
ago. I'm not aware of a way to resolve this issue at the moment.

2017-03-22 16:47 GMT+01:00 Alex Parvulescu
:

Hi,

To give more background this came about during an investigation
into a

slow

offline compaction but it may affect any running FileStore as
well (to

be

verified). I don't think it's related to oak-run itself, but
more with the way we

map

files, and so far it looks like a bug (there is no reasonable

explanation

for mapping each tar file twice).

Took a quick look at the TarReader but there are not many
changes in

this

area 1.0 vs. 1.4 branches. If no one has better ideas, I'll
create an oak issue and investigate

this a

bit further.

thanks, alex


On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston 
wrote:


Hi, I am looking at Oak-run and I see 2x the mapped memory
between 1.0.29

and

1.4.10. It looks like in 1.0.29 each segment file is mapped
into

memory

once, but in 1.4.10 its mapped into memory 2x.

Is this expected ?

Its not great for page faults. Best Regards Ian

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Alex Parvulescu]

Hi,

To add what I have found so far. This seems related to OAK-4274, but I
think there might be a twist in there somewhere.
I'm pretty sure that this method is the one introducing the extra full
mapping of the repository: FileStoreHelper.checkFileStoreVersionOrFail [0].
Disabling this method takes the 2x mapping away completely.

The reason I'm saying it is relate to OAK-4274 is because I looked at a
heap dump to verify what is keeping the references to the readonly store
and there are no live ones, the refs should be GC'ed, but for some reason
they are not.

I'm still poking around, did not create an oak issue yet. Still pending is
to verify if this affects other areas than oak-run.

Feedback is more than welcome!

best,
alex

[0]
https://github.com/apache/jackrabbit-oak/blob/1.6/oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/segment/FileStoreHelper.java#L209






On Thu, Mar 23, 2017 at 12:10 PM, Ian Boston  wrote:

> Hi,
> Based on the page fault behaviour, I think the areas mapped and reported by
> pmap are being actively accessed by the JVM. The number of page faults for
> Oak 1.4.11 is well over 2x the number of page faults for Oak 1.0.29 on the
> same VM, with the same DB when performing an oak-run offline compaction.
> This is on the same VM with the same repository in the same state. The Tar
> files are not the same, but 1 copy of the tar files is 32GB in both
> instances, 1.4.11 maps 64GB as mentioned before.
>
> I dont know if its the behaviour seen in OAK-4274. I have seen similar in
> the past. I was not confident that a GC cycle did unmap, but it would be
> logical.
> Best Regards
> Ian
>
> On 23 March 2017 at 09:07, Francesco Mari 
> wrote:
>
> > You might be hitting OAK-4274, which I discovered quite some time ago.
> > I'm not aware of a way to resolve this issue at the moment.
> >
> > 2017-03-22 16:47 GMT+01:00 Alex Parvulescu :
> > > Hi,
> > >
> > > To give more background this came about during an investigation into a
> > slow
> > > offline compaction but it may affect any running FileStore as well (to
> be
> > > verified).
> > > I don't think it's related to oak-run itself, but more with the way we
> > map
> > > files, and so far it looks like a bug (there is no reasonable
> explanation
> > > for mapping each tar file twice).
> > >
> > > Took a quick look at the TarReader but there are not many changes in
> this
> > > area 1.0 vs. 1.4 branches.
> > > If no one has better ideas, I'll create an oak issue and investigate
> > this a
> > > bit further.
> > >
> > > thanks,
> > > alex
> > >
> > >
> > > On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston  wrote:
> > >
> > >> Hi,
> > >> I am looking at Oak-run and I see 2x the mapped memory between 1.0.29
> > and
> > >> 1.4.10. It looks like in 1.0.29 each segment file is mapped into
> memory
> > >> once, but in 1.4.10 its mapped into memory 2x.
> > >>
> > >> Is this expected ?
> > >>
> > >> Its not great for page faults.
> > >> Best Regards
> > >> Ian
> > >>
> >
>

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Ian Boston]

Hi,
Based on the page fault behaviour, I think the areas mapped and reported by
pmap are being actively accessed by the JVM. The number of page faults for
Oak 1.4.11 is well over 2x the number of page faults for Oak 1.0.29 on the
same VM, with the same DB when performing an oak-run offline compaction.
This is on the same VM with the same repository in the same state. The Tar
files are not the same, but 1 copy of the tar files is 32GB in both
instances, 1.4.11 maps 64GB as mentioned before.

I dont know if its the behaviour seen in OAK-4274. I have seen similar in
the past. I was not confident that a GC cycle did unmap, but it would be
logical.
Best Regards
Ian

On 23 March 2017 at 09:07, Francesco Mari  wrote:

> You might be hitting OAK-4274, which I discovered quite some time ago.
> I'm not aware of a way to resolve this issue at the moment.
>
> 2017-03-22 16:47 GMT+01:00 Alex Parvulescu :
> > Hi,
> >
> > To give more background this came about during an investigation into a
> slow
> > offline compaction but it may affect any running FileStore as well (to be
> > verified).
> > I don't think it's related to oak-run itself, but more with the way we
> map
> > files, and so far it looks like a bug (there is no reasonable explanation
> > for mapping each tar file twice).
> >
> > Took a quick look at the TarReader but there are not many changes in this
> > area 1.0 vs. 1.4 branches.
> > If no one has better ideas, I'll create an oak issue and investigate
> this a
> > bit further.
> >
> > thanks,
> > alex
> >
> >
> > On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston  wrote:
> >
> >> Hi,
> >> I am looking at Oak-run and I see 2x the mapped memory between 1.0.29
> and
> >> 1.4.10. It looks like in 1.0.29 each segment file is mapped into memory
> >> once, but in 1.4.10 its mapped into memory 2x.
> >>
> >> Is this expected ?
> >>
> >> Its not great for page faults.
> >> Best Regards
> >> Ian
> >>
>

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Francesco Mari]

You might be hitting OAK-4274, which I discovered quite some time ago.
I'm not aware of a way to resolve this issue at the moment.

2017-03-22 16:47 GMT+01:00 Alex Parvulescu :
> Hi,
>
> To give more background this came about during an investigation into a slow
> offline compaction but it may affect any running FileStore as well (to be
> verified).
> I don't think it's related to oak-run itself, but more with the way we map
> files, and so far it looks like a bug (there is no reasonable explanation
> for mapping each tar file twice).
>
> Took a quick look at the TarReader but there are not many changes in this
> area 1.0 vs. 1.4 branches.
> If no one has better ideas, I'll create an oak issue and investigate this a
> bit further.
>
> thanks,
> alex
>
>
> On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston  wrote:
>
>> Hi,
>> I am looking at Oak-run and I see 2x the mapped memory between 1.0.29 and
>> 1.4.10. It looks like in 1.0.29 each segment file is mapped into memory
>> once, but in 1.4.10 its mapped into memory 2x.
>>
>> Is this expected ?
>>
>> Its not great for page faults.
>> Best Regards
>> Ian
>>

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Alex Parvulescu]

Hi,

To give more background this came about during an investigation into a slow
offline compaction but it may affect any running FileStore as well (to be
verified).
I don't think it's related to oak-run itself, but more with the way we map
files, and so far it looks like a bug (there is no reasonable explanation
for mapping each tar file twice).

Took a quick look at the TarReader but there are not many changes in this
area 1.0 vs. 1.4 branches.
If no one has better ideas, I'll create an oak issue and investigate this a
bit further.

thanks,
alex


On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston  wrote:

> Hi,
> I am looking at Oak-run and I see 2x the mapped memory between 1.0.29 and
> 1.4.10. It looks like in 1.0.29 each segment file is mapped into memory
> once, but in 1.4.10 its mapped into memory 2x.
>
> Is this expected ?
>
> Its not great for page faults.
> Best Regards
> Ian
>

Oak 1.0.29 vs 1.4.10 memory mapping.

[Ian Boston]

Hi,
I am looking at Oak-run and I see 2x the mapped memory between 1.0.29 and
1.4.10. It looks like in 1.0.29 each segment file is mapped into memory
once, but in 1.4.10 its mapped into memory 2x.

Is this expected ?

Its not great for page faults.
Best Regards
Ian