Hi,
I'm looking for opinions on OAK-11184 [0] and in particular the current
draft PR#1779 [1] and its suggestion to introduce a new, tiny cache.
What happens in OAK-11184 is when reading a node to which an all-new
property (revision) was just added, and that new property revision is
either (a) not yet committed or (b) not yet visible (for a peer cluster
instance), it causes a complete scan of previous documents. Reason for
this is once it found a property key it eagerly looks for the newest
visible revision. And if that's not in the main document, it needs to
scan all previous documents. That's how it works. For a new property
however, no previous document will have any revision for it yet - hence
that scan ultimately results in finding nothing (or if the property
existed in earlier days, it will have been removed, otherwise the main
document would have two revision snow and we'd not be in this situation).
The case "a) not yet committed" is easy to avoid: It can actually easily
detect such a situation (by checking commit value). And then simply not
scan previous documents, as the only possible case of a not yet
committed revision as the only revision on the man document is exactly
for a new property (hence previous document scan will result in nothing,
hence that's not needed). So a) is easy.
The difficult case is b) - as b) can also happen if reading with a
checkpoint. So in this case it can't statically know that it is a new
property case. So there is no easy way out of this.
So without static detection, the currently known solutions are:
1) introduce a (tiny) cache for a scan of previous documents for a
particular property not finding any revisions whatsoever (not even
invisible, uncommitted ones). That means it will still have to be done
once, unfortunately, and that might still take a certain time - but
after that, any subsequent similar lookup will be avoidable, as the
cache can then be used. (the cache key and values are actually tiny: it
can use just the previous document id and the property key, that's it)
2) another approach is to always add two revisions when adding a new
property : the first revision would be a newly introduced one : an
always-visible revision mapping to null. The second revision would be
the same as was done previously: the current revision mapping to the
actual new value. When a cluster node thus reads that second revision
(which it does first, as it orders in reverse) and notices that it's not
yet visible, it would fall back to the first revision that one would
always be visible and map to null (kind of like "_deleted:false"). Thus
it can avoid previous document scans. This solution seems faster - but
it is a change in the stored data, uses a bit more space which
ultimately then also requires garbage collection. But it is somewhat
intriguing.
For now in my PR [1] I went for approach 1)
But knowing the delicacy of introducing a new cache I wanted to hear and
discuss other opinions of this group.
Thanks,
Cheers,
Stefan
--
[0] https://issues.apache.org/jira/browse/OAK-11184
[1] https://github.com/apache/jackrabbit-oak/pull/1779
