Hi !
firs, I'd like to wish you all, users and dev, a very happy 2020 !
2019 was quite slow, for many reasons. I can't talk about others, but on
my side, I had a big change in my day job just sucked all my free time,
and made me vary hard to participate since august. Things are cooling
down now (changing for a new job is always very taxing in term of free
time).
Since august, I spent time on Mavibot, implementing transaction support.
It was pretty hard, with some concurrent issues that took me one month
to get fixed (mainly because I wasn't able to dedicate more than a few
hours per week on it). It's not yet committed, but it works. The biggest
plus is that when you activate a transaction, and make more than one
update on a B-tree, all being done in memory up to the final commit,
it's way, way faster. It more or less double the number of updates per
second when you double the number of updates per transaction (well,
obviously less than double ;-). There are reasons for this speedup :
- the updated B-tree get flushed on disk only once, we then save many
pages writes (typically, the root page get flushed only once per txn,
instead of being flushed for every update.
- technical B-trees (CopiedPages and BtreeOfBtrees) and recordManager
header get flushed only once too
There are some other improvements too: I have replaced the BtreeOfBtrees
by a ListOfBtrees. It's way more dense, as it only keeps the latest
offset of each existing B-tree. That also means we don't write an extra
B-tree when we commit a transaction. The drawback is that we have to
write the full list after each commit, but we are talking about only a
few pages (each page containing up to 500 B-tree offset for a 4Kb page).
The second drawback is that of we have a brutal crash, we may lose some
old B-tree references if they were used by long lasting transactions.
Note that we still keep a track of all the used B-tree reference in
memory until they get removed.
The free pages management is not active at this point, and this is the
next step to be completed. I want to change the existing algorithm to
something better. Currently, we only clean unused pages when the oldest
active read transaction is released, which is an issue if we have very
long lasting read transactions: we may have a growing database that is
never cleaned up if the transaction is never released. The alternative
algorithm uses the CopiedPageBtree structure that keep a track of copied
pages when a write transaction has been committed. It allows us to get
rid of pages from in-between read transaction when they get released,
assuming those pages have been copied. The algorithm is slighly more
complex to implement, but guarantee that long lasting transactions won't
clog the database.
Otherwise, a LDAP API release has been done, the first GA. This is quite
a milestone ! There is still a lot of work to do, and that includes the
documentation, which is lagging.
On ApacheDS front, this is a bit more complicated. I have triaged all
the issues, adding each of them inti some categories, and started to
work on the PasswordPolicies issues (there are many). It takes a lot of
time though...
Here is what I intend to do then :
- cut a release as is, in order to have a code base that use teh 2.0
LDAP API. Some people are waiting for month to have a test envirronement
that contains some major fixes, so regardless on where I'm on PP fixes,
I think it's urgent to have this release done.
- fix the packaging. We need to have a Mac OSX package that works, and
that means something signed (the new Apple policy for installable
package is a bit strict...). I have this signature available, I just
need to get it applied to the package. We also need a Windows 64
package. Currently, we use a 32bits library - NSIS - which is not
maintained anymore. I'm looking into YAJSW, which seems quite
compatible. Then we need to get Mavibot upgraded with the version i'm
working on, and make it the default database, to solve the long lasting
database corruption that bog is for years...
Plus many other bugs to work on !
That means we will have many releases before reaching a GA :/
At this point, I may need some hands !
Regarding Studio, I think Stefan is waiting for ApacheDS to be released
to cut a big fix release. Stefan, feel free to comment.
In any case, we need some help in all the aspects of the project
(ApacheDS, LDAP API, Fortress,Studio, etc), and that include doculmentation.
I haven't mentioned Docker, but we would appreciate any effort to
containerize ApacheDS, Fortress and Studio. This is not complex, Shawn
did it for Fortres, but it would be cool to have it documented. I wonder
if it's not the proper approach compared to creating packages for
various platforms...
Anyway, lots of work on my plate, but happy to do it !
Feel free to comment, fo course !
Thanks
Emmanuel
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
