Re: ApacheDS with Mavibot anytime soon?

[Ashma Shrestha]

Carlos,

Thanks for the help.

@Emmanuel - I know you have a busy schedule however, is there a timeline to
when we can expect a fix? Currently we are working a system which might
make this scenario of concurrent update and search occur more.

Ashma

On Thu, Mar 24, 2016 at 6:02 PM, Accorsi, Carlo <carlo.acco...@siemens.com>
wrote:

> HI Ashma,
> We have all our directory content dumped out as an LDIF on a periodic
> schedule. If the system crashes, we delete the partition folders on the
> file system, restart the server and  re-inject the entries from the ldif.
>
> If your directory is small < 1000 entries, you can just get them perhaps
> all in one search result.
> Otherwise use a paged result to loop through each entry and call:
>
> String ldif = LdifUtils.convertToLdif(entry)
>
> and write / append the string value to a file. This is your backup ldif
> file.
>
> Hope this helps..
>
>
>
>
> -Original Message-
> From: Ashma Shrestha [mailto:ashma.shres...@gmail.com]
> Sent: Thursday, March 24, 2016 4:38 PM
> To: users@directory.apache.org
> Subject: Re: ApacheDS with Mavibot anytime soon?
>
> Hi Carlos,
>
> You have mentioned "we get the 'ERR_554 double get for block' and then we
> need to restore. We have an automated way of doing this now so it's not too
> disruptive when it happens." . I am running into the same ERR_554 issue,
> can you let me know how you fixed it?
>
> Thanks,
>
> Ashma
>
> On Mon, Sep 28, 2015 at 9:15 AM, <carlo.acco...@ibs-ag.com> wrote:
>
> > Emmanuel! Wow!! Thank you for that most detailed and comprehensive
> > write up of the concurrency problem you are working to solve with
> > Mavibot.  The design sounds world class.
> > I now fully appreciate the complexity of this transaction issue and
> > the ASCII art helped make it clear. We realize you're all doing your
> > best to devise a robust solution for this and so we'll sit tight for now.
> > I appreciate the time you took to explain this to us. Have fun at the
> > conference.
> > Best, Carlo Accorsi
> >
> >
> >
> >
> > -Original Message-
> > From: Emmanuel Lécharny [mailto:elecha...@gmail.com]
> > Sent: Sunday, September 27, 2015 1:23 PM
> > To: users@directory.apache.org
> > Subject: Re: ApacheDS with Mavibot anytime soon?
> >
> > Le 23/09/15 21:41, carlo.acco...@ibs-ag.com a écrit :
> > > Hi,
> >
> > hi,
> >
> > sorry for the delayed response ! I saw your mail 3 days ago, but I
> > forgot to answer back then, being caught in a storm of delaying events...
> > >
> > > We have one customer that wants to get replication implemented.
> > > We've
> > set it up in house and it seems to work well.
> > > The trouble is there are 10's of thousands of users and all sorts of
> > concurrent activity. They're on M16 and a few times per month, we get
> > the
> > 'ERR_554 double get for block' and then we need to restore.
> > > We have an automated way of doing this now so it's not too
> > > disruptive
> > when it happens.
> > Hopefully. Still I understand this is not convenient.
> > >
> > > My question is, would adding replication just add more concurrent
> > activity to the servers and lead more frequently to the ERR_554
> situation?
> > For the consumer, it will be just as if the updates were coming from a
> > standard user. So, no, I don't think it's a real problem.
> > >
> > > Also he's asked to upgrade to M20 but my hunch is that for this
> > > issue it
> > won't matter much because we're still on JDBM?
> >
> > True. It's just that some bugs have been fixed in M20.
> > >
> > > Finally, we're eager to test out a release with the Mavibot backed.
> > > I'm sure it's possible to build ourselves but we're trying to keep
> > > the
> > moving parts to a minimum and use only ApacheDS releases.
> > I understand.
> >
> > Mavibot work has been delayed, and we are working on it as much as we
> > can, but it's not enough. Kiran and I have day jobs atm that forbid us
> > to dedicate as much time as would be necessary to complete the last
> > part that is mandatory to get this problem fixed : the transaction
> > system. We restarted to 'think' about the best implementation 2 weeks
> > ago (on your free time) and we expect to be able to spend more time on
> > it next week, as we will both be at Budapest, during the Apache
> Conference.
> >
> > Transaction  support (the way we need it, ie cross-btree) is not that
> > simple to impelment. We have a prett

Re: ApacheDS with Mavibot anytime soon?

[Ashma Shrestha]

 | | 4 | | 0 | | 0 |  Was 1
>+---+ +---+ +---+ +---+
>|   |>|   |>|   |--o  |   |--o
>+---+ +---+ +---+ +---+
> o--|   |<|   |<|   |<|   |
>+---+ +---+ +---+ +---+
>
> In this case, we will have 2 threads dealing with two elements to be
> cleaned up. This is not a problem, as T9 will now detach the N-2 element
> from teh N-1 one and T9 will detach the N-1 element from the list. All is
> safe.
>
> About the removed elements
> --
>
> Once a thread has detached some elements which counters are equal to 0,
> what should we do with them ? As we will update the disk with the removed
> pages, we have to send them to teh Writter thread. This can be done by
> pushing teminto a deque (and more specifically a
> ConcurrentLinkedDeque)
>
> Another option is that the writer thread handle the cleaning of this queue
> when it is weaken up or periodically. In this case, we don't need to push
> the element sinto a deque, and the reader threads will just set the counter
> to 0.
>
> Second case
> ---
>
> What if we accept the removal of elements from the middle of the list ?
> This is way more complex, as we have to be sure we don't break the chain.
> One way to get rid of this constraint is to enforce the writer thread to do
> the job, by periodically checing the whole list (which shuld not be too
> costly). In this case, the reader thread never update the element's
> parent's next pointer, they just decrement teh counter.
> When this counter becomes 0, then we can remove the element.
>
> Solution
> 
>
> We will separate the threads into 2 categories :
> - readers (we may have many)
> - writer (we have only one)
>
> The readers always peek the latest revision (which is the head of the
> list). It then increment the counter. When teh thread does not need the
> revision anymore, the counter is decremented. This is the only action the
> reader thread will ever do on the elements and on the list, but it will
> always keep a reference to the element into the session.
>
> The writer thread has a lot more to do :
> - it is responsible for adding element at the top of the list (see upper).
> Once the head is swapped, every new reader will get the added element as a
> reference.
> - it also has to check in the list for the elements which counter is 0, if
> teh list is longer than 1 element. This check can be done after each
> update, or periodically (every N seconds if the number of elements is > 1,
> or after M additions) if tht help saving updates on disk.
>
> Referencing pages
> -
>
> All the B-tree revisions are kept into the BoB B-tree, so it's easy to
> retrieve the rootPage for a given revision.
>
> As all the pages can't be hold in memory, we sometime have to reach a page
> from disk or from the cache. This is done following this simple algorithm :
>
>   for a given page with offset O
> if the cache contains an instance for O
>   return the page
> else
>   fetch the page with offset O from disk
>   store it in the cache
>
> In our case, we just add a layer, which will check in the working memory
> for the instance of a page, given its offset.
>
>
>


-- 
Ashma Shrestha
PHP Developer
(770) 310-9456
ashma.shres...@gmail.com