On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu <[email protected]> wrote: > On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <[email protected]> wrote: >> >> Hi, >> a few points and questions: >> *I am planning to use a common ID for all partitions. I checked Hbase >> partition and it uses UUID as the ID. Is it ok to use this for all >> partitions? >> > > This great and something we wanted to do for a very long time as Emmanuel > stated. I am working on JDBM and AVL partitions to make them usee UUID. Also I am making partitions expose their master tabe and indices. For making all partitions use UUID: * I can change partitions to work with UUID directly and make them return master tables as MasterTable<Entry,UUID> and index tables in a similar way *or I can paratmetrize partition interface like Partition<ID> . They can return master table as <MasterTable,ID> and index tables in a similar way. This is looks to me the right way but it eventually leads to more changes.
which one do you think is better? > >> >> *It seems that I need to move txn and log package implementations to >> core-shared because of the recent reorg. Coresession and nexus >> implementation is moved there and interceptors might need these >> packages. >> > > +1 > >> >> * When preparing txn log edits, I will need to log Modification and >> Attribute objects and keep them in memory for a while. Is it necessary >> to clone these objects while doing the logging at partition nexus >> level? > > Don't think so. > >> >> In other words, will they be modified after reaching nexus >> level. I am thinking no but I wanted to check because I looked at >> changelog implementation and it does cloning. >> > > Thanks, > Alex > >> >> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <[email protected]> >> wrote: >> > Hi ! >> > >> > comments inline... >> > >> > On 11/1/11 12:19 PM, Selcuk AYA wrote: >> >> >> >> thanks for the feedback! Please see inline: >> > >> > <snip/> >> >> >> >> >> >> I guess the xdbm-partition Maven module is now going to be pretty damn >> >> thin >> >> or non-existent. Was this module destroyed and if not what actually >> >> remains? >> >> I think some util classes for tree based implementations remain there. >> >> Also there some avl related classes. >> > >> > We can review this project and eventually move the remaining classes >> > elsewhere. >> > >> >> >> >>>> The todo is the following: >> >>>> *add changes to keep track of dn space changes. >> >>>> *test txn manager services >> >>>> *move the modification code path in AbstractBTreePartition in xdbm to >> >>>> high up to core. Probably modifications should be done in >> >>>> partitionNexus and partition nexus should handle preparing txn log >> >>>> edits and wal them. The overall flow for modifications will be: >> >>>> -DefaultDirectoryService: >> >>>> -begintxn >> >>>> -execute interceptor chain >> >>>> - handle txn abort, or conflict. >> >>>> >> >>> Moving this into the PartitionNexus might not be a good idea but no >> >>> problem >> >>> for now we can move it later. Let me explain why: >> >>> Eventually we're going to enable a root Partition with Partition >> >>> nesting >> >>> and >> >>> so when this happens the PartitionNexus will just be another nestable >> >>> Partition since these will have to handle routing based on DN to other >> >>> partitions residing/nested under it. >> >>> I see two possible locations for this functionality: >> >>> (1) Let the InterceptorChain itself handle this since it can demarcate >> >>> the >> >>> start and end of calls into the chain with Txn begin and abort/commit >> >>> calls. >> >>> It does this by making calls against the TxnManager which I guess is >> >>> the >> >>> entire facade for the transaction subsystem. >> >>> ---OR--- >> >>> (2) Handle Txn demarcation within the CoreSession. However this might >> >>> not >> >>> be >> >>> optimal due to the need to handle additional logic which might be >> >>> required >> >>> for handling chain re-entry concerns. >> >>> NOTE: I've not actually looked at the code after these major moves so >> >>> my >> >>> advice might not be very dependable. I will try to setup my >> >>> environment >> >>> to >> >>> get a better idea of these matters. >> >>> However for the time being do whatever actually makes this thing work. >> >>> Let's >> >>> follow an agile methodology. This thing is big. So let's get it >> >>> working >> >>> with >> >>> solid test coverage then we can actually look at shuffling things >> >>> around >> >>> to >> >>> optimal positions. Not saying what you've chosen is wrong ... it might >> >>> just >> >>> present the need for some additional refactoring when other features >> >>> might >> >>> need to be introduced. >> >>> These are some of the biggest changes to the architecture to have >> >>> taken >> >>> place in years and you're doing a great job. >> >> >> >> You are right for txn demarcation. I wrote it wrong in the email. >> >> Demarcation has to be done either at defaultcoresession or >> >> defaultoperationmanager. >> > >> > OperationManager is probably the right place. >> > >> > For inner operations, I'm almost 100% sure they aren't modifiying >> > anything >> > (the only modification done as a inner operation has been removed a >> > while >> > ago : it was dealing with the addition of the ModifyTimeStamp and >> > ModifiersName attribute, and it' snow done in the main operation). So we >> > are >> > safe if they are done in the current external transaction. >> > >> >> >> >> as for where to handle the change logic, as you mentioned we need a >> >> place where all interceptor chain routes end up for modification and >> >> ideally we should handle modification logic above partitions using >> >> master table and index interfaces so that we have a common place to >> >> prepare and apply txn log edits. PartitionNexus seems to fit this >> >> requirement for now. If we add another layer above PartitionNexus >> >> which can get master and index table from below layers and work with >> >> them, we should be able to move the change logic up there. >> >> >> >>>> *move xdbm-search to core as well. Making search transactional will >> >>>> mostly be mostly mechanica after this point I think(hope). It should >> >>>> just use the wrappers the txn manager provides for index, master and >> >>>> cursors it gets from the partitions. >> >>>> >> >>>> *handle caches various interceptors keep. I am thinking of handling >> >>>> this with a common read-write lock. >> >>>> >> >>> This was the latest issue for which I see some more threads. Will look >> >>> at >> >>> that as well. >> >> >> >> This is the issue I talked you about briefly. These are mostly admin >> >> caches that change infrequently and that are read mostly. The >> >> difficulty with them is that they are not always entry caches. They >> >> might map Dns to some logical property of the entry. A simple and good >> >> example is notaliascache (whether we really need this is open to >> >> debate but we have it now). >> > >> > This is a very interesting point. The notAliasCache is definitively >> > questionable. The question though is to find a better way to deal with >> > it. >> > An option would be to not support alias (yes, I know, pretty drastic, >> > but if >> > it allows us to speed up the delivery of a production ready server, I >> > think >> > this would be a good tradeoff. Also note that once we have a solid base, >> > we >> > can reintroduce alias handling in a future version). >> > >> >> This maps a Dn to whether the entry is an >> >> alias. If we followed the normal way of merging what is read from >> >> partitions with the txn log, we would have to a way of merging what is >> >> read from this notaliascache with the log. This is not very difficult >> >> but as we have quite a number of these caches, having a separate merge >> >> and update logic for each of them is a pain and error prone. >> >> >> >> Instead, what I thought was to use a single system wide read-write >> >> lock for these caches. Since they are read mostly, a txn might update >> >> its lock to exclusive when it needs to change one of these caches and >> >> might release it when it actually commits/aborts. >> > >> > We may start a new thread to discuss further those cache issues. We have >> > many caches, for may different thinsg, and many different approaches to >> > deal >> > with the (MRU, EhCache, etc...). It's probably a good timing for >> > clarifying >> > this item... >> > >> > -- >> > Regards, >> > Cordialement, >> > Emmanuel Lécharny >> > www.iktek.com >> > >> > > > > > -- > Best Regards, > -- Alex >
