John If i understand property, once my partition has been created, if i ensure that every auxiliary data has been created (idx, int, sp, dic, ...), i would be able to use it as a read-only partition from multiple processes, am i right ?
My concern is that i'm not exactly sure when all these auxiliary files get created. I know that if i build indexes, the .idx, .int and .dic gets created, but it seems that the .sp gets created on demand later. In my use case, this is not a big deal to have a copy for each process, but if i can avoid it, i'll be more that happy :) Thanks, -----Original Message----- From: K. John Wu [mailto:[email protected]] Sent: Thursday, January 19, 2012 5:04 PM To: FastBit Users Cc: Dominique Prunier Subject: Re: [FastBit-users] Multiple process using the same partition Hi, Dominique, FastBit is thread-safe in the sense that one process can spawn multiple threads and each thread can perform read and write operations on the same data partition. There is sufficient locking mechanism to prevent them from interfering with each other. However, there is no mechanism implemented that allow multiple processes to work with the same data partition where more than one of them is attempting to write to the data directory. With that said, it is relatively straightforward to arrange for one process to do the writing and others to do the reading only. Furthermore, you can have multiple writers each writing to a different directory structure and have a set of reads that are aware of all the directories. For the purpose of monitoring the data directory that might be updated by a separate writer, ibis::part class has a function named updateData which can be called to find out whether the data directory has been updated since it was last called. This should allow the "readers" a chance to catch up with the writers when necessary. Hope this helps. John PS: The query processing operations are read-only operations. Appending new data and building new indexes are "write" operations because then actually writes new data to the data directory. PPS: There is actually a shadowing mechanism in ibis::part that allows on to work with one copy while another copy is being updated. When the update operations are complete, the two version can swapped and the older version is automatically updated. However, this process is designed for a multi-thread environment, not available is you are using separate processes. On 1/19/12 1:31 PM, Dominique Prunier wrote: > Hi, > > > > I was wondering if FastBit was supporting concurrency between multiple > processes trying to access the same partition on disk. > > > > Thanks, > > > > */Dominique Prunier/**//* > > APG Lead Developper > > Logo-W4N-100dpi > > 4388, rue Saint-Denis > > Bureau 309 > > Montreal (Quebec) H2J 2L1 > > Tel. +1 514-842-6767 x310 > > Fax +1 514-842-3989 > > [email protected] <mailto:[email protected]> > > www.watch4net.com <http://www.watch4net.com/> > > / / > > /This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise private information. If you have > received it in error, please notify the sender immediately and delete > the original. Any other use of this electronic mail by you is prohibited. > > //Ce message est pour le récipiendaire désigné seulement et peut > contenir des informations privilégiées, propriétaires ou autrement > privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur > immédiatement et effacez l'original. Toute autre utilisation de ce > courrier électronique par vous est prohibée./// > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
