Hi, Dominique, FastBit SVN revision 463 has a couple of extra functions that enable ibis::part::buildIndexes to create all necessary auxiliary files, even the nasty .sp files for text columns. Please give it a try when you get a chance.
Thanks. John On 1/19/12 3:02 PM, Dominique Prunier wrote: > John > > If i understand property, once my partition has been created, if i ensure > that every auxiliary data has been created (idx, int, sp, dic, ...), i would > be able to use it as a read-only partition from multiple processes, am i > right ? > > My concern is that i'm not exactly sure when all these auxiliary files get > created. I know that if i build indexes, the .idx, .int and .dic gets > created, but it seems that the .sp gets created on demand later. > > In my use case, this is not a big deal to have a copy for each process, but > if i can avoid it, i'll be more that happy :) > > Thanks, > > -----Original Message----- > From: K. John Wu [mailto:[email protected]] > Sent: Thursday, January 19, 2012 5:04 PM > To: FastBit Users > Cc: Dominique Prunier > Subject: Re: [FastBit-users] Multiple process using the same partition > > Hi, Dominique, > > FastBit is thread-safe in the sense that one process can spawn > multiple threads and each thread can perform read and write operations > on the same data partition. There is sufficient locking mechanism to > prevent them from interfering with each other. > > However, there is no mechanism implemented that allow multiple > processes to work with the same data partition where more than one of > them is attempting to write to the data directory. > > With that said, it is relatively straightforward to arrange for one > process to do the writing and others to do the reading only. > Furthermore, you can have multiple writers each writing to a different > directory structure and have a set of reads that are aware of all the > directories. > > For the purpose of monitoring the data directory that might be updated > by a separate writer, ibis::part class has a function named updateData > which can be called to find out whether the data directory has been > updated since it was last called. This should allow the "readers" a > chance to catch up with the writers when necessary. > > Hope this helps. > > John > > PS: The query processing operations are read-only operations. > Appending new data and building new indexes are "write" operations > because then actually writes new data to the data directory. > > PPS: There is actually a shadowing mechanism in ibis::part that allows > on to work with one copy while another copy is being updated. When > the update operations are complete, the two version can swapped and > the older version is automatically updated. However, this process is > designed for a multi-thread environment, not available is you are > using separate processes. > > > On 1/19/12 1:31 PM, Dominique Prunier wrote: >> Hi, >> >> >> >> I was wondering if FastBit was supporting concurrency between multiple >> processes trying to access the same partition on disk. >> >> >> >> Thanks, >> >> >> >> */Dominique Prunier/**//* >> >> APG Lead Developper >> >> Logo-W4N-100dpi >> >> 4388, rue Saint-Denis >> >> Bureau 309 >> >> Montreal (Quebec) H2J 2L1 >> >> Tel. +1 514-842-6767 x310 >> >> Fax +1 514-842-3989 >> >> [email protected] <mailto:[email protected]> >> >> www.watch4net.com <http://www.watch4net.com/> >> >> / / >> >> /This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise private information. If you have >> received it in error, please notify the sender immediately and delete >> the original. Any other use of this electronic mail by you is prohibited. >> >> //Ce message est pour le récipiendaire désigné seulement et peut >> contenir des informations privilégiées, propriétaires ou autrement >> privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur >> immédiatement et effacez l'original. Toute autre utilisation de ce >> courrier électronique par vous est prohibée./// >> >> >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
