Hi, Jon,
There was a odd logic error in function flushDir that should have been
corrected now. The change is in SVN as revision 835. Please give it
a try when you get the chance.
The documentation inconsistency should also be fixed now. My
intention is to remove all files whose name starts with the given
directory name. The incoming argument must be a whole directory name
(either absolute path or relative path, but the same path must have
been used to construct the data partition to start with), without
leading or trailing blanks in the name. If there is a directory named
"a" and another named "a1", when the argument to flushDir is "a", this
function should remove every file in "a", but leave those in "a1" alone.
John
On 9/29/15 1:24 PM, Jon Strabala wrote:
> John,
>
> Your right (of course) - calls to either of the following fucntions do
> indeed free memory mapped data - my mistake.
>
> ibis::fileManager::instance().flushDir(char *dir);
> ibis::fileManager::instance().flushFile(char *file);
>
> In src/fileManager.cpp the description for ibis::fileManager::flushDir
> says
>
> "remove all files from the specified directory (include all sub
> directories)"
>
> I do have a question about the above statement as it dosn't seem to
> match my experiences. Say I have a directory tree like
> /db/2015/09/{01,02,03,04} e.g. four sub directories 01, 02, 03 and 04
> which are fastbit partitions that each have "-part.txt" and fastbit
> data and index files. Note in this example everyting else is just a
> directory.
>
> When I specify an actual "leaf" directory (e.g. an actual fastbit
> directory) like "/db/2015/09/03" the flush works as expected and all
> files in that directory are removed.
>
> ibis::fileManager::instance().flushDir("/db/2015/09/03"); //
> works just fine
>
> But when I specify a parent directory like "/db/2015/09" or
> "/db/2015/" which just holds lower level directories (e.g. NO
> "-part.txt" files) the flush does not seem to iterate and work on the
> sub directories despite the description.
>
> ibis::fileManager::instance().flushDir("/db/2015/09");
> // fails to remove fastbit sub directories like "/db/2015/09/01" ....
>
>
> FYI, I'm indifferent about the functionality it's just that the
> description seems a bit misleading. Perhaps the comment to
> ibis::fileManager::flushDir should be revised to say:
>
> "remove all files from the specified directory (does not include
> all sub directories)"
>
>
> - Jon
>
> On Mon, Sep 28, 2015 at 10:16 AM, K. John Wu <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Jon,
>
> From what I can see in the file fileManager.cpp, both
> fileManager::clear and fileManager::flushFile attempts to remove files
> from both list of mapped files as well as those simply read into
> memory. If you know exactly what files you want to remove from the
> file cache, you definitely could invoke flushFile directly. If you
> encounters any problems, please let us know.
>
> If you are update a data directory, you can also use
> ibis::fileManager::flushDir.
>
> John
>
>
> On 9/28/15 9:32 AM, Jon Strabala wrote:
> > John,
> >
> > There are two functions e.g. in fileManager.cpp
> >
> > ibis::fileManager::clear // attempts to free resources across
> > all files
> > ibis::fileManager::flushFile // attempts to free resources for
> one file
> >
> > I believe that only ibis::fileManager::clear seems to call
> > mapped.clear(); and incore.clear().
> >
> > I don't think ibis::fileManager::flushFile removes memmorry map
> > references - is there a reason for this (or am I mistaken about this
> > function) ?
> >
> > Since I am updating partitions in real-time in a multi threaded
> > environment I need to make sure that there is no activity (e.g.
> set a
> > write lock) and also flush stale cached data. Right now I use
> > ibis::fileManager::clear which seems like overkill considering I
> might
> > only be updating one partition out of 500. I would like to use
> > ibis::fileManager::flushFile since it appears that it would have a
> > minimal performance impact, but as I said I don'tr think it will
> free
> > a memory mapped reference.
> >
> > Best Regards,
> >
> > Jon Strabala
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected] <mailto:[email protected]>
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
> _______________________________________________
> FastBit-users mailing list
> [email protected] <mailto:[email protected]>
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>
>
>
>
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
