Hi, I thought I should add my 2c on the issue as I have done some work in this area I hope to turn into a submission soon (if there is interest from the commuity).
When working with large VPB data sets I found the current osga format did not scale very well to files 100+ gig in size. Just opening the archive could take many minutes as it read the full index. To overcome these issues I have implemented an SQL Lite implementation of the OSG archive interface (public domain simple SQL single file databse). This is working very well and is still very responsive with 5+ million individual entries and archive file sizes over 100 gig. Given the approach I have taken, it will be important to at least allow any getDirectoryContents implementaion to avoid calling getFileNames. For my implementation of getDirectoryContents I would write a specific SQL query. While I have implemented the getFileNames method I intent to avoid ever calling this if possible as with 5million + entries it is not very useful or performant. As a side note, I have been very happy with the performance and flexablility I have gained by using SQL Lite as the back end. It forfills the needs of an OSG archive very well and has enabled me to very easily extend the contents of the archive to include application specific non-OSG based data. Cheers, Brad -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Robert Osfield Sent: Saturday, 30 April 2011 12:54 AM To: OpenSceneGraph Submissions Subject: Re: [osg-submissions] Registry modifications Hi Bradley, Rather than dive in right away and merge you submission I thought I'd prep the ground for it - first up added your suggested methods to osgDB::Archive. I have also implemented these methods in the osga plugin, although as yet I have only compiled then and will need obviously need to test them properly by adding extra checks to the osgarchive example. I have run out working week so will have to leave this to others or for another day. I have now checked these changes into svn/trunk, could you please update you submission to use the osgDB::Archive as your ArchiveExtended is no longer required. I would also like you to review my implementation of OSGA_Archive::getDirectoryContents(..) as I believe this is the type of functionality you intended, but I've added handling of the possibility of different slashes being used in the archive vs calling functions directory name. The implementation is below (also in svn/trunk's src/osgPlugins/osga). It occurred to me that would could probably implement the getDirectoryContents() method directly in the base class if we change this function to use the getFileNames() function to get all the filenames from the underlying implementation. This wouldn't be quite as computationally efficient, but would avoid the potential for the implementations to diverge in their interpretation of how they are meant to work. Another possible approproach would be to put a helper method into base class to help with the identification of contenders for a directory name match. Thoughts? Robert. -- Implementation of OSGGA_Archive::getDirectoryContents(const std::string&) const: osgDB::DirectoryContents OSGA_Archive::getDirectoryContents(const std::string& dirName) const { osgDB::DirectoryContents files; if (dirName.empty() || dirName==".") { getFileNames(files); return files; } bool dirNameHasTrailingSlash = false; char d = dirName[dirName.size()-1]; if (d=='\\' || d=='/') dirNameHasTrailingSlash = true; for(FileNamePositionMap::const_iterator itr=_indexMap.begin(); itr!=_indexMap.end(); ++itr) { const std::string& filename = itr->first; if (filename.size()>dirName.size()) { // check for match of directory name while accounting for potential // differences in types of slashes unsigned int i=0; for(; i<dirName.size(); ++i) { char f = filename[i]; char d = dirName[i]; if (f=='\\') f='/'; if (d=='\\') d='/'; if (f!=d) break; } if (i==dirName.size()) { bool directoryMatched = false; if (!dirNameHasTrailingSlash) { // no trailing slash on dirName so we won't have matched // the required slash on filename to make sure the next character is slash. char f = filename[i]; // check for slash if (f=='\\' || f=='/') { // found slash, now need to skip over it. ++i; directoryMatched = true; } } else { directoryMatched = true; } if (directoryMatched) { files.push_back(filename.substr(i)); } } } } return files; } _______________________________________________ osg-submissions mailing list [email protected] http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org DISCLAIMER:--------------------------------------------------------------------------- This e-mail transmission and any documents, files and previous e-mail messages attached to it are private and confidential. They may contain proprietary or copyright material or information that is subject to legal professional privilege. They are for the use of the intended recipient only. Any unauthorised viewing, use, disclosure, copying, alteration, storage or distribution of, or reliance on, this message is strictly prohibited. No part may be reproduced, adapted or transmitted without the written permission of the owner. If you have received this transmission in error, or are not an authorised recipient, please immediately notify the sender by return email, delete this message and all copies from your e-mail system, and destroy any printed copies. Receipt by anyone other than the intended recipient should not be deemed a waiver of any privilege or protection. Thales Australia does not warrant or represent that this e-mail or any documents, files and previous e-mail messages attached are error or virus free. -------------------------------------------------------------------------------------- _______________________________________________ osg-submissions mailing list [email protected] http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org
