Hi Brad, Keeping the getDirectoryContexts() virtual would allow both a default implementation that uses getFileNames() and an local implementation that overrides this when appropriate.
Robert. On Wed, May 4, 2011 at 4:14 AM, Christiansen, Brad <[email protected]> wrote: > Hi, > > I thought I should add my 2c on the issue as I have done some work in this > area I hope to turn into a submission soon (if there is interest from the > commuity). > > When working with large VPB data sets I found the current osga format did not > scale very well to files 100+ gig in size. Just opening the archive could > take many minutes as it read the full index. To overcome these issues I have > implemented an SQL Lite implementation of the OSG archive interface (public > domain simple SQL single file databse). This is working very well and is > still very responsive with 5+ million individual entries and archive file > sizes over 100 gig. > > Given the approach I have taken, it will be important to at least allow any > getDirectoryContents implementaion to avoid calling getFileNames. For my > implementation of getDirectoryContents I would write a specific SQL query. > While I have implemented the getFileNames method I intent to avoid ever > calling this if possible as with 5million + entries it is not very useful or > performant. > > As a side note, I have been very happy with the performance and flexablility > I have gained by using SQL Lite as the back end. It forfills the needs of an > OSG archive very well and has enabled me to very easily extend the contents > of the archive to include application specific non-OSG based data. > > Cheers, > > Brad > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Robert > Osfield > Sent: Saturday, 30 April 2011 12:54 AM > To: OpenSceneGraph Submissions > Subject: Re: [osg-submissions] Registry modifications > > Hi Bradley, > > Rather than dive in right away and merge you submission I thought I'd > prep the ground for it - first up added your suggested methods to > osgDB::Archive. I have also implemented these methods in the osga > plugin, although as yet I have only compiled then and will need > obviously need to test them properly by adding extra checks to the > osgarchive example. I have run out working week so will have to leave > this to others or for another day. > > I have now checked these changes into svn/trunk, could you please > update you submission to use the osgDB::Archive as your > ArchiveExtended is no longer required. I would also like you to > review my implementation of OSGA_Archive::getDirectoryContents(..) as > I believe this is the type of functionality you intended, but I've > added handling of the possibility of different slashes being used in > the archive vs > calling functions directory name. The implementation is below (also > in svn/trunk's src/osgPlugins/osga). > > It occurred to me that would could probably implement the > getDirectoryContents() method directly in the > base class if we change this function to use the getFileNames() > function to get all the filenames from the > underlying implementation. This wouldn't be quite as computationally > efficient, but would avoid the potential > for the implementations to diverge in their interpretation of how they > are meant to work. Another possible approproach would be to put a > helper method into base class to help with the identification of > contenders for a directory name match. > > Thoughts? > Robert. > > -- Implementation of OSGGA_Archive::getDirectoryContents(const > std::string&) const: > > > osgDB::DirectoryContents OSGA_Archive::getDirectoryContents(const > std::string& dirName) const > { > osgDB::DirectoryContents files; > if (dirName.empty() || dirName==".") > { > getFileNames(files); > return files; > } > > bool dirNameHasTrailingSlash = false; > char d = dirName[dirName.size()-1]; > if (d=='\\' || d=='/') dirNameHasTrailingSlash = true; > > for(FileNamePositionMap::const_iterator itr=_indexMap.begin(); > itr!=_indexMap.end(); > ++itr) > { > const std::string& filename = itr->first; > if (filename.size()>dirName.size()) > { > // check for match of directory name while accounting for potential > // differences in types of slashes > unsigned int i=0; > for(; i<dirName.size(); ++i) > { > char f = filename[i]; > char d = dirName[i]; > if (f=='\\') f='/'; > if (d=='\\') d='/'; > if (f!=d) break; > } > > if (i==dirName.size()) > { > bool directoryMatched = false; > if (!dirNameHasTrailingSlash) > { > // no trailing slash on dirName so we won't have matched > // the required slash on filename to make sure the > next character is slash. > char f = filename[i]; > > // check for slash > if (f=='\\' || f=='/') > { > // found slash, now need to skip over it. > ++i; > directoryMatched = true; > } > } > else > { > directoryMatched = true; > } > > if (directoryMatched) > { > files.push_back(filename.substr(i)); > } > } > } > } > return files; > } > _______________________________________________ > osg-submissions mailing list > [email protected] > http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org > > > > DISCLAIMER:--------------------------------------------------------------------------- > This e-mail transmission and any documents, files and previous e-mail messages > attached to it are private and confidential. They may contain proprietary or > copyright > material or information that is subject to legal professional privilege. They > are for > the use of the intended recipient only. Any unauthorised viewing, use, > disclosure, > copying, alteration, storage or distribution of, or reliance on, this message > is > strictly prohibited. No part may be reproduced, adapted or transmitted > without the > written permission of the owner. If you have received this transmission in > error, or > are not an authorised recipient, please immediately notify the sender by > return email, > delete this message and all copies from your e-mail system, and destroy any > printed > copies. Receipt by anyone other than the intended recipient should not be > deemed a > waiver of any privilege or protection. Thales Australia does not warrant or > represent > that this e-mail or any documents, files and previous e-mail messages > attached are > error or virus free. > -------------------------------------------------------------------------------------- > > _______________________________________________ > osg-submissions mailing list > [email protected] > http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org > _______________________________________________ osg-submissions mailing list [email protected] http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org
