Attached is my modified ReaderWriterZipFS.cpp. It currently only has
small changes to make it use osgDB::findFileInPath(). However, these
changes probably aren't helpful unless you override findFileInPath so
that it can search inside .zip files, like so:
// Override osgDB::findFileInPath so that it can search inside of zip
files as well as regular directories.
string osgDB::findFileInPath(const string& filename, const
FilePathList& filepath, CaseSensitivity caseSensitivity){
if (filename.empty())
return filename;
if (!isFileNameNativeStyle(filename))
return findFileInPath(convertFileNameToNativeStyle(filename),
filepath, caseSensitivity);
for(FilePathList::const_iterator itr=filepath.begin();
itr!=filepath.end(); ++itr){
OSG_DEBUG << "itr='" <<*itr<< "'\n";
string path = itr->empty() ? filename : concatPaths(*itr,
filename);
path = getRealPath(path);
OSG_DEBUG << "findFileInPath() : trying " << path << " ...\n";
if(fileExists(path)){
OSG_DEBUG << "findFileInPath() : USING " << path <<
"\n";
return path;
}
#ifndef WIN32
// windows already case insensitive so no need to retry..
else if (caseSensitivity==CASE_INSENSITIVE){
string foundfile =
findFileInDirectory(filename,*itr,CASE_INSENSITIVE);
if (!foundfile.empty())
return foundfile;
}
#endif
string::size_type pos(path.find(".zip"));
if(pos != string::npos){
string zip_filename(path, 0, pos + 4);
if(fileExists(zip_filename)){
// Open the zip file
struct zip* zf = zip_open(zip_filename.c_str(),
0, NULL);
if(zf){
// find index to file in zip
string filename_in_zip(path, pos + 4);
stripPrecedingSlashes(filename_in_zip);
int location = zip_name_locate(zf,
filename_in_zip.c_str(), 0);
zip_close(zf);
if(location >= 0){
OSG_DEBUG << "findFileInPath()
: USING " << path << "\n";
return path;
}
}
}
}
}
return string();
}
A side concern here is that half of my new findFileInPath is code
copied straight from OSG. But I can't include copied code in my
commercial app. To use this function legally do I need to package
this inside its own library, license that library under OSG's license,
and release the library's source?
I fixed the shader loading problem by overriding
Shader::loadShaderSourceFromFile(). This is necessary because this
function does the actual opening of the shader file. So I replaced
the guts with my game's new file loading utilities. This one is so
totally rewritten that it doesn't give me the same licensing worries
as findFileInPath.
Would love to see some of this functionality get into OSG. Or at
least have the hooks necessary so that these messy function overrides
are not necessary. But I haven't yet come up with a way to modify OSG
to make this possible. Any suggestions?
I still haven't tried converting everything to zlib in order to remove
the libzip dependency. I'll get to that one of these days. Currently
I'm trying to use this infrastructure to load my .wav and .ogg files,
but that's not an OSG problem.
--
Terry Welsh
mogumbo 'at' gmail.com
www.reallyslick.com
>
> Message: 25
> Date: Tue, 6 Sep 2011 11:17:01 -0400
> From: Jason Beverage <[email protected]>
> To: OpenSceneGraph Users <[email protected]>
> Subject: Re: [osg-users] tight integration of virtual file system
> Message-ID:
> <CAMcxSMt=gtyqugofedwsy8y16gavoijqwm_rbewg52+hkjm...@mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Terry,
>
> On Sun, Sep 4, 2011 at 9:29 PM, Terry Welsh <[email protected]> wrote:
>> Excellent ZipFS plugin Jason. ?Thank you for the help. ?I think that
>> was the hardest part of the problem. ?I finally got around to working
>> on this more and I have the other half of the problem mostly solved.
> Thanks, glad it worked for you. We don't use it anymore for osgEarth
> but I think in general it's pretty useful.
>
>> It looks like OSG uses zlib for all its compression/decompression.
>> Just out of curiosity, how did you pick libzip instead? ?Or did you
>> not write ZipFS?
> I did write the ZipFS plugin, we chose libzip over just using zlib b/c
> it had easy access to dealing with zip archives. If zlib itself has
> something in there that we can use instead of having another
> dependency that would be great but I didn't really look that hard
> since libzip worked so well.
>
> If you have a submission for the osgEarth that you need integrated
> just send along the whole modified file or do a pull request on github
> and I'll take a look at it.
>
> Jason
>
>> To recap, I'm trying to set an OSG_FILE_PATH such as
>> mydirectory/data.zip and load files out of the specified archive. ?At
>> first I discovered the Registry class's FindFileCallback. ?I thought I
>> could just make one of those and have it search inside of zip files.
>> However, the work that I really needed to replace happened when the
>> default Registry::findDataFileImplementation would call
>> osgDB::findFileInPath in FileUtils. ?So I decided to override
>> findFileInPath so that would look for paths containing ".zip" and try
>> to locate the desired file inside the zip file.
>>
>> That almost worked. ?I also had to modify ReaderWriterZipFS::readFile
>> so that it uses OSG's directory searching. ?At the beginning of the
>> function I added
>>
>> std::string filenameWithPath =
>> osgDB::Registry::instance()->findDataFile(fullFileName, options,
>> CASE_SENSITIVE);
>>
>> and then I use filenameWithPath instead of fullFileName where appropriate.
>>
>> So now I can load almost all my game resources from one .zip file
>> instead of a directory full of files. ?I haven't tested performance
>> yet, but there's no noticeable change. ?.osg files load with all their
>> textures.
>>
>> Here's the catch: .osg files don't get loaded with their shaders. ?I
>> believe this is because shaders don't use the standard OSG plugin
>> loading paradigm. ?Instead they simply use
>> Shader::loadShaderSourceFromFile(). ?I haven't put any thought into
>> this last detail yet (I hope it's not too messy), but I need a
>> programming break....
>> --
>> Terry Welsh
>> mogumbo 'at' gmail.com
>> www.reallyslick.com
>>
>>>
>>> Message: 9
>>> Date: Fri, 5 Aug 2011 08:50:10 -0400
>>> From: Jason Beverage <[email protected]>
>>> To: OpenSceneGraph Users <[email protected]>
>>> Subject: Re: [osg-users] tight integration of virtual file system
>>> Message-ID:
>>> ? ? ? ?<camcxsmsly2m20aoa5z7tubjob1xyrqqkmnz9z9lslzmmkqu...@mail.gmail.com>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Hi Terry,
>>>
>>> osgEarth has a generic zip based plugin called ZipFS that is
>>> implemented as an osgDB plugin and isn't dependent on osgEarth. ?We
>>> originally developed it to support using a zip file of imagery tiles
>>> for a cache.
>>>
>>> It allows you to load files like:
>>> osgDB::readNodeFile("c:/data/models.zip/cow.osg");
>>>
>>> I haven't used it for quite awhile but last I checked it was working
>>> just fine. ?You can get the osgearth source code from github at
>>> https://github.com/gwaldron/osgearth
>>>
>>> If anything it could be a start to what you're looking for.
>>>
>>> Thanks,
>>>
>>> Jason
>>>
/* -*-c++-*- */
/* osgEarth - Dynamic map generation toolkit for OpenSceneGraph
* Copyright 2008-2009 Pelican Ventures, Inc.
* http://osgearth.org
*
* osgEarth is free software; you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>
*/
#include <osg/Notify>
#include <osgDB/FileNameUtils>
#include <osgDB/FileUtils>
#include <osgDB/Registry>
#include <osgDB/ReadFile>
#include <osgDB/WriteFile>
#include <OpenThreads/ReentrantMutex>
#include <sstream>
#include <string.h>
#include "zip.h"
using namespace osg;
using namespace osgDB;
static OpenThreads::ReentrantMutex s_mutex;
/**
* The ZipFS plugin allows you to treat zip files almost like a virtual file system.
* You can read and write objects from zips using paths like c:/data/models.zip/cow.osg where cow.osg is a file within the models.zip file.
*/
class ReaderWriterZipFS : public osgDB::ReaderWriter
{
public:
enum ObjectType
{
OBJECT,
IMAGE,
HEIGHTFIELD,
NODE
};
ReaderWriterZipFS()
{
supportsExtension( "zipfs", "Zip virtual file system" );
}
virtual const char* className()
{
return "ZIP virtual file system";
}
virtual ReadResult readNode(const std::string& file_name, const Options* options) const
{
return readFile(NODE, file_name, options);
}
virtual ReadResult readImage(const std::string& file_name, const Options* options) const
{
return readFile(IMAGE, file_name, options);
}
virtual ReadResult readObject(const std::string& file_name, const Options* options) const
{
return readFile(OBJECT, file_name, options);
}
virtual ReadResult readHeightField(const std::string& file_name, const Options* options) const
{
return readFile(HEIGHTFIELD, file_name, options);
}
virtual WriteResult writeObject(const osg::Object& obj, const std::string& fileName, const osgDB::ReaderWriter::Options* options) const
{
return writeFile(OBJECT, &obj, fileName, options);
}
virtual WriteResult writeImage(const osg::Image& image, const std::string& fileName, const osgDB::ReaderWriter::Options* options) const
{
return writeFile(IMAGE, &image, fileName, options);
}
virtual WriteResult writeHeightField(const osg::HeightField& hf, const std::string& fileName, const osgDB::ReaderWriter::Options* options) const
{
return writeFile(HEIGHTFIELD, &hf, fileName, options);
}
virtual WriteResult writeNode(const osg::Node& node, const std::string& fileName, const osgDB::ReaderWriter::Options* options) const
{
return writeFile(NODE, &node, fileName,options);
}
WriteResult writeFile(ObjectType objectType, const osg::Object* object, osgDB::ReaderWriter* rw, std::ostream& fout, const osgDB::ReaderWriter::Options* options) const
{
switch (objectType)
{
case(OBJECT): return rw->writeObject(*object, fout, options);
case(IMAGE): return rw->writeImage(*(dynamic_cast<const osg::Image*>(object)), fout, options);
case(HEIGHTFIELD): return rw->writeHeightField(*(dynamic_cast<const osg::HeightField*>(object)), fout, options);
case(NODE): return rw->writeNode(*(dynamic_cast<const osg::Node*>(object)), fout,options);
default: break;
}
return WriteResult::FILE_NOT_HANDLED;
}
ReadResult readFile(ObjectType objectType, osgDB::ReaderWriter* rw, std::istream& fin, const osgDB::ReaderWriter::Options* options) const
{
switch (objectType)
{
case(OBJECT): return rw->readObject(fin, options);
case(IMAGE): return rw->readImage(fin, options);
case(HEIGHTFIELD): return rw->readHeightField(fin, options);
case(NODE): return rw->readNode(fin, options);
default: break;
}
return ReadResult::FILE_NOT_HANDLED;
}
ReadResult readFile(ObjectType objectType, const std::string &fullFileName, const osgDB::ReaderWriter::Options* options) const
{
OpenThreads::ScopedLock<OpenThreads::ReentrantMutex> lock(s_mutex);
std::string filenameWithPath = osgDB::Registry::instance()->findDataFile(fullFileName, options, CASE_SENSITIVE);
//This plugin allows you to treat zip files almost like virtual directories. So, the pathname to the file you want in the zip should
//be of the format c:\data\myzip.zip\images\foo.png
std::string::size_type len = filenameWithPath.find(".zip");
if (len == std::string::npos)
{
osg::notify(osg::INFO) << "ReaderWriterZipFS: Path does not contain zip file" << std::endl;
return ReadResult::FILE_NOT_HANDLED;
}
std::string zipFile = filenameWithPath.substr(0, len + 4);
zipFile = osgDB::findDataFile(zipFile);
zipFile = osgDB::convertFileNameToNativeStyle( zipFile );
//Return if the file doesn't exist
if (!osgDB::fileExists( zipFile )) return ReadResult::FILE_NOT_FOUND;
osg::notify(osg::INFO) << "ReaderWriterZipFS::readFile ZipFile path is " << zipFile << std::endl;
std::string zipEntry = filenameWithPath.substr(len+4);
//Strip the leading slash from the zip entry
if ((zipEntry.length() > 0) &&
((zipEntry[0] == '/') || (zipEntry[0] == '\\')))
{
zipEntry = zipEntry.substr(1);
}
//Lipzip returns filenames with '/' rather than '\\', even on Windows. So, convert the zip entry to Unix style
zipEntry = osgDB::convertFileNameToUnixStyle(zipEntry);
osg::notify(osg::INFO) << "Zip Entry " << zipEntry << std::endl;
//See if we can get a ReaderWriter for the zip entry before we even try to unzip the file
ReaderWriter* rw = osgDB::Registry::instance()->getReaderWriterForExtension(osgDB::getFileExtension(zipEntry));
if (!rw)
{
osg::notify(osg::NOTICE) << "Could not find ReaderWriter for " << zipEntry << std::endl;
return ReadResult::FILE_NOT_HANDLED;
}
int err;
//Open the zip file
struct zip* pZip = zip_open(zipFile.c_str(), ZIP_CHECKCONS, &err);
if (pZip)
{
//List the files
/*int numFiles = zip_get_num_files(pZip);
osg::notify(osg::NOTICE) << zipFile << " has " << numFiles << " files " << std::endl;
for (int i = 0; i < numFiles; ++i)
{
osg::notify(osg::NOTICE) << i << ": " << zip_get_name(pZip, i, 0) << std::endl;
}*/
//Find the index within the zip file for the given zip entry
int zipIndex = zip_name_locate(pZip, zipEntry.c_str(), 0);
osg::notify(osg::INFO) << "ReaderWriterZipFS: ZipFile index " << zipIndex << std::endl;
if (zipIndex < 0)
{
osg::notify(osg::INFO) << "Could not find zip entry " << zipEntry << " in " << zipFile << std::endl;
//Make sure to close the zipfile
zip_close(pZip);
return ReadResult::FILE_NOT_FOUND;
}
//Open the entry for reading
zip_file* pZipFile = zip_fopen_index(pZip, zipIndex, 0);
if (pZipFile)
{
//Read the data from the entry into a std::string
int dataSize = 0;
std::string data;
do{
char buffer[1024];
dataSize = zip_fread(pZipFile, buffer, 1024);
if (dataSize > 0)
{
data.append((char*)buffer, dataSize);
}
}while (dataSize > 0);
//Close the zip entry and the actual zip file itself
zip_fclose(pZipFile);
zip_close(pZip);
std::stringstream strstream(data);
return readFile(objectType, rw, strstream, options);
}
}
else
{
osg::notify(osg::NOTICE) << "ReaderWriterZipFS::readFile couldn't open zip " << zipFile << " full filename " << fullFileName << std::endl;
}
return ReadResult::FILE_NOT_HANDLED;
}
WriteResult writeFile(ObjectType objectType, const osg::Object* object, const std::string& fullFileName, const osgDB::ReaderWriter::Options* options) const
{
OpenThreads::ScopedLock<OpenThreads::ReentrantMutex> lock(s_mutex);
std::string::size_type len = fullFileName.find(".zip");
if (len == std::string::npos)
{
osg::notify(osg::INFO) << "ReaderWriterZipFS: Path does not contain zip file" << std::endl;
return WriteResult::FILE_NOT_HANDLED;
}
//It is possible that the zip file doesn't currently exist, so we just use getRealPath instead of findDataFile as in the readFile method
std::string zipFile = osgDB::getRealPath(fullFileName.substr(0, len + 4));
std::string path = osgDB::getFilePath(zipFile);
//If the path doesn't currently exist, create it
if (!osgDB::fileExists(path) && !osgDB::makeDirectory(path))
{
osg::notify(osg::WARN) << "Couldn't create path " << path << std::endl;
}
osg::notify(osg::INFO) << "ReaderWriterZipFS::writeFile ZipFile path is " << zipFile << std::endl;
std::string zipEntry = fullFileName.substr(len+4);
//Strip the leading slash from the zip entry
if ((zipEntry.length() > 0) &&
((zipEntry[0] == '/') || (zipEntry[0] == '\\')))
{
zipEntry = zipEntry.substr(1);
}
//Lipzip returns filenames with '/' rather than '\\', even on Windows. So, convert the zip entry to Unix style
zipEntry = osgDB::convertFileNameToUnixStyle(zipEntry);
osg::notify(osg::INFO) << "Zip Entry " << zipEntry << std::endl;
//See if we can get a ReaderWriter for the zip entry before we even try to unzip the file
ReaderWriter* rw = osgDB::Registry::instance()->getReaderWriterForExtension(osgDB::getFileExtension(zipEntry));
if (!rw)
{
osg::notify(osg::INFO) << "Could not find ReaderWriter for " << zipEntry << std::endl;
return WriteResult::FILE_NOT_HANDLED;
}
int err;
//Open the zip file
struct zip* pZip = zip_open(zipFile.c_str(), ZIP_CREATE|ZIP_CHECKCONS, &err);
if (pZip)
{
//Write the data to the stream
std::ostringstream strstream;
writeFile(objectType, object, rw, strstream, options);
char *data = new char[strstream.str().length()];
memcpy(data, strstream.str().c_str(), strstream.str().size());
WriteResult wr;
struct zip_source *zs = zip_source_buffer(pZip, data, strstream.str().length(), 0);
if (zs)
{
if (zip_add(pZip, zipEntry.c_str(), zs) != -1)
{
wr = WriteResult::FILE_SAVED;
}
else
{
osg::notify(osg::NOTICE) << "Couldn't add zip source " << std::endl;
wr = WriteResult::ERROR_IN_WRITING_FILE;
}
}
else
{
osg::notify(osg::NOTICE) << "Couldn't create zip source " << std::endl;
wr = WriteResult::ERROR_IN_WRITING_FILE;
}
zip_close(pZip);
delete[] data;
return wr;
}
else
{
osg::notify(osg::NOTICE) << "ReaderWriterZipFS::writeFile couldn't open zip " << zipFile << " full filename " << fullFileName << std::endl;
}
return WriteResult::FILE_NOT_HANDLED;
}
};
REGISTER_OSGPLUGIN(zipfs, ReaderWriterZipFS)
_______________________________________________
osg-users mailing list
[email protected]
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
