(I forgot to answer this) > Do you know, if there are plans for remote FileIO in version 2.0 in this direction?
No idea. However, you can get pretty far with what you have today. Implement your own istream that fetches files from remote, send that to the SceneFileHandler and also call osg::ImageFileHandler::the().setReadCB(...) with your own function. (that has access to some global or thread-local data which allows it to fetch the related picture-files' istreams as well, sending those to imagefilehandler::read() . ) I have done this for a simple vrml-file & pic inside a zip, so it should work in all cases, even if your data has to be fetched via pigeon. :) Oh, and my zip-datasource is lacking in that it can't load zips within zips (or from any another datasource), only from disk. (I think it is a limitation of the ZipIOs lib, but I can't really figure out why it has to be like that, since it uses an ifstream/istream internally to read. I tried 'fixing' it but coudn't get it to work.) Cheers, /Marcus Marcus Lindblom wrote: > Hi Christoph, > > Well, the loader setup is embarrasingly simple, so I can share that > wihout worries. See the attached bsFileSys.xxx files (it requires > boost::filesystem, but that is not necessary, merely a convenience). > The zip-datasource implements this for a zip-archive (to read from > directories within a zip-file) by using ZipIOs. The latter works > somewhat ok but I need to reconfigure our app a lot to make it truly > usable for us (we have some relative paths & includes in scripts that > mess things up quite a bit currently). > > What I really wanted was a good virtual file system with zip and > whatnot-support, but I haven't been able to find anything like that on > the net. I could also imagine using URI's and a xml-alike content > resolver framework, but neither of this is top-priority at all. > > The cache system is not really what you're looking for, I think. It > serves to speed up reloading previously read & optimized content as > gzipped .osb files (not a single cache, but individial files), for > faster subsequent runs (tristripping and sharing opts can take time, > esp in debug). I'm using boost-iostream there, to add a > gzip-(de)compressor to the original data-stream (usually from file) to > reduce disk-access, which is the limiting factor for us. It does not > work when loading the zipped .osb-files, as the > boost-gzip-decompressor cannot provide random access to the file. (I > probably could get away with decompressing to memory and reading from > that, but I need to figure out how large the decompressed file is.. > which I should be able to figure out by checking the streampos.. hmm > :) .. > > Anyway, since it, at the moment, doesn't work with .osg & zipped cache > files, it will need some work before it should be released. It also > depends on the boost::iostreams library, which may not be ideal, but > since OpenSG 2.0 seems to be boost-dependent anyway, it might not > matter much when push comes to shove. :) > > Cheers, > /Marcus > > Christoph Fünfzig wrote: >> Hi Marcus, >> >> thanks for your response. >> Your framework seems interesting. Your concern is to cache the loaded >> files >> into an OSB file? Can you perhaps send me the sources? >> >> Because of maintainance I am quite wary about large external libraries, >> so I am searching for small and easy to integrate ones. >> >> Do you know, if there are plans for remote FileIO in version 2.0 in >> this direction? >> >> Christoph >> >> >> Marcus Lindblom wrote: >> >>> Hi Christoph, >>> >>> Just some related comments: >>> >>> I would like something like that as well, and I have written some >>> very basic framework for our application which allows pluggable >>> content-providers (file/zip/ftp/http, etc). I've only implemented >>> file & zip (via the ZipIOs lib). However, I ran in to a problem in >>> my caching system, since the .osb-reader seemed to require >>> random-access to the stream (I get an exception 'no random access' >>> when trying to read osg-files from a zipistream.). VRML works fine >>> though. >>> >>> That might be something to watch out for, depending on what you're >>> trying to load. >>> >>> /Marcus >>> >>> P.S. I didn't mention this on the list at the time because it's just >>> a nice-to-have for us and I figured it might solve itself in 2.0 or >>> so. :) >>> >>> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Opensg-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/opensg-users >> > > ------------------------------------------------------------------------ > > #include "stdafx.h" > > #include <base/base/bsFileSys.h> > #include <base/base/bsFunctional.h> > #include <base/base/bsLog.h> > #include <base/base/bsSingleton.h> > #include <base/base/bsRefPtr.h> > > #include <boost/iostreams/device/back_inserter.hpp> > #include <boost/iostreams/copy.hpp> > #include <boost/io/ios_state.hpp> > > namespace fs = boost::filesystem; > namespace io = boost::iostreams; > > namespace mtc > { > namespace styx > { > namespace base > { > using std:: string; > > namespace > { > static LogDecorator s_log("FileSys"); > > // end anonymous namespace > } > > /// FileSysProxy which forwards fs requests to the current target > class FileSysProxy : > public DataSource > { > > public: > typedef std::deque<DataSource *> Sources; > > Sources m_sources; > > FileSysProxy() { m_sources.push_back(&m_stdsys); } > > protected: > virtual bool doRead(const std::string& id, Buffer& buf) > { > bool found = false; > for (Sources::iterator i = m_sources.begin(); > !found && i != m_sources.end(); ++i) { > std::string where = (*i)->find(id); > if (!where.empty()) { > found = (*i)->read(where, buf); > } > } > > return found; > } > > virtual std::string doFind(const std::string& id) > { > std::string str; > for (Sources::iterator i = m_sources.begin(); > str.empty() && i != m_sources.end(); ++i) { > str = (*i)->find(id); > } > > return str; > } > > virtual IStreamPtr doOpen(const std::string& id) > { > IStreamPtr is; > for (Sources::iterator i = m_sources.begin(); !is && i != > m_sources.end(); > ++i) { > std::string where = (*i)->find(id); > if (!where.empty()) { > is = (*i)->open(where); > } > } > > return is; > } > > private: > StdFileSys m_stdsys; > }; > > MTC_IMPLEMENT_NIFTYCOUNTER(FileSysInit); > > int GetLongevity(FileSysProxy* ) > { > return 3; > } > > typedef SingletonHolder<FileSysProxy, CreateUsingNew, SingletonWithLongevity> > FileSysProxyS; > > /////////////////////////////////////////// > // Buffer > Buffer::Buffer(size_t size) : > m_data(size) > { } > > Buffer::~Buffer() > { } > > std::istream & operator >>(std::istream & is, Buffer & buf) { > boost::io::ios_flags_saver ifs(is, std::ios::binary); > return is.read(buf.data(), boost::numeric_cast < std::streamsize > > (buf.size())); > } > > std::ostream & operator <<(std::ostream & os, Buffer & buf) { > boost::io::ios_flags_saver ifs(os, std::ios::binary); > return os.write(buf.data(), boost::numeric_cast < std::streamsize > > (buf.size())); > } > > /////////////////////////////////////////// > DataSource::~DataSource() > { } > > bool DataSource::read(const std::string& id, Buffer& buf) > { > return doRead(id, buf); > } > > > std::string DataSource::find(const std::string& id) > { > return doFind(id); > } > > DataSource::IStreamPtr DataSource::open(const std::string& id) > { > return doOpen(id); > } > > /////////////////////////////////////////// > // FileSys > DataSource& FileSys::the() > { > return FileSysProxyS::the(); > } > > void FileSys::addDataSource(DataSource* ds) > { > assert(ds); > FileSysProxyS::the().m_sources.push_front(ds); > } > > void FileSys::subDataSource(DataSource* ds) > { > assert(ds); > FileSysProxyS::type::Sources& s = FileSysProxyS::the().m_sources; > s.erase(std::find(s.begin(), s.end(), ds)); > } > > std::string FileSys::tempdir() > { > char* dir = 0; > if ((dir = getenv("STYXTMP")) != 0) { } else if ((dir = getenv("TMP")) != > 0) { } else if ((dir = getenv("TEMP")) != 0) { } else { > s_log.info("Unable to determine temp directory. Using '/tmp'."); > } > > return dir ? std::string(dir) : "/tmp"; > } > > /////////////////////////////////////////// > // StdFileSys > bool StdFileSys::doRead(const std::string& id, Buffer& buf) > { > if(IStreamPtr is = doOpen(id)) { > std::deque<char> data; > io::copy(*is, io::back_inserter(data)); > if (!is->fail()) { > buf.resize(data.size()); > std::copy(data.begin(), data.end(), buf.data()); > } else { > s_log.err("Error during read of file '%s'.", id.c_str()); > } > > return !is->fail(); > } else { > return false; > } > } > > std::string StdFileSys::doFind(const std::string& id) > { > /*$off*/ > try { > fs::path path(id, fs::native); > if (!path.is_complete()) { > path = fs::system_complete(path); > } > return path.native_file_string(); > } catch(fs::filesystem_error & e) { > s_log.err("Unable to build path for %s: %s", id.c_str(), e.what()); > return std::string(); > } > /*$on*/ > } > > DataSource::IStreamPtr StdFileSys::doOpen(const std::string& id) > { > IStreamPtr is; > > fs::path path(id, fs::native); > if (!path.is_complete()) { > std::string file = find(id); > if (file.empty()) { > return is; > } > path = fs::path(file, fs::native); > } > > try { > is.reset(new fs::ifstream(path, std::ios::binary)); > } catch (std::ios_base::failure &) { > is.reset(); > } > > if (is && !*is) { > is.reset(); > } > > if(!is) { > std::ostringstream os; > os << "Error opening " << id << " at " << path.string(); > s_log.err(os.str()); > } > > return is; > } > } > } > } > > ------------------------------------------------------------------------ > > #ifndef MENTICE_STYXBASE_BASE_FILESYS_H > #define MENTICE_STYXBASE_BASE_FILESYS_H > > > #include <base/base/bsNiftyCounter.h> > > namespace mtc > { > namespace styx > { > namespace base > { > /** simple byte-buffer wrapper > */ > class STYXBASE_DECLSPEC Buffer > { > > public: > Buffer(size_t size = 0); > ~ Buffer(); > > void resize(size_t len) { m_data.resize(len); } > char* data() { return size() > 0 ? &m_data.front() : 0; } > size_t size() { return m_data.size(); } > std::string toString() { > return std::string(data(), size()); > } > > private: > std::vector<char> m_data; > }; > > STYXBASE_DECLSPEC std::istream& operator>>(std::istream &is, Buffer &buf); > STYXBASE_DECLSPEC std::ostream& operator<<(std::ostream &os, Buffer &buf); > > // input data > class STYXBASE_DECLSPEC DataSource > { > public: > typedef boost::shared_ptr<std::istream> IStreamPtr; > > virtual ~DataSource(); > > /// Data load (loads into buffer) > bool read(const std::string& id, Buffer& buf); > > /// File lookup (returns path to file, may or may not be to local > filesystem) > std::string find(const std::string& id); > > /// Data access (opens file, returns null if unsuccessful) > IStreamPtr open(const std::string& id); > > protected: > virtual bool doRead(const std::string& id, Buffer& buf) = 0; > virtual std::string doFind(const std::string& id) = 0; > virtual IStreamPtr doOpen(const std::string& id) = 0; > }; > > /// FileSystem wrapper module > class STYXBASE_DECLSPEC FileSys : boost::noncopyable > { > > public: > /** returns a proxy fwd to the currect global filesys > (StdFileSys is default) > */ > static DataSource& the(); > > /// sets the global filesystem, null objects are ignored > static void addDataSource(DataSource* ds); > static void subDataSource(DataSource* ds); > > /// Retrieve temporary directory on this system > static std::string tempdir(); > }; > > /// simple FileSys which uses std::istream > class STYXBASE_DECLSPEC StdFileSys : > public DataSource > { > > protected: > /// Uses fopen(),fread() > virtual bool doRead(const std::string& id, Buffer& buf); > /// Returns the id wrapped as-is in a string > virtual std::string doFind(const std::string& id); > /// Opens file using ifstream > virtual IStreamPtr doOpen(const std::string& id); > }; > > /// @} > > MTC_DECLARE_NIFTYCOUNTER(FileSysInit, STYXBASE_DECLSPEC); > > ///////////////////////////////////////////////////////////////////////////// > > > } > } > } > > #endif > > ------------------------------------------------------------------------ > > #include <stdafx.h> > > #include <base/fw/fwZipDataSource.h> > > #include <base/script/scpScriptClassImpl.h> > #include <base/base/bsLog.h> > > #ifdef _DEBUG > #pragma message("Styx link to lib file: zipiosd.lib") > #pragma comment(lib, "zipiosd.lib") > #pragma message("Styx link to lib file: zlibd.lib") > #pragma comment(lib, "zlibd.lib") > #else > #pragma message("Styx link to lib file: zipios.lib") > #pragma comment(lib, "zipios.lib") > #pragma message("Styx link to lib file: zlib.lib") > #pragma comment(lib, "zlib.lib") > #endif > > namespace mtc > { > namespace styx > { > namespace fw > { > namespace > { > base::LogDecorator s_log("ZipDataSource"); > } > > MTC_IMPLEMENT_SCRIPTABLE(ZipDataSource) > > void ZipDataSource::initType(SCI& type) > { > type.attr("source").bind(setZipFile); > } > > ZipDataSource::ZipDataSource() > { } > > ZipDataSource::~ZipDataSource() > { } > > void ZipDataSource::setZipFile(const std::string& file) > { > // if (IStreamPtr isp = base::FileSys::the().open(file)) { > //m_istream = isp; > //m_zipfile.reset(new zipios::ZipFile(*m_istream, "zip://" + file + > '/')); > if(true) { > m_zipfile.reset(new zipios::ZipFile(file)); > if (m_zipfile->isValid()) { > s_log.info("Loaded zip %s with %i entries.", file.c_str(), > m_zipfile->size()); > return; > } > s_log.err("Zip file %s not valid.", file.c_str()); > m_zipfile.reset(); > m_istream.reset(); > } else { > s_log.err("Zip file %s not found.", file.c_str()); > } > > throw std::runtime_error("error loading zip file"); > } > > std::string ZipDataSource::doFind(const std::string& id) > { > if (m_zipfile) { > if (m_zipfile->getEntry(id) != 0) { > return m_zipfile->getName() + id; > } > } > > return std::string(); > } > > ZipDataSource::IStreamPtr ZipDataSource::doOpen(const std::string& id) > { > if (m_zipfile) { > std::string zipfile = id.substr(0, m_zipfile->getName().size()); > std::string zipentry = id.substr(m_zipfile->getName().size()); > > if (zipfile == m_zipfile->getName()) { > return IStreamPtr(m_zipfile->getInputStream(zipentry)); > } > } > return IStreamPtr(); > } > } > } > } > > ------------------------------------------------------------------------ > > #ifndef MENTICE_STYXBASE_FW_ZIPDATASOURCE_H > #define MENTICE_STYXBASE_FW_ZIPDATASOURCE_H > > #include <base/script/scpScriptObject.h> > #include <base/base/bsFileSys.h> > > #undef IGNORE > #include <zipios++/zipfile.h> > > namespace mtc > { > namespace styx > { > namespace fw > { > /// Calls update on updatables > class STYXBASE_DECLSPEC ZipDataSource : > public script::ScriptObject, > public base::StdFileSys > { > MTC_DECLARE_SCRIPTABLE(ZipDataSource, script::ScriptObject) > > public: > ZipDataSource(); > ~ ZipDataSource(); > > void setZipFile(const std::string& zipfile); > > protected: > virtual std::string doFind(const std::string& id); > virtual IStreamPtr doOpen(const std::string& id); > > private: > std::string m_filename; > IStreamPtr m_istream; > boost::scoped_ptr<zipios::ZipFile> m_zipfile; > > }; > > MTC_INITIALIZE_SCRIPTABLE(ZipDataSource) > } > } > } > > #endif > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Opensg-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/opensg-users > ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Opensg-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensg-users
