Hi Christoph,
Well, the loader setup is embarrasingly simple, so I can share that
wihout worries. See the attached bsFileSys.xxx files (it requires
boost::filesystem, but that is not necessary, merely a convenience). The
zip-datasource implements this for a zip-archive (to read from
directories within a zip-file) by using ZipIOs. The latter works
somewhat ok but I need to reconfigure our app a lot to make it truly
usable for us (we have some relative paths & includes in scripts that
mess things up quite a bit currently).
What I really wanted was a good virtual file system with zip and
whatnot-support, but I haven't been able to find anything like that on
the net. I could also imagine using URI's and a xml-alike content
resolver framework, but neither of this is top-priority at all.
The cache system is not really what you're looking for, I think. It
serves to speed up reloading previously read & optimized content as
gzipped .osb files (not a single cache, but individial files), for
faster subsequent runs (tristripping and sharing opts can take time, esp
in debug). I'm using boost-iostream there, to add a gzip-(de)compressor
to the original data-stream (usually from file) to reduce disk-access,
which is the limiting factor for us. It does not work when loading the
zipped .osb-files, as the boost-gzip-decompressor cannot provide random
access to the file. (I probably could get away with decompressing to
memory and reading from that, but I need to figure out how large the
decompressed file is.. which I should be able to figure out by checking
the streampos.. hmm :) ..
Anyway, since it, at the moment, doesn't work with .osg & zipped cache
files, it will need some work before it should be released. It also
depends on the boost::iostreams library, which may not be ideal, but
since OpenSG 2.0 seems to be boost-dependent anyway, it might not matter
much when push comes to shove. :)
Cheers,
/Marcus
Christoph Fünfzig wrote:
Hi Marcus,
thanks for your response.
Your framework seems interesting. Your concern is to cache the loaded files
into an OSB file? Can you perhaps send me the sources?
Because of maintainance I am quite wary about large external libraries,
so I am searching for small and easy to integrate ones.
Do you know, if there are plans for remote FileIO in version 2.0 in this
direction?
Christoph
Marcus Lindblom wrote:
Hi Christoph,
Just some related comments:
I would like something like that as well, and I have written some very
basic framework for our application which allows pluggable
content-providers (file/zip/ftp/http, etc). I've only implemented file &
zip (via the ZipIOs lib). However, I ran in to a problem in my caching
system, since the .osb-reader seemed to require random-access to the
stream (I get an exception 'no random access' when trying to read
osg-files from a zipistream.). VRML works fine though.
That might be something to watch out for, depending on what you're
trying to load.
/Marcus
P.S. I didn't mention this on the list at the time because it's just a
nice-to-have for us and I figured it might solve itself in 2.0 or so. :)
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Opensg-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensg-users
#include "stdafx.h"
#include <base/base/bsFileSys.h>
#include <base/base/bsFunctional.h>
#include <base/base/bsLog.h>
#include <base/base/bsSingleton.h>
#include <base/base/bsRefPtr.h>
#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/io/ios_state.hpp>
namespace fs = boost::filesystem;
namespace io = boost::iostreams;
namespace mtc
{
namespace styx
{
namespace base
{
using std:: string;
namespace
{
static LogDecorator s_log("FileSys");
// end anonymous namespace
}
/// FileSysProxy which forwards fs requests to the current target
class FileSysProxy :
public DataSource
{
public:
typedef std::deque<DataSource *> Sources;
Sources m_sources;
FileSysProxy() { m_sources.push_back(&m_stdsys); }
protected:
virtual bool doRead(const std::string& id, Buffer& buf)
{
bool found = false;
for (Sources::iterator i = m_sources.begin();
!found && i != m_sources.end(); ++i) {
std::string where = (*i)->find(id);
if (!where.empty()) {
found = (*i)->read(where, buf);
}
}
return found;
}
virtual std::string doFind(const std::string& id)
{
std::string str;
for (Sources::iterator i = m_sources.begin();
str.empty() && i != m_sources.end(); ++i) {
str = (*i)->find(id);
}
return str;
}
virtual IStreamPtr doOpen(const std::string& id)
{
IStreamPtr is;
for (Sources::iterator i = m_sources.begin(); !is && i !=
m_sources.end();
++i) {
std::string where = (*i)->find(id);
if (!where.empty()) {
is = (*i)->open(where);
}
}
return is;
}
private:
StdFileSys m_stdsys;
};
MTC_IMPLEMENT_NIFTYCOUNTER(FileSysInit);
int GetLongevity(FileSysProxy* )
{
return 3;
}
typedef SingletonHolder<FileSysProxy, CreateUsingNew, SingletonWithLongevity>
FileSysProxyS;
///////////////////////////////////////////
// Buffer
Buffer::Buffer(size_t size) :
m_data(size)
{ }
Buffer::~Buffer()
{ }
std::istream & operator >>(std::istream & is, Buffer & buf) {
boost::io::ios_flags_saver ifs(is, std::ios::binary);
return is.read(buf.data(), boost::numeric_cast < std::streamsize >
(buf.size()));
}
std::ostream & operator <<(std::ostream & os, Buffer & buf) {
boost::io::ios_flags_saver ifs(os, std::ios::binary);
return os.write(buf.data(), boost::numeric_cast < std::streamsize >
(buf.size()));
}
///////////////////////////////////////////
DataSource::~DataSource()
{ }
bool DataSource::read(const std::string& id, Buffer& buf)
{
return doRead(id, buf);
}
std::string DataSource::find(const std::string& id)
{
return doFind(id);
}
DataSource::IStreamPtr DataSource::open(const std::string& id)
{
return doOpen(id);
}
///////////////////////////////////////////
// FileSys
DataSource& FileSys::the()
{
return FileSysProxyS::the();
}
void FileSys::addDataSource(DataSource* ds)
{
assert(ds);
FileSysProxyS::the().m_sources.push_front(ds);
}
void FileSys::subDataSource(DataSource* ds)
{
assert(ds);
FileSysProxyS::type::Sources& s = FileSysProxyS::the().m_sources;
s.erase(std::find(s.begin(), s.end(), ds));
}
std::string FileSys::tempdir()
{
char* dir = 0;
if ((dir = getenv("STYXTMP")) != 0) { } else if ((dir = getenv("TMP")) !=
0) { } else if ((dir = getenv("TEMP")) != 0) { } else {
s_log.info("Unable to determine temp directory. Using '/tmp'.");
}
return dir ? std::string(dir) : "/tmp";
}
///////////////////////////////////////////
// StdFileSys
bool StdFileSys::doRead(const std::string& id, Buffer& buf)
{
if(IStreamPtr is = doOpen(id)) {
std::deque<char> data;
io::copy(*is, io::back_inserter(data));
if (!is->fail()) {
buf.resize(data.size());
std::copy(data.begin(), data.end(), buf.data());
} else {
s_log.err("Error during read of file '%s'.", id.c_str());
}
return !is->fail();
} else {
return false;
}
}
std::string StdFileSys::doFind(const std::string& id)
{
/*$off*/
try {
fs::path path(id, fs::native);
if (!path.is_complete()) {
path = fs::system_complete(path);
}
return path.native_file_string();
} catch(fs::filesystem_error & e) {
s_log.err("Unable to build path for %s: %s", id.c_str(), e.what());
return std::string();
}
/*$on*/
}
DataSource::IStreamPtr StdFileSys::doOpen(const std::string& id)
{
IStreamPtr is;
fs::path path(id, fs::native);
if (!path.is_complete()) {
std::string file = find(id);
if (file.empty()) {
return is;
}
path = fs::path(file, fs::native);
}
try {
is.reset(new fs::ifstream(path, std::ios::binary));
} catch (std::ios_base::failure &) {
is.reset();
}
if (is && !*is) {
is.reset();
}
if(!is) {
std::ostringstream os;
os << "Error opening " << id << " at " << path.string();
s_log.err(os.str());
}
return is;
}
}
}
}
#ifndef MENTICE_STYXBASE_BASE_FILESYS_H
#define MENTICE_STYXBASE_BASE_FILESYS_H
#include <base/base/bsNiftyCounter.h>
namespace mtc
{
namespace styx
{
namespace base
{
/** simple byte-buffer wrapper
*/
class STYXBASE_DECLSPEC Buffer
{
public:
Buffer(size_t size = 0);
~ Buffer();
void resize(size_t len) { m_data.resize(len); }
char* data() { return size() > 0 ? &m_data.front() : 0; }
size_t size() { return m_data.size(); }
std::string toString() {
return std::string(data(), size());
}
private:
std::vector<char> m_data;
};
STYXBASE_DECLSPEC std::istream& operator>>(std::istream &is, Buffer &buf);
STYXBASE_DECLSPEC std::ostream& operator<<(std::ostream &os, Buffer &buf);
// input data
class STYXBASE_DECLSPEC DataSource
{
public:
typedef boost::shared_ptr<std::istream> IStreamPtr;
virtual ~DataSource();
/// Data load (loads into buffer)
bool read(const std::string& id, Buffer& buf);
/// File lookup (returns path to file, may or may not be to local
filesystem)
std::string find(const std::string& id);
/// Data access (opens file, returns null if unsuccessful)
IStreamPtr open(const std::string& id);
protected:
virtual bool doRead(const std::string& id, Buffer& buf) = 0;
virtual std::string doFind(const std::string& id) = 0;
virtual IStreamPtr doOpen(const std::string& id) = 0;
};
/// FileSystem wrapper module
class STYXBASE_DECLSPEC FileSys : boost::noncopyable
{
public:
/** returns a proxy fwd to the currect global filesys
(StdFileSys is default)
*/
static DataSource& the();
/// sets the global filesystem, null objects are ignored
static void addDataSource(DataSource* ds);
static void subDataSource(DataSource* ds);
/// Retrieve temporary directory on this system
static std::string tempdir();
};
/// simple FileSys which uses std::istream
class STYXBASE_DECLSPEC StdFileSys :
public DataSource
{
protected:
/// Uses fopen(),fread()
virtual bool doRead(const std::string& id, Buffer& buf);
/// Returns the id wrapped as-is in a string
virtual std::string doFind(const std::string& id);
/// Opens file using ifstream
virtual IStreamPtr doOpen(const std::string& id);
};
/// @}
MTC_DECLARE_NIFTYCOUNTER(FileSysInit, STYXBASE_DECLSPEC);
/////////////////////////////////////////////////////////////////////////////
}
}
}
#endif
#include <stdafx.h>
#include <base/fw/fwZipDataSource.h>
#include <base/script/scpScriptClassImpl.h>
#include <base/base/bsLog.h>
#ifdef _DEBUG
#pragma message("Styx link to lib file: zipiosd.lib")
#pragma comment(lib, "zipiosd.lib")
#pragma message("Styx link to lib file: zlibd.lib")
#pragma comment(lib, "zlibd.lib")
#else
#pragma message("Styx link to lib file: zipios.lib")
#pragma comment(lib, "zipios.lib")
#pragma message("Styx link to lib file: zlib.lib")
#pragma comment(lib, "zlib.lib")
#endif
namespace mtc
{
namespace styx
{
namespace fw
{
namespace
{
base::LogDecorator s_log("ZipDataSource");
}
MTC_IMPLEMENT_SCRIPTABLE(ZipDataSource)
void ZipDataSource::initType(SCI& type)
{
type.attr("source").bind(setZipFile);
}
ZipDataSource::ZipDataSource()
{ }
ZipDataSource::~ZipDataSource()
{ }
void ZipDataSource::setZipFile(const std::string& file)
{
// if (IStreamPtr isp = base::FileSys::the().open(file)) {
//m_istream = isp;
//m_zipfile.reset(new zipios::ZipFile(*m_istream, "zip://" + file +
'/'));
if(true) {
m_zipfile.reset(new zipios::ZipFile(file));
if (m_zipfile->isValid()) {
s_log.info("Loaded zip %s with %i entries.", file.c_str(),
m_zipfile->size());
return;
}
s_log.err("Zip file %s not valid.", file.c_str());
m_zipfile.reset();
m_istream.reset();
} else {
s_log.err("Zip file %s not found.", file.c_str());
}
throw std::runtime_error("error loading zip file");
}
std::string ZipDataSource::doFind(const std::string& id)
{
if (m_zipfile) {
if (m_zipfile->getEntry(id) != 0) {
return m_zipfile->getName() + id;
}
}
return std::string();
}
ZipDataSource::IStreamPtr ZipDataSource::doOpen(const std::string& id)
{
if (m_zipfile) {
std::string zipfile = id.substr(0, m_zipfile->getName().size());
std::string zipentry = id.substr(m_zipfile->getName().size());
if (zipfile == m_zipfile->getName()) {
return IStreamPtr(m_zipfile->getInputStream(zipentry));
}
}
return IStreamPtr();
}
}
}
}
#ifndef MENTICE_STYXBASE_FW_ZIPDATASOURCE_H
#define MENTICE_STYXBASE_FW_ZIPDATASOURCE_H
#include <base/script/scpScriptObject.h>
#include <base/base/bsFileSys.h>
#undef IGNORE
#include <zipios++/zipfile.h>
namespace mtc
{
namespace styx
{
namespace fw
{
/// Calls update on updatables
class STYXBASE_DECLSPEC ZipDataSource :
public script::ScriptObject,
public base::StdFileSys
{
MTC_DECLARE_SCRIPTABLE(ZipDataSource, script::ScriptObject)
public:
ZipDataSource();
~ ZipDataSource();
void setZipFile(const std::string& zipfile);
protected:
virtual std::string doFind(const std::string& id);
virtual IStreamPtr doOpen(const std::string& id);
private:
std::string m_filename;
IStreamPtr m_istream;
boost::scoped_ptr<zipios::ZipFile> m_zipfile;
};
MTC_INITIALIZE_SCRIPTABLE(ZipDataSource)
}
}
}
#endif
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Opensg-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensg-users
