Re: Filesystems and files [Was: Re: The obligation of free stuff: Google Storage]
This thread reminded me of something I'd posted a while ago:
---snip---
Date: Wed, 26 Nov 2008 14:23:11 +
From: Tim Bunce
To: Richard Hainsworth , perl6-language@perl.org
Subject: Re: Files, Directories, Resources, Operating Systems
On Wed, Nov 26, 2008 at 12:40:41PM +0100, Mark Overmeer wrote:
> We should focus on OS abstraction.
>
> [...] the design of this needs to be free from historical mistakes.
>
And avoid making too many new ones. There must be useful prior art around.
Java, for example, has a FileSystem abstraction java.nio.file.FileSystem
http://openjdk.java.net/projects/nio/javadoc/java/nio/file/FileSystem.html
which has been extended, based on leasons learnt, in the NIO.2 project
("JSR 203: More New I/O APIs for the JavaTM Platform ("NIO.2")
APIs for filesystem access, scalable asynchronous I/O operations,
socket-channel binding and configuration, and multicast datagrams.")
which enables things like being able to transparently treat a zip file
as a filesystem:
http://blogs.sun.com/rajendrag/entry/zip_file_system_provider_implementation
See http://javanio.info/filearea/nioserver/WhatsNewNIO2.pdf
Tim.
p.s. I didn't know any of that when I started to write this "look for
prior art" email, but a little searching turned up these examples.
I'm sure there are more in other realms, but NIO.2 certainly looks like a
rich source of good ideas derived from a wide range of experience.
---snip---
See http://javanio.info/filearea/nioserver/WhatsNewNIO2.pdf
plus http://java.sun.com/developer/technicalArticles/javase/nio/
There are many hard-learnt lessons in there that we can benefit from.
At the very least the APIs give us "things to think about".
Tim.
Re: Filesystems and files [Was: Re: The obligation of free stuff: Google Storage]
On Wed, Jun 30, 2010 at 4:29 AM, Richard Hainsworth wrote:
It is normally implied that a program already has a 'local' environment,
> including a 'local' filesystem. Thus the syntax
> my $fn = open('/path/to/directory/filename', :r) or die $!;
> implies a local file sytem.
>
> The idea of an implied local system suggests an implied local environment.
> The contents of %*ENV and @*INC seem to be assumed to be local, thought this
> is not specified. Given the development of the internet, this is an
> assumption I think should be made implicit, as well as the mechanism for
> adding remote resources via paths through a network.
>
> Would it make sense to define $*FS as the implied local file system, and
> thus that a bare 'open' is sugar for
> my $fh = $*FS.open('/path/to/directory/filename', :r);
>
Yep, that makes perfect sense. Once I have a working VFS object that could
be stored in there, that's probably the best way to go, unless someone
proposes another way between now and then.
--
Aaron Sherman
Email or GTalk: a...@ajs.com
http://www.ajs.com/~ajs
Re: Filesystems and files [Was: Re: The obligation of free stuff: Google Storage]
On Wed, Jun 30, 2010 at 10:29 AM, Richard Hainsworth
wrote:
> Would it make sense to define $*FS as the implied local file system, and
> thus that a bare 'open' is sugar for
> my $fh = $*FS.open('/path/to/directory/filename', :r);
>
> This then means that there is an implicit
> $*FS.connect();
> that makes the local system available to the program.
>
> I wonder whether this would also be a way of unifying the program interface
> for difference Operating Systems, in that a program running on a *nix
> machine would have $*FS of type IO::Filesystem::POSIX, while $*FS for a
> Windows machine would have type IO::Filesystem::Windows, etc.
>
> Then it would be possible, as Aaron has suggested to have
> my $remote-fs = IO::Filesystem::Google.connect(%args);
> my $fh = $remote-fs.open($path-or-url, :r);
>
> and then treat $fh as specified elsewhere.
>
> Morover, it would then be possible to do
> $*FS = $remote-fs;
>
> I would propose that this sort of flexibility would be useful for programs
> that are embedded in other virtual environments, such as browser plugins, or
> programs intended to run on thin clients that do not have their own
> filesystems.
>
I like this idea. It's flexible and extendable but has sane defaults.
Leon
Re: Filesystems and files [Was: Re: The obligation of free stuff: Google Storage]
Sounds like a sound generalization to make. On Wed, Jun 30, 2010 at 1:29 AM, Richard Hainsworth wrote: > This then means that there is an implicit > $*FS.connect(); > that makes the local system available to the program. "mount" is the jargon to make a filesystem available, looking backwards, though perhaps "connect" is more accurate going forwards. -y
Filesystems and files [Was: Re: The obligation of free stuff: Google Storage]
If I might offer a late viewpoint after reading the Aaron's expanded
email (attached below).
When originally I suggested using 'open' instead of 'connect', the aim
was to keep consistency with the paradigm of files on the local system.
However, as Aaron's post suggests, when dealing with remote 'files',
there is an additional layer of functionality that must be introduced,
namely the need to 'connect' to the filesystem so that files on it can
be opened. Thus it is wrong to conflate 'open' with 'connect'.
It is normally implied that a program already has a 'local' environment,
including a 'local' filesystem. Thus the syntax
my $fn = open('/path/to/directory/filename', :r) or die $!;
implies a local file sytem.
The idea of an implied local system suggests an implied local
environment. The contents of %*ENV and @*INC seem to be assumed to be
local, thought this is not specified. Given the development of the
internet, this is an assumption I think should be made implicit, as well
as the mechanism for adding remote resources via paths through a network.
Would it make sense to define $*FS as the implied local file system, and
thus that a bare 'open' is sugar for
my $fh = $*FS.open('/path/to/directory/filename', :r);
This then means that there is an implicit
$*FS.connect();
that makes the local system available to the program.
I wonder whether this would also be a way of unifying the program
interface for difference Operating Systems, in that a program running on
a *nix machine would have $*FS of type IO::Filesystem::POSIX, while $*FS
for a Windows machine would have type IO::Filesystem::Windows, etc.
Then it would be possible, as Aaron has suggested to have
my $remote-fs = IO::Filesystem::Google.connect(%args);
my $fh = $remote-fs.open($path-or-url, :r);
and then treat $fh as specified elsewhere.
Morover, it would then be possible to do
$*FS = $remote-fs;
I would propose that this sort of flexibility would be useful for
programs that are embedded in other virtual environments, such as
browser plugins, or programs intended to run on thin clients that do not
have their own filesystems.
Another possibility would be to have
my $windows-from-linux = IO::Filesystem::Windows.connect(%args);
my $linux-system = $*FS;
$*FS = $windows-from-linux;
And then the files on a dual boot system can be accessed by the program.
On 06/21/10 02:35, Aaron Sherman wrote:
First off, I again have to caution that this thread is conflating
"open" with filesystem interaction. While open is one of many ways of
interacting with a filesystem, it isn't even remotely sufficient (nor
my immediate focus). One can ask for and modify filesystem metadata,
security information, and so on as well as that for individual objects
within the filesystem (which in the POSIX model is mostly files and
directories). In a traditional POSIX/Unix model, programs (other than
key OS utilities) don't usually do much with the structure of the
filesystem. That's meant as an interactive task for an admin. However,
in building a cloud-storage aware VFS-layer, managing the filesystem
in terms of layout, allocation, security (access methods,
authorization and authentication), payment models, and many other
features are expected to be embodied in the access model. Just as an
example, choosing and laying out what Amazon calls "buckets" is the
equivalent of partitioning. That does need an interface.
Now, we can just translate the Python bindings for Google Storage (and
I believe there are already Perl 5 bindings for Amazon S3), but my
inclination is to build a generic VFS that can handle POSIX-like
filesystems as well as everything else from Windows/Mac specific
features to full-blown cloud storage to more user-oriented storage
options (Dropbox comes to mind).
Every addressable storage model which could be treated as a filesystem
should have a place in the Perl 6 VFS.
Now, as to the question of overloading "open"... I'm not sure. I mean,
it's pretty easy to say:
URI.new($path).open(:ro)
or
open(URI.new($path), :ro)
When what you want is a VFS object, and I kind of like the idea of the
standard open on a string having POSIX semantics.
Now, to your question, C.J.
On Fri, Jun 18, 2010 at 3:03 PM, C.J. Adams-Collier
wrote:
Define "opening a file" for me. Is it something that's associated with a
filehandle, by definition? Do TCP sockets count?
Opening a file isn't a well defined operation. You have to be more specific.
In your question you're conflating the evaluation of a filesystem
namespace token (which is one of many possible modes of filesystem
interaction), returning a filehandle object that represents access to
the named object with evaluation of a socket namespace token and
returning a similar filehandle object that represents that object.
There are, of course crossovers (filesystem pipes) and other
operations that yield filehandle objects (various IPC operations that
aren't exactly sockets, for exampl
Re: The obligation of free stuff: Google Storage
First off, I again have to caution that this thread is conflating "open" with filesystem interaction. While open is one of many ways of interacting with a filesystem, it isn't even remotely sufficient (nor my immediate focus). One can ask for and modify filesystem metadata, security information, and so on as well as that for individual objects within the filesystem (which in the POSIX model is mostly files and directories). In a traditional POSIX/Unix model, programs (other than key OS utilities) don't usually do much with the structure of the filesystem. That's meant as an interactive task for an admin. However, in building a cloud-storage aware VFS-layer, managing the filesystem in terms of layout, allocation, security (access methods, authorization and authentication), payment models, and many other features are expected to be embodied in the access model. Just as an example, choosing and laying out what Amazon calls "buckets" is the equivalent of partitioning. That does need an interface. Now, we can just translate the Python bindings for Google Storage (and I believe there are already Perl 5 bindings for Amazon S3), but my inclination is to build a generic VFS that can handle POSIX-like filesystems as well as everything else from Windows/Mac specific features to full-blown cloud storage to more user-oriented storage options (Dropbox comes to mind). Every addressable storage model which could be treated as a filesystem should have a place in the Perl 6 VFS. Now, as to the question of overloading "open"... I'm not sure. I mean, it's pretty easy to say: URI.new($path).open(:ro) or open(URI.new($path), :ro) When what you want is a VFS object, and I kind of like the idea of the standard open on a string having POSIX semantics. Now, to your question, C.J. On Fri, Jun 18, 2010 at 3:03 PM, C.J. Adams-Collier wrote: > Define "opening a file" for me. Is it something that's associated with a > filehandle, by definition? Do TCP sockets count? Opening a file isn't a well defined operation. You have to be more specific. In your question you're conflating the evaluation of a filesystem namespace token (which is one of many possible modes of filesystem interaction), returning a filehandle object that represents access to the named object with evaluation of a socket namespace token and returning a similar filehandle object that represents that object. There are, of course crossovers (filesystem pipes) and other operations that yield filehandle objects (various IPC operations that aren't exactly sockets, for example). Now, if you want to unify some of that territory, you can build a VFS layer. Today, that's often done via URIs, just because they're handy for Identifying Universal Resources, but what happens when you call open (or equivalent) on such a token is still an open question, and I didn't seek to answer it in this thread. Franky, I don't think that it's something that SHOULD be answered prior to building the VFS layer itself, because that layer might dictate some design decisions, but my high level impulse is to say that open on a VFS token (be it a URI or some othe complex data) will yield a VFS-back-end specific "handle". Such a handle would likely "do" IO::Handle and friends as well as something VFS-specific. -- Aaron Sherman Email or GTalk: a...@ajs.com http://www.ajs.com/~ajs
Re: The obligation of free stuff: Google Storage
On Thu, 2010-06-10 at 21:17 -0400, Brandon S. Allbery KF8NH wrote: > On Jun 10, 2010, at 07:22 , Leon Timmermans wrote: > > I agree it should be similar to normal FS interactoin to make matters > > as intuitive as possible, but I horrified by the idea of overloading > > open() that way. That's a PHP mistake I wouldn't like seeing repeated. > > If you want open to do something that's not really opening a file, you > > should be explicit about it. > > > Out of curiosity and at a tangent: do you feel the same about FUSE? > Gnome-VFS? > Ooh! I love tangents! Define "opening a file" for me. Is it something that's associated with a filehandle, by definition? Do TCP sockets count? gpg: NOTE: signature key D5B75EB5 expired Sat 01 Mar 2008 01:57:54 PM PST gpg: NOTE: signature key A8A3A3DF expired Thu 11 Jun 2009 11:25:49 AM PDT gpg: NOTE: signature key 11A281C1 expired Sat 21 Jan 2006 06:30:28 AM PST gpg: NOTE: signature key 4460CBCA expired Tue 04 Jan 2005 05:53:23 PM PST gpg: NOTE: signature key 42FE143A expired Sun 20 May 2007 03:27:18 PM PDT gpg: NOTE: signature key F25370CB expired Wed 27 May 2009 12:02:01 AM PDT gpg: NOTE: signature key 42B5A9F6 expired Wed 15 Jul 2009 04:47:12 PM PDT gpg: NOTE: signature key A7C89464 expired Wed 15 Jul 2009 04:46:53 PM PDT gpg: NOTE: signature key C0050B4F expired Wed 15 Jul 2009 04:47:36 PM PDT gpg: NOTE: signature key 38693AFC expired Wed 20 Sep 2000 09:02:20 PM PDT gpg: NOTE: signature key B59BD712 expired Tue 17 Mar 2009 10:57:52 AM PDT gpg: NOTE: signature key AA208D9E expired Mon 31 May 2010 04:00:35 PM PDT gpg: NOTE: signature key 4E26A87D expired Sun 22 Nov 2009 03:46:42 AM PST C.J. Adams-Collier KF7BMP signature.asc Description: This is a digitally signed message part
Re: The obligation of free stuff: Google Storage
On 06/10/2010 05:07 AM, Mark J. Reed wrote:
On Thursday, June 10, 2010, Leon Timmermans wrote:
I agree it should be similar to normal FS interactoin to make matters
as intuitive as possible, but I horrified by the idea of overloading
open() that way
But open is already overloaded in p5, with pipes etc. We don't want
to repeat the mistakes of the past, and the fact that open(FH, $foo)
could run an arbitrary shell command was arguably a mistake, but
transparent access to storage where possible is the way to go.
We can provide a way to limit the behavior when you want - a pragma,
option to open(), or just sticking "file://" in front of a value to
force its interpretation as a plain pathname - but even then, who
knows what Fuse filesystem might be mounted and doing strange things
with your innocuous-looking file access? I'd rather have that
flexibility directly supported in the language.
Of course, different types of storage have different features; there's
no completely unified interface. But to the extent that a cloud disk
system or document database functions as a collection of data blobs
accessed by a pathlike key, enabling the standard filesystem access
pattern for it (in addition to whatever specific functionality it
needs) makes sense, IMHO.
Doesn't it make sense to just make it a method for the class/role?
my Amazon $fn .= open("$path-to-input-file-location/$file-name", :r) or
die $!;
--
Kevan Benson
A-1 Networks
Re: The obligation of free stuff: Google Storage
On Thu, Jun 10, 2010 at 2:07 PM, Mark J. Reed wrote: > But open is already overloaded in p5, with pipes etc. We don't want > to repeat the mistakes of the past, and the fact that open(FH, $foo) > could run an arbitrary shell command was arguably a mistake, but > transparent access to storage where possible is the way to go. I think that 3 argument open fixed that issue for perl 5: with it you would have to be way more explicit about wanting to open. I think something similar would work for me in this case too, a flag like :remote. > We can provide a way to limit the behavior when you want - a pragma, > option to open(), or just sticking "file://" in front of a value to > force its interpretation as a plain pathname - but even then, who > knows what Fuse filesystem might be mounted and doing strange things > with your innocuous-looking file access? I'd rather have that > flexibility directly supported in the language. For me that sounds like the wrong default, mostly because I'm worried about remote file inclusion vulnerabilities. > Of course, different types of storage have different features; there's > no completely unified interface. But to the extent that a cloud disk > system or document database functions as a collection of data blobs > accessed by a pathlike key, enabling the standard filesystem access > pattern for it (in addition to whatever specific functionality it > needs) makes sense, IMHO. I fully agree with that. Leon
Re: The obligation of free stuff: Google Storage
On Jun 10, 2010, at 07:22 , Leon Timmermans wrote: I agree it should be similar to normal FS interactoin to make matters as intuitive as possible, but I horrified by the idea of overloading open() that way. That's a PHP mistake I wouldn't like seeing repeated. If you want open to do something that's not really opening a file, you should be explicit about it. Out of curiosity and at a tangent: do you feel the same about FUSE? Gnome-VFS? -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu electrical and computer engineering, carnegie mellon universityKF8NH PGP.sig Description: This is a digitally signed message part
Re: The obligation of free stuff: Google Storage
On Thursday, June 10, 2010, Leon Timmermans wrote: > I agree it should be similar to normal FS interactoin to make matters > as intuitive as possible, but I horrified by the idea of overloading > open() that way But open is already overloaded in p5, with pipes etc. We don't want to repeat the mistakes of the past, and the fact that open(FH, $foo) could run an arbitrary shell command was arguably a mistake, but transparent access to storage where possible is the way to go. We can provide a way to limit the behavior when you want - a pragma, option to open(), or just sticking "file://" in front of a value to force its interpretation as a plain pathname - but even then, who knows what Fuse filesystem might be mounted and doing strange things with your innocuous-looking file access? I'd rather have that flexibility directly supported in the language. Of course, different types of storage have different features; there's no completely unified interface. But to the extent that a cloud disk system or document database functions as a collection of data blobs accessed by a pathlike key, enabling the standard filesystem access pattern for it (in addition to whatever specific functionality it needs) makes sense, IMHO. . -- Mark J. Reed
Re: The obligation of free stuff: Google Storage
On Thu, Jun 10, 2010 at 9:15 AM, Richard Hainsworth
wrote:
> Ideally [at least, what I would like], managing a file on a remote resource
> should be the same as managing one locally, eg.
>
> my Amazon $fn = open("$path-to-input-file-location/$file-name", :r) or die
> $!;
> for $fn.readlines { };
> $fn.close;
>
> my Google $fn = open("$path-to-output-file-location/$file-name", :w) or die
> $!;
> for @lots-of-data -> $item { $fn.say: $item };
> $fn.close;
>
I agree it should be similar to normal FS interactoin to make matters
as intuitive as possible, but I horrified by the idea of overloading
open() that way. That's a PHP mistake I wouldn't like seeing repeated.
If you want open to do something that's not really opening a file, you
should be explicit about it.
Leon
Re: The obligation of free stuff: Google Storage
Ideally [at least, what I would like], managing a file on a remote
resource should be the same as managing one locally, eg.
my Amazon $fn = open("$path-to-input-file-location/$file-name", :r) or
die $!;
for $fn.readlines { };
$fn.close;
my Google $fn = open("$path-to-output-file-location/$file-name", :w) or
die $!;
for @lots-of-data -> $item { $fn.say: $item };
$fn.close;
If it is possible to make file on a cloud system as easy to manipulate
as a local file, that will aid the acceptibility of perl6.
Hence, the use of 'open', 'close', 'say', 'readlines', rather than
'connect', 'put', 'get'.
The issue - it seems to me - is that the connection between a computer
and its local filesystem has such a low probability of failure (viz.,
disk failure or disk full) that is can normally be ignored, but there is
relatively high probability of a connection failure between the computer
and a remote site and so it would be normal to have exception handling
in the code.
That means the designer of the API should indicate how these exceptions
should be handled so that a program doesnt hang in a loop that is
expecting data from a remote resource whose server has disconnected.
On 06/10/2010 03:24 AM, Aaron Sherman wrote:
On Wed, Jun 9, 2010 at 10:04 AM, Aaron Sherman wrote:
Has anyone begun to consider what kind of filesystem interface we want
for things like sftp, Amazon S3, Google Storage and other remote
storage possibilities? Is there any extant work out there, or should I
just start spit-balling?
In the absence of anything forthcoming and a totally arbitrary sense
of urgency ;-) here's what I think I should do:
IO::FileSystems (S32) gives us some basics and the Path role also
provides some useful features.
I will start there and build an IO::FileSystems::VFS roughly like:
class IO::VFS is IO::FileSystems {
...
# Session data if applicable
has IO::VFS::Session $.session;
# Many methods take a $context which, if supplied
# will contain back-end specific data such as restart markers
# or payment model information. I'll probably define
# a role for the context parameter, but otherwise
# leave it pretty loose as a back-end specific structure.
# A simple operation that guarantees a round-trip to the filesystem
method nop($context?) { ... }
# list of sub-IO::VFS partitions/buckets/etc
method targets($context?) { ... }
method find_target($locator, $context?) { ... }
# Means of acquiring file-level access through a VFS
method find($locator, $enc = $.session.encoding, $context?) { ... }
method glob(Str $matcher, $enc = $.session.encoding, $context?) { ... }
# Like opening and writing to filehandle, but the operation is totally
# opaque and might be a single call, senfile or anything else.
# Note that this doesn't replace $obj.find($path).write(...)
method put($locator, $data, $enc = $.session.encoding, $context?) { ... }
# Atomic copy/rename, etc. are logically filesystem operations, even though
# they might have counterparts at the file level. The distinction being that
# at the filesystem level I never know nor care what the contents of the
# file are, I just ask for an operation to be performed on a given path.
method copy($from, $to, $enc = $.session.encoding, $context?) { ... }
method rename($from, $to, $enc = $.session.encoding, $context?) { ... }
method delete($locator, $enc = $.session.encoding, $context?) { ... }
# service-level ACLs if any
method acl($locator, $context?) { ... }
}
The general model I imagine would be something like:
my IO::VFS::S3 $s3 .= new();
$s3.session.connect($amazonlogininfo);
my $bucket = $s3.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::GoogleStorage $goog .= new();
$goog.session.connect($googlelogininfo);
my $bucket = $goog.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::SFTP $sftp .= new();
$sftp.session.connect(:host, :user, :password);
my $filesystem = $sftp.find_target("/tmp");
$filesystem.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $filesystem.find("quote.txt").uri; # using sftp:...
Notice that everything after $obj.session.connect is identical except
for my choice of variable names. In fact, you shouldn't have to worry
about what storage back-end you're using as long as you have a valid
VFS handle. Really path names are the only thing that might trip you
up.
Thoughts?
I think that in order to do this, I'll need the following support
libraries which may or may not exist (I'll be looking into these):
IO::FileSystems
Path
HTTP (requires socket IO, MIME, base64, etc.)
Various crypto libs
I don't intend to provide a finished implementation of any of these
where they don't already exi
Re: The obligation of free stuff: Google Storage
On Wed, Jun 9, 2010 at 10:04 AM, Aaron Sherman wrote:
> Has anyone begun to consider what kind of filesystem interface we want
> for things like sftp, Amazon S3, Google Storage and other remote
> storage possibilities? Is there any extant work out there, or should I
> just start spit-balling?
In the absence of anything forthcoming and a totally arbitrary sense
of urgency ;-) here's what I think I should do:
IO::FileSystems (S32) gives us some basics and the Path role also
provides some useful features.
I will start there and build an IO::FileSystems::VFS roughly like:
class IO::VFS is IO::FileSystems {
...
# Session data if applicable
has IO::VFS::Session $.session;
# Many methods take a $context which, if supplied
# will contain back-end specific data such as restart markers
# or payment model information. I'll probably define
# a role for the context parameter, but otherwise
# leave it pretty loose as a back-end specific structure.
# A simple operation that guarantees a round-trip to the filesystem
method nop($context?) { ... }
# list of sub-IO::VFS partitions/buckets/etc
method targets($context?) { ... }
method find_target($locator, $context?) { ... }
# Means of acquiring file-level access through a VFS
method find($locator, $enc = $.session.encoding, $context?) { ... }
method glob(Str $matcher, $enc = $.session.encoding, $context?) { ... }
# Like opening and writing to filehandle, but the operation is totally
# opaque and might be a single call, senfile or anything else.
# Note that this doesn't replace $obj.find($path).write(...)
method put($locator, $data, $enc = $.session.encoding, $context?) { ... }
# Atomic copy/rename, etc. are logically filesystem operations, even though
# they might have counterparts at the file level. The distinction being that
# at the filesystem level I never know nor care what the contents of the
# file are, I just ask for an operation to be performed on a given path.
method copy($from, $to, $enc = $.session.encoding, $context?) { ... }
method rename($from, $to, $enc = $.session.encoding, $context?) { ... }
method delete($locator, $enc = $.session.encoding, $context?) { ... }
# service-level ACLs if any
method acl($locator, $context?) { ... }
}
The general model I imagine would be something like:
my IO::VFS::S3 $s3 .= new();
$s3.session.connect($amazonlogininfo);
my $bucket = $s3.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::GoogleStorage $goog .= new();
$goog.session.connect($googlelogininfo);
my $bucket = $goog.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::SFTP $sftp .= new();
$sftp.session.connect(:host, :user, :password);
my $filesystem = $sftp.find_target("/tmp");
$filesystem.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $filesystem.find("quote.txt").uri; # using sftp:...
Notice that everything after $obj.session.connect is identical except
for my choice of variable names. In fact, you shouldn't have to worry
about what storage back-end you're using as long as you have a valid
VFS handle. Really path names are the only thing that might trip you
up.
Thoughts?
I think that in order to do this, I'll need the following support
libraries which may or may not exist (I'll be looking into these):
IO::FileSystems
Path
HTTP (requires socket IO, MIME, base64, etc.)
Various crypto libs
I don't intend to provide a finished implementation of any of these
where they don't already exist (I may not even end up with a final
implementation of the VFS layer), but at least I'll get far enough
along that others who want to work on this will have a starting point,
and I'll want to at least have a test that fakes its way all the way
down to creating a remote file on all three services, even if most of
the operations involve passing on blobs of data generated by
equivalent calls in other languages.
--
Aaron Sherman
Email or GTalk: a...@ajs.com
http://www.ajs.com/~ajs
The obligation of free stuff: Google Storage
On a lark, I submitted a request to Google for membership in the Google Storage beta on the basis of doing something virtual filesystemish for Perl 6. The bastards gave me an account, so now I feel as if I should do something. Has anyone begun to consider what kind of filesystem interface we want for things like sftp, Amazon S3, Google Storage and other remote storage possibilities? Is there any extant work out there, or should I just start spit-balling? -- Aaron Sherman Email or GTalk: a...@ajs.com http://www.ajs.com/~ajs
