On Wed, Jun 9, 2010 at 10:04 AM, Aaron Sherman <a...@ajs.com> wrote:
> Has anyone begun to consider what kind of filesystem interface we want
> for things like sftp, Amazon S3, Google Storage and other remote
> storage possibilities? Is there any extant work out there, or should I
> just start spit-balling?
In the absence of anything forthcoming and a totally arbitrary sense
of urgency ;-) here's what I think I should do:
IO::FileSystems (S32) gives us some basics and the Path role also
provides some useful features.
I will start there and build an IO::FileSystems::VFS roughly like:
class IO::VFS is IO::FileSystems {
...
# Session data if applicable
has IO::VFS::Session $.session;
# Many methods take a $context which, if supplied
# will contain back-end specific data such as restart markers
# or payment model information. I'll probably define
# a role for the context parameter, but otherwise
# leave it pretty loose as a back-end specific structure.
# A simple operation that guarantees a round-trip to the filesystem
method nop($context?) { ... }
# list of sub-IO::VFS partitions/buckets/etc
method targets($context?) { ... }
method find_target($locator, $context?) { ... }
# Means of acquiring file-level access through a VFS
method find($locator, $enc = $.session.encoding, $context?) { ... }
method glob(Str $matcher, $enc = $.session.encoding, $context?) { ... }
# Like opening and writing to filehandle, but the operation is totally
# opaque and might be a single call, senfile or anything else.
# Note that this doesn't replace $obj.find($path).write(...)
method put($locator, $data, $enc = $.session.encoding, $context?) { ... }
# Atomic copy/rename, etc. are logically filesystem operations, even though
# they might have counterparts at the file level. The distinction being that
# at the filesystem level I never know nor care what the contents of the
# file are, I just ask for an operation to be performed on a given path.
method copy($from, $to, $enc = $.session.encoding, $context?) { ... }
method rename($from, $to, $enc = $.session.encoding, $context?) { ... }
method delete($locator, $enc = $.session.encoding, $context?) { ... }
# service-level ACLs if any
method acl($locator, $context?) { ... }
}
The general model I imagine would be something like:
my IO::VFS::S3 $s3 .= new();
$s3.session.connect($amazonlogininfo);
my $bucket = $s3.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::GoogleStorage $goog .= new();
$goog.session.connect($googlelogininfo);
my $bucket = $goog.find_target($bucket_name);
$bucket.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $bucket.find("quote.txt").uri;
or
my IO::VFS::SFTP $sftp .= new();
$sftp.session.connect(:host<storage>, :user<ajs>, :password<iforgotit>);
my $filesystem = $sftp.find_target("/tmp");
$filesystem.put("quote.txt", "Now is the time for all good men...\n");
say "URI: ", $filesystem.find("quote.txt").uri; # using sftp:...
Notice that everything after $obj.session.connect is identical except
for my choice of variable names. In fact, you shouldn't have to worry
about what storage back-end you're using as long as you have a valid
VFS handle. Really path names are the only thing that might trip you
up.
Thoughts?
I think that in order to do this, I'll need the following support
libraries which may or may not exist (I'll be looking into these):
IO::FileSystems
Path
HTTP (requires socket IO, MIME, base64, etc.)
Various crypto libs
I don't intend to provide a finished implementation of any of these
where they don't already exist (I may not even end up with a final
implementation of the VFS layer), but at least I'll get far enough
along that others who want to work on this will have a starting point,
and I'll want to at least have a test that fakes its way all the way
down to creating a remote file on all three services, even if most of
the operations involve passing on blobs of data generated by
equivalent calls in other languages.
--
Aaron Sherman
Email or GTalk: a...@ajs.com
http://www.ajs.com/~ajs
