This and other RFCs are available on the web at http://dev.perl.org/rfc/ =head1 TITLE Modify open() to support FileObjects and Extensibility =head1 VERSION Maintainer: Nathan Wiger <[EMAIL PROTECTED]> Date: 04 Aug 2000 Last-Modified: 11 Aug 2000 Version: 3 Mailing List: [EMAIL PROTECTED] Number: 14 =head1 ABSTRACT Currently, C<open()>, C<opendir()>, C<sysopen()>, and other file open functions are given handle arguments, whose values are twiddled if a filehandle can be created: open(HANDLE, "<$filename"); open PIPE, "| $program"; open my $fh, "<$filename"; opendir DIR, $dirname; sysopen(HANDLE, $filename, O_RDWR|O_CREAT, 0666); There are several problems with this approach: 1. The calling style is uncharacteristic of other Perl funcs 2. There is no way to support a list of return values 3. There is no way to overload or extend them In order to make these functions more consistent with other constructor like functions (i.e. new(), etc), they should be changed to instead return B<first-class fileobjects>: $fo = open "<$filename" or die; $po = open "|$program" or die; $do = open dir $dirname or die; $so = open sys $filename, O_RDWR|O_CREAT, 0666; This would make these functions more internally consistent within Perl, as well as allowing for the power of true B<fileobjects> and an extensibile C<open()>. =head1 DESCRIPTION =head2 Overview First, this RFC assumes that B<fileobjects> will be $ single-whatzitz (thanks Tom) types, which seems to have reached an overall informal consensus. As many have observed, the current filehandle mechanism is insufficient and largely "a historial accident". The goal of the redesign of file handles into full-fledged B<fileobjects> is to make them as flexible and powerful as other objects within Perl, which still retaining a means to interact with them simply. Since we are redesigning filehandles to be true B<fileobjects>, we should revise their constructor functions as well, returning B<fileobjects> and providing extensibility. As shown above, in the simplest case this would change the C<open()> function to: $fo = open $filename or die; If successful, C<open()> and its relatives will return B<fileobjects>. On failure, they will return undef. This still allows the user to test the return value of C<open()> (as shown above) by checking for a "true" (C<fileobject>) or "false" (undef) condition. =head2 New Syntax of C<open()> The syntax of the C<open()> function would be changed as follows: $fileobject, [ @params ] = open [ $handler ] $file, [ @args ]; Let's examine this syntax more closely: $fileobject - Replacement object for current filehandles. @params - Optional parameters that may be returned in a list context. These may be things such as the owner for a true file, or the content-type for a web document. $handler - The class from which to load the appropriate file methods, the default being "file". This is not really a class, but rather a registered handler. This handler type is bound to a class given by the user, or taken from a set of core methods. Think Apache handler. $file - File to open. This might be a real file or directory, but might also be a website, port for a socket, ftp server, ipc pipe, rpc client, and so on. @args - Optional arguments to pass to the handler's C<open>. The C<open> function, as I propose it, is an overloaded and extensible function that differs from other constructors in that it returns a valid B<fileobject>. This object can then be used in C<read>, C<print>, and other such file functions. As such, development of classes that can handle objects on new platforms (ex: Mac, Palm) and handle new types of files (XML documents, etc) is much quicker. Plus, these modules are much lighter weight since they don't have to reinvent the wheel every time. =head2 Simple Scalar Form In the simplest, "looks like Perl 5" form, C<open()> can take one file parameter, which is then opened per the descriptor provided and the corresponding B<fileobject> returned. Here are some examples (note that C<my> has been left out for clarity): # Read from a file $fo = open "</etc/passwd" or die; print while (<$fo>); close $fo; # Write a file to a pipe $mailpipe = open "|/usr/lib/sendmail" or die; ($motd, $owner) = open "</etc/motd"; # return owner in list die unless $owner == 0; # owner not root while (<$motd>) { print $mailpipe; } close $motd; # Go fork yourself ($myself, $pid) = open "-|" or exec 'ls'; # return PID in list print while (<$myself>); close $myself; # not myself anymore, hah! ;-) In addition, the C<$file> argument becomes optional in this new syntax. If not supplied, it defaults to C<$_>, making it consistent with other Perl functions: for (@filenames) { my $fo = open; push @open_handles, $fo; } # ... stuff happens ... for (@open_handles) { close; } Perhaps this specific example is ugly (and useless), but there are probably other situations in which one could take advantage of this. =head2 True First-Class FileObjects One major limitation of Perl's current filehandles is that they are bareword scalars, with no object properties or power. The redesign of simple filehandles into first-class B<fileobjects> allows us to give them full object-oriented power, while still allowing them to be used in a simple manner as shown above. Each object can contain methods to allow us to access features of that B<fileobject> much more efficiently. Here are some proposed default accessor methods of B<fileobjects>. Each of these would return the appropriate value, or undef if not available. This is a brief listing; the intent would be to support all of the current C<FileHandle> methods and then some. In a string context, the filename as opened (including descriptors like | and <) is returned. $fo->STRING - Same as $fo->filename $fo->filename - Name of the file, web document, port, etc $fo->type - One of 'pipe', 'file', 'ipc', etc, ala want() $fo->mode - Way the file was opened (|,<,>+,etc) $fo->fileno - System file number $fo->dup - Returns a duplicate of the current B<fileobject> $fo->pid - Return current PID of the process (if pipe/fork) In addition, these functions would allow you to modify key elements of the B<fileobject>: $fo->autoflush - Sets buffer flushing policy $fo->untaint - Removes tainting from that data source $fo->options - Some syscalls, like C<socket()>, allow you to set options which affect the handling of C<$fo> If we decide that B<fileobjects> should be persistent across C<close()> operations, we could define the following functions: $fo->open - Object methods to open/close C<fileobjects> $fo->close $fo->is_open - 1 or undef, depending on the state of the object $fo->is_closed Why would a B<fileobject> be persistent across C<close()> operations? If it contains lots of properties, it may be a waste if we simply want to close it to make sure buffers are flushed or bandwidth is not wasted on TCP connections. We could use C<close()> to flush buffers and tidy things up, but not destroy the object until the end of the script or C<undef $fo> was called (similar to C<FileHandle>). =head2 Extensible Handler Bindings In addition to the standard file form, C<open()> can also now take an optional handler name (the default is "file") from which to load the appropriate methods. This gives us easy access to methods that open Directory, Socket, HTTP, FTP, or other types of files, meaning we no longer have to start from the ground up every time we want to open a new type of "file". Here, C<open handlers> work much like Apache handlers. You specify a certain type of handler (for example, "file", "dir", "http", "ftp", and so on) as what should be used to operate on the given argument. The only requirement is that the handler return one of two values: 1. A valid C<fileobject> which contains certain mandatory object methods (exact methods yet to be hammered out). 2. undef if the file can't be handled, which allows stacked handlers. As such, when a user specifies something like: # Open a dir $dir = open dir "/usr/bin"; Then the "dir" handler is called by Perl's standard indirect object notation: $dir = dir->open("/usr/bin"); However, "dir" would not correspond directly to a class, but rather an event handler. To paraphrase Tom Hughes's great explanation of this, the core should provide a way for modules to register themselves as being handlers for certain files. For example, you might load C<LWP::UserAgent>, which would register as being a valid "http" handler. Or, you might load C<Net::FTP>, which would register itself as being an "ftp" handler. For example: use LWP::UserAgent; register open, 'http' => LWP::UserAgent::new; # first stab $web = open http "http://www.yahoo.com", 'GET'; The C<register> line gives an example of what might happen. This might be specified by the user (as shown), or might be down automatically by loading modules under a certain part of the lib tree. The best mechanism for this should probably be addressed in a separate RFC, since it has far more general applications than just for C<open>. These handlers should be stackable. If an C<open> handler returns undef, then the next one in line tries to open it. If no handler can open a specific file, then undef is returned from C<open> to the user, consistent with current behavior. For example, we might might stack "dir" handlers so that only certain users can look at the contents of "/usr/sbin", for example: use Unix::OnlyRootSeesUsrSbin; register open, 'dir' => Unix::OnlyRootSeesUsrSbin::open; $dir = open dir "/usr/sbin"; # not root? no /usr/sbin for you! Here are some more examples which could be use to provide essentially native file access for many different media: # Open a file # Note "file" is the default class so doesn't have to be specified $motd = open ">/etc/motd" or die; print $motd @data; close $motd; # Open a directory $dir = open dir "/usr/bin"; @files = grep !/^\..*/, <$dir>; # no more readdir close $dir; # no more closedir # Open a client socket with IO::Socket # By overloading < and > we can do clients and servers! $socket = open socket "< 25", PF_INET, SOCK_STREAM, TCP; @input = <$socket>; close $socket; do_something(@input); # Open a remote webpage $http = open http "http://www.perl.com/", GET; @doc = <$http>; print @doc if $http->content_type eq 'text/html'; close $http; # Open an ftp connection $ftp = open ftp "ftp.perl.com"; $ftp->cwd('CPAN') or die; @files = <$ftp>; # overloading as dir close $ftp; # Open a file like sysopen() # print $sysfile automatically calls $sysfile->print $sysfile = open sys "/etc/issue", O_RDWR|O_CREAT, 0644; print $sysfile "Hello world!", $buf, $len, $offset; The advantages to using such extensible <open handlers> are twofold: 1. No more reinventing the wheel to support new file types on different platforms or networks. 2. Easy integration so that <> can address files, dirs, pipes, websites, ipc communications, sockets, rpc... Assuming that these handlers all agree to return C<fileobjects> with a consistent set of properties, this could lead to great optimizations since the structure of these C<fileobjects> will be known ahead of time. =head2 Custom File Methods In addition, handlers can provide a custom set of C<read>, C<print>, and other functions that do different things from the core Perl set. So, a Unix/NT integration module might include a C<print> method that delimits strings differently based on filesystem. Or, a module might provide a custom C<close> method that does special cleanup before closing a file. This is all accomplished automatically by Perl's indirect object notation. All of a given C<open> module's functions would inherit from the core file methods. For example: package MyHttpOpen; sub open { # do stuff... return $new_http_fileobject; } sub print { # overrides CORE::print() # maybe it PUTs or POSTs data } Then, in your script, you would say: use MyHttpOpen; register open, 'http' => MyHttpOpen; $post = open http "http://upload.mydomain.com/upload.cgi", POST; print $post @filedata; # calls $post->print $reply = <$post>; # calls CORE::read In this example, a custom C<print> is provided. However, a corresponding C<read> is not provided, so C<CORE::read> is used (which should actually be C<SUPER::read> assuming inheritance is setup correctly) . =head1 IMPLEMENTATION =head2 Syntax Changes The C<open> function would have to be altered to accept this new syntax and return a full-fledged B<fileobject>. The C<close> function would remain basically unchanged, acting on the B<fileobject> (or C<$_> if none is specified). A method must be developed for registering classes as event handlers. This has many more general applications than just for C<open>, and so should be addressed in a separate RFC. For example, this process might be able to be unified across C<open>, signals, alarms, and so on. =head2 Function Deletions Because this syntax is flexible enough to handle B<any> type of file operation, the following functions should be removed from Perl 6: # Replaced by 'open dir' opendir readdir # dir->read instead closedir # close instead seekdir # dir->seek instead rewinddir telldir seekdir The first four functions have easy alternatives per this RFC. The last three do not currently. If they are not widely used (I suspect they're not, especially since they're not implemented everywhere), they should be moved to a compatibility module (or axed). # Replaced by 'open sys' sysopen sysread # sys->read instead syswrite # sys->write instead The sys* family of file functions can, in my opinion, be safely removed because the following alternative makes the low-level handling of files much easier: $sysin = open sys "/etc/motd", O_RDONLY; $sysout = open sys "/tmp/datafile", O_RDWR|O_CREAT, 0644; # read calls $sysin->read custom method $data = read $sysin, $buf, $blksize; # print calls $sysout->print, alias to $sysout->write print $sysout "Hello, world!", $buf, $len, $offset; My own experience is that if a file is C<sysopen>'ed, usually C<sysread> and C<syswrite> are used on it (and vice versa). As such, the above seems more natural and consistent. # Replaced by 'open socket' socket setsockopt # socket->options connect # overload <> or socket->connect shutdown # close instead While "open socket" is admittedly 5 characters more than just "socket", these are made up for by the fact that "close" is 3 characters shorter than "shutdown", and "options" is another 3 shorter than "setsockopt". So, you're actually 1 character ahead! :-) =head2 Performance In order to prevent performance hits, anything that is packaged as a default file type (such as files, pipes, directories, sockets, ipc, and so on) must be highly optimized for interaction with this new version of C<open()>. Basically, anything living under the C<IO::> tree should be ripped apart and put back together again, or replaced with a new version under C<Open::> or something similar. =head1 CHANGES 1. Modified C<open> syntax to use indirect object notation 2. Added the concept of an C<open handler> and proposals for how handlers could be stacked and how classes could register handlers 3. Expanded the list of functions to remove =head1 REFERENCES RFC 33: Eliminate bareword filehandles RFC 30: STDIN, STDOUT, and STDERR should be renamed RFC 49: Objects should have builtin stringifying STRING method Tom Christiansen's great analysis of file object methods Tim Jenness's suggestion to use optimized IO objects for all I/O Nick Ing-Simmon's suggestion to hide IO:: classes from the user Tom Hughes's formalization of a way to register open() handlers Chaim's suggestion of a way to register classes in Perl 6 Everyone else on perl6-language-io, you guys are great! Thanks. :-)
