RE: lazy file reading in H98
> As a somewhat separate issue, the logic of half-closed, while it makes > sense for files, is somewhat bothersome for sockets. If I read the > docs correctly, once I've grabed the contents lazily, I can no longer > write to the handle. However, for some things I'd like to do with > sockets, this is a problem. > > Case in point, consider a protocol where I recieve an XML request on a > socket, and from that generate a response. Sure I can pull the XML > request in peice by peice using hGetLine (or whatever), such an > approach seems tedious. > > I'd much rather do this: > >do contents <- hGetContents socket > hPutStr socket (processXML contents) > hClose socket > > I can fake this (I think) with unsafeInterleaveIO, but I'd love to > have such behavior built in. Yes - I came across this recently while working on GHC's IO library. There are important distinctions between read/write handles to a file and read/write handles to some kind of full-duplex data stream such as a socket or pipe. A read/write handle to a stream is exactly equivalent to two separate handles one of which is used for reading and the other writing. In fact you can do this with GHC's socket library by making two handles from the same socket (this is a workaround for your problem above). A read/write file on the other hand contains a shared file pointer, which means it's impossible to use separate buffers for reading and writing, unless you ensure that only one is in use at a time. GHC's IO library goes to some trouble to keep this invariant. It might make sense to extend this distinction to the API: with a duplex handle it should be possible to perform write operations on a semi-closed handle. Cheers, Simon
RE: lazy file reading in H98
Sigbjorn Finne <[EMAIL PROTECTED]> wrote, > The behaviour is well defined, Haskell98 enforces a > single-writer, multiple reader locking on files (Sec. > 11.3.1 of the library report) which also extends > to semi-closed handles/files. Good point - I missed that one. In other words, the report already enforces Solution (1) and both GHC and Hugs do not conform to the report. One may argue that the report is not entirely precise, as 11.3.1 talks only about handles, whereas we had problems with `readFile' and `writeFile', which do not make the use of handles explicit. So, a clarification may be in order here (as I searched the report for occurrences of `readFile' explicit mentioning of this would actually have helped in my case). Cheers, Manuel > > -Original Message- > > From: Simon Peyton-Jones > > Sent: Tuesday, September 05, 2000 01:10 > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED] > > Cc: Simon Peyton-Jones > > Subject: RE: lazy file reading in H98 > > > > > > Folks, > > > > Finalising the Haskell 98 report is (still) on my to-do list. > > I plan to > > do it after ICFP, but I need a clear week which is why I've been > > procrastinating. > > And comments do keep coming in occasionally. Manuel's is a > > case in point. > > > > > > I'd be interested to hear people's opinion about the lazy-file read > > question. > > I'm not prepared to add new functions to Haskell 98, but I think > > the clarification of (1) or (2) below would be useful. (2) is nice > > but it makes *all* file reading more expensive, perhaps significantly > > so (e.g. making a complete copy of the file). So I am > > personally inclined > > to go for (1) and require Haskell programmers to do the > > consequent file-name > > changing themselves. > > > > I'm sending this message to the haskell-cafe! > > > > Simon > > > > | -Original Message- > > | From: Manuel M. T. Chakravarty [mailto:[EMAIL PROTECTED]] > > | Sent: 05 September 2000 02:10 > > | To: [EMAIL PROTECTED] > > | Subject: lazy file reading in H98 > > | > > | > > | In an assignment, in my class, we came across a lack of > > | specification of the behaviour of `Prelude.readFile' and > > | `IO.hGetContents' and IMHO also a lack of functionality. As > > | both operations read a file lazily, subsequent writes to the > > | same file are potentially disastrous. In this assignment, > > | the file was used to make a Haskell data structure > > | persistent over multiple runs of the program - ie, > > | > > | readFile fname >>= return . read > > | > > | at the start of the program and > > | > > | writeFile fname . show > > | > > | at the end of the program. For certain inputs, where the > > | data structure stored in the file was only partially used, > > | the file was overwritten before it was fully read. > > | > > | H98 doesn't really specify what happens in this situation. > > | I think, there are two ways to solve that: > > | > > | (1) At least, the definition should say that the behaviour > > | is undefined if a program every writes to a file that it > > | has read with `readFile' or `hGetContents' before. > > | > > | (2) Alternatively, it could demand more sophistication from > > | the implementation and require that upon opening of a > > | file for writing that is currently semi-closed, the > > | implementation has to make sure that the contents of the > > | semi-closed file is not corrupted before it is fully > > | read.[1] > > | > > | In the case that solution (1) is chosen, I think, we should > > | also have something like `strictReadFile' (and > > | `hStrictGetContents') which reads the whole file before > > | proceeding to the next IO action. Otherwise, in situations > > | like in the mentioned assignment, you have to resort to > > | reading the file character by character, which seems very > > | awkward. > > | > > | So, overall, I think solution (2) is more elegant. > > | > > | Cheers, > > | Manuel > > | > > | [1] On Unix-like (POSIX?) systems, unlinking the file and > > | then opening the writable file would be sufficient. On > > | certain legacy OSes, the implementation would have to > > | read the rest of the file into memory before creating > > | a new file under the same name. > > | > >
Re: lazy file reading in H98
On Tue, Sep 05, 2000 at 01:10:07AM -0700, Simon Peyton-Jones wrote: > Finalising the Haskell 98 report is (still) on my to-do list. I > plan to do it after ICFP, but I need a clear week which is why I've > been procrastinating. And comments do keep coming in occasionally. > Manuel's is a case in point. > I'd be interested to hear people's opinion about the lazy-file read > question. I'm not prepared to add new functions to Haskell 98, but > I think the clarification of (1) or (2) below would be useful. (2) > is nice but it makes *all* file reading more expensive, perhaps > significantly so (e.g. making a complete copy of the file). So I am > personally inclined to go for (1) and require Haskell programmers to > do the consequent file-name changing themselves. As a somewhat separate issue, the logic of half-closed, while it makes sense for files, is somewhat bothersome for sockets. If I read the docs correctly, once I've grabed the contents lazily, I can no longer write to the handle. However, for some things I'd like to do with sockets, this is a problem. Case in point, consider a protocol where I recieve an XML request on a socket, and from that generate a response. Sure I can pull the XML request in peice by peice using hGetLine (or whatever), such an approach seems tedious. I'd much rather do this: do contents <- hGetContents socket hPutStr socket (processXML contents) hClose socket I can fake this (I think) with unsafeInterleaveIO, but I'd love to have such behavior built in. -- -- Jeffrey Straszheim | A sufficiently advanced -- Systems Engineer, Programmer| regular expression is -- http://www.shadow.net/~stimuli | indistinguishable from -- stimuli AT shadow DOT net | magic
RE: lazy file reading in H98
The behaviour is well defined, Haskell98 enforces a single-writer, multiple reader locking on files (Sec. 11.3.1 of the library report) which also extends to semi-closed handles/files. A bug report to the maintainers of whatever produced the reported misbehaviour would be good. --sigbjorn > -Original Message- > From: Simon Peyton-Jones > Sent: Tuesday, September 05, 2000 01:10 > To: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Cc: Simon Peyton-Jones > Subject: RE: lazy file reading in H98 > > > Folks, > > Finalising the Haskell 98 report is (still) on my to-do list. > I plan to > do it after ICFP, but I need a clear week which is why I've been > procrastinating. > And comments do keep coming in occasionally. Manuel's is a > case in point. > > > I'd be interested to hear people's opinion about the lazy-file read > question. > I'm not prepared to add new functions to Haskell 98, but I think > the clarification of (1) or (2) below would be useful. (2) is nice > but it makes *all* file reading more expensive, perhaps significantly > so (e.g. making a complete copy of the file). So I am > personally inclined > to go for (1) and require Haskell programmers to do the > consequent file-name > changing themselves. > > I'm sending this message to the haskell-cafe! > > Simon > > | -Original Message- > | From: Manuel M. T. Chakravarty [mailto:[EMAIL PROTECTED]] > | Sent: 05 September 2000 02:10 > | To: [EMAIL PROTECTED] > | Subject: lazy file reading in H98 > | > | > | In an assignment, in my class, we came across a lack of > | specification of the behaviour of `Prelude.readFile' and > | `IO.hGetContents' and IMHO also a lack of functionality. As > | both operations read a file lazily, subsequent writes to the > | same file are potentially disastrous. In this assignment, > | the file was used to make a Haskell data structure > | persistent over multiple runs of the program - ie, > | > | readFile fname >>= return . read > | > | at the start of the program and > | > | writeFile fname . show > | > | at the end of the program. For certain inputs, where the > | data structure stored in the file was only partially used, > | the file was overwritten before it was fully read. > | > | H98 doesn't really specify what happens in this situation. > | I think, there are two ways to solve that: > | > | (1) At least, the definition should say that the behaviour > | is undefined if a program every writes to a file that it > | has read with `readFile' or `hGetContents' before. > | > | (2) Alternatively, it could demand more sophistication from > | the implementation and require that upon opening of a > | file for writing that is currently semi-closed, the > | implementation has to make sure that the contents of the > | semi-closed file is not corrupted before it is fully > | read.[1] > | > | In the case that solution (1) is chosen, I think, we should > | also have something like `strictReadFile' (and > | `hStrictGetContents') which reads the whole file before > | proceeding to the next IO action. Otherwise, in situations > | like in the mentioned assignment, you have to resort to > | reading the file character by character, which seems very > | awkward. > | > | So, overall, I think solution (2) is more elegant. > | > | Cheers, > | Manuel > | > | [1] On Unix-like (POSIX?) systems, unlinking the file and > | then opening the writable file would be sufficient. On > | certain legacy OSes, the implementation would have to > | read the rest of the file into memory before creating > | a new file under the same name. > | >
RE: lazy file reading in H98
Simon Peyton-Jones <[EMAIL PROTECTED]> wrote, > I'd be interested to hear people's opinion about the lazy-file read > question. > I'm not prepared to add new functions to Haskell 98, but I think > the clarification of (1) or (2) below would be useful. (2) is nice > but it makes *all* file reading more expensive, perhaps significantly > so (e.g. making a complete copy of the file). I don't agree that it makes all file reading more expensive. My proposal is to do the unlink game on Unix (no extra memory costs) and actual read the file only on OSes that can't do the unlink trick and *only* when the file is written to (ie, only when we have a conflict). In other words, only in the situation which currently causes file corruption and would be completely outlawed in (1), extra resources are required. So, we have additional costs only when needed. As pointed out by Ketil, the costs on Unix are negligible. Maybe there are also ways to handle this gracefully on NT. Moreover, programs that do not want to run the risk of increased memory use on legacy OSes can still copy the file (something they would have to do currently anyway). In fact the bigger problem is to recognise the case where we write to a file, which is semi-closed (different file names can point to the same physical file, eg, when sym links are used). In Unix, this is again quite easy, because we can compare the inode number of the two files.[1] In other OSes, we can at least fall back to comparing file names. So, I think, it is quite clear how to implement the proposed functionality and I don't see significant costs for the general case. Manuel [1] In cases - like NFS - where this doesn't work, the underlying file system usually already makes only very weak statements about consistency. So, I think, we don't have worry about this. > | -Original Message- > | From: Manuel M. T. Chakravarty [mailto:[EMAIL PROTECTED]] > | Sent: 05 September 2000 02:10 > | To: [EMAIL PROTECTED] > | Subject: lazy file reading in H98 > | > | > | In an assignment, in my class, we came across a lack of > | specification of the behaviour of `Prelude.readFile' and > | `IO.hGetContents' and IMHO also a lack of functionality. As > | both operations read a file lazily, subsequent writes to the > | same file are potentially disastrous. In this assignment, > | the file was used to make a Haskell data structure > | persistent over multiple runs of the program - ie, > | > | readFile fname >>= return . read > | > | at the start of the program and > | > | writeFile fname . show > | > | at the end of the program. For certain inputs, where the > | data structure stored in the file was only partially used, > | the file was overwritten before it was fully read. > | > | H98 doesn't really specify what happens in this situation. > | I think, there are two ways to solve that: > | > | (1) At least, the definition should say that the behaviour > | is undefined if a program every writes to a file that it > | has read with `readFile' or `hGetContents' before. > | > | (2) Alternatively, it could demand more sophistication from > | the implementation and require that upon opening of a > | file for writing that is currently semi-closed, the > | implementation has to make sure that the contents of the > | semi-closed file is not corrupted before it is fully > | read.[1] > | > | In the case that solution (1) is chosen, I think, we should > | also have something like `strictReadFile' (and > | `hStrictGetContents') which reads the whole file before > | proceeding to the next IO action. Otherwise, in situations > | like in the mentioned assignment, you have to resort to > | reading the file character by character, which seems very > | awkward. > | > | So, overall, I think solution (2) is more elegant. > | > | Cheers, > | Manuel > | > | [1] On Unix-like (POSIX?) systems, unlinking the file and > | then opening the writable file would be sufficient. On > | certain legacy OSes, the implementation would have to > | read the rest of the file into memory before creating > | a new file under the same name. > |
Re: lazy file reading in H98
Simon Peyton-Jones <[EMAIL PROTECTED]> writes: > I'm not prepared to add new functions to Haskell 98, but I think > the clarification of (1) or (2) below would be useful. (2) is nice > but it makes *all* file reading more expensive, perhaps significantly > so (e.g. making a complete copy of the file). Well, as noted ( > | [1] On Unix-like (POSIX?) systems, unlinking the file and > | then opening the writable file would be sufficient. On > | certain legacy OSes, the implementation would have to > | read the rest of the file into memory before creating > | a new file under the same name. ) the unlink shouldn't cost much, except in terms of disk space, and I'm mostly using Unix systems anyway, so I'd vastly prefer that. Surely the file wouldn't need to be loaded into memory on any but the most atrocious OSes? A rename into some temporary file should be possible at low cost. There are of course problems with that approach: What if you only have write access to the file, and not its directory? And using a tmp directory might mean moving data across file systems, instead of just renaming. Could NT's (or others) multi-forked files be useful, I wonder? I.e. rename the default fork (stream, whatever) in the file, redirect subsequent reads, and write a new default fork? > So I am personally inclined to go for (1) and require Haskell > programmers to do the consequent file-name changing themselves. What can I say? Yuk. This would probably mean that I'd have to do a lot of explicit file copying, and probably more than really necessary, just to be on the safe side. In other words, while option (2) might have a cost on less developed systems, option (1) will carry at least that cost on all systems. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
RE: lazy file reading in H98
Folks, Finalising the Haskell 98 report is (still) on my to-do list. I plan to do it after ICFP, but I need a clear week which is why I've been procrastinating. And comments do keep coming in occasionally. Manuel's is a case in point. I'd be interested to hear people's opinion about the lazy-file read question. I'm not prepared to add new functions to Haskell 98, but I think the clarification of (1) or (2) below would be useful. (2) is nice but it makes *all* file reading more expensive, perhaps significantly so (e.g. making a complete copy of the file). So I am personally inclined to go for (1) and require Haskell programmers to do the consequent file-name changing themselves. I'm sending this message to the haskell-cafe! Simon | -Original Message- | From: Manuel M. T. Chakravarty [mailto:[EMAIL PROTECTED]] | Sent: 05 September 2000 02:10 | To: [EMAIL PROTECTED] | Subject: lazy file reading in H98 | | | In an assignment, in my class, we came across a lack of | specification of the behaviour of `Prelude.readFile' and | `IO.hGetContents' and IMHO also a lack of functionality. As | both operations read a file lazily, subsequent writes to the | same file are potentially disastrous. In this assignment, | the file was used to make a Haskell data structure | persistent over multiple runs of the program - ie, | | readFile fname >>= return . read | | at the start of the program and | | writeFile fname . show | | at the end of the program. For certain inputs, where the | data structure stored in the file was only partially used, | the file was overwritten before it was fully read. | | H98 doesn't really specify what happens in this situation. | I think, there are two ways to solve that: | | (1) At least, the definition should say that the behaviour | is undefined if a program every writes to a file that it | has read with `readFile' or `hGetContents' before. | | (2) Alternatively, it could demand more sophistication from | the implementation and require that upon opening of a | file for writing that is currently semi-closed, the | implementation has to make sure that the contents of the | semi-closed file is not corrupted before it is fully | read.[1] | | In the case that solution (1) is chosen, I think, we should | also have something like `strictReadFile' (and | `hStrictGetContents') which reads the whole file before | proceeding to the next IO action. Otherwise, in situations | like in the mentioned assignment, you have to resort to | reading the file character by character, which seems very | awkward. | | So, overall, I think solution (2) is more elegant. | | Cheers, | Manuel | | [1] On Unix-like (POSIX?) systems, unlinking the file and | then opening the writable file would be sufficient. On | certain legacy OSes, the implementation would have to | read the rest of the file into memory before creating | a new file under the same name. |
