Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by simonmar): You can't do an atomic non-blocking read, e.g. if another thread happened to read from the console and grabbed the character after you had called `fdReady` but before calling `read`, then the `read` would block. It is unlikely to happen, because someone would have to use raw `System.Win32` operations to get separate access to the console or whatever, but it's possible. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by simonmar): In principle this is a good change, however I don't agree with the changes in semantics of `hGetBufNonBlocking` and `hGetBufSome`. As it stands right now, `hGetBuf`, `hGetBufNonBlocking` and `hGetBufSome` will all read the same amount of data, if the data is available without blocking. But in your version (unless I've misread it), `hGetBufNonBlocking` will read at most a buffer of data, which seems to me to be an abstraction violation - the buffer size is supposed to be invisible to callers of the `hGetBuf` family. Why make this change? It should be possible to loop and read more data in the same way as `hGetBuf`. The only other minor quibble I have with this patch is that the documentation for `readBuffered` and friends could be improved - it isn't clear what the purpose of the buffer argument is (without reading the code). -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by joeyadams): Thanks for the review. As it stands right now, hGetBuf, hGetBufNonBlocking and hGetBufSome will all read the same amount of data, if the data is available without blocking. I don't think this is the case anymore; see #5843 and commit [https://github.com/ghc/packages- base/commit/370fc0b455f6a03283fbd5c0baa5d08d9115379d 370fc0b]. The issue is that some devices don't support non-blocking I/O (e.g. file handles on Windows). Here's the problematic sequence: * Call `hWaitForInput`. It returns `True` when the buffer is not empty. * Call `hGetBufSome`, which we expect will not block. It first reads bytes from the buffer, then tries to do a non-blocking read. Since the non-blocking IO methods block on Windows, `hGetBufSome` blocks, even after it has read some data. So here are some of our options: * Change semantics of `hGetBufSome` so that it will only block if there is no input. However, there is no guarantee that `hGetBufSome` will return all immediately available bytes. * On Windows, simulate nonblocking and IODevice.ready by continuously issuing reads in a forked thread to a buffer, and having I/O functions read from that buffer. * Deprecate `hGetBufNonBlocking`, `hPutBufNonBlocking`, `hWaitForInput`, and the non-blocking `RawIO` and `BufferedIO` methods. Come up with an alternative design that does not require the system to support non- blocking operation. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by simonmar): Ok, I see that `hGetBufSome` already has the behaviour that I claimed is erroneous. That's sad. I'd really like to declare this to be a bug in the Windows implementation of Handles, but perhaps that's impractical. I think it might be possible to implement non-blocking I/O on Windows, but you have to do it differently for every type of Handle (console, socket, com port etc.). We do have a working implementation of `hWaitForInput` (see `libraries/base/cbits/inputReady.c`) but it's horrible. Maybe deprecating the non-blocking APIs is the right way, since we are providing the ability to do asynchronous I/O using threads, and that's much nicer. But before we do that, we should look at how people are using these APIs. The only user of `hGetBufSome` that I know of, lazy bytestring, works just fine with the read a random amount of data semantics. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by joeyadams): For Windows, couldn't we implement non-blocking I/O correctly by first calling the `inputReady.c` function? That is, change `readRawBufferPtrNoBlock` and `writeRawBufferPtrNoBlock` to first call `fdReady` to make sure the operation won't block before proceeding. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: new Priority: normal| Milestone: Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Changes (by simonpj): * owner: joeyadams = simonmar * difficulty: = Unknown Comment: Simon M, you look like the most plausible reviewer, since you did the earlier patch. Simon -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: simonmar Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Unknown Testcase:| Blockedby: Blocking:|Related: ---+ Changes (by simonpj): * priority: normal = high * status: new = patch * milestone: = 7.8.1 -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: joeyadams Type: bug | Status: new Priority: normal| Milestone: Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Testcase:| Blockedby: Blocking:|Related: ---+ Comment(by joeyadams): Here are my changes, if anyone wants to review them: https://github.com/joeyadams/packages-base/tree/custom-device-fix `sh validate` passes on Ubuntu 10.04 64-bit (except for some unrelated tests which fail because my installed LLVM is too old). I'm still working on a test case. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
---+
Reporter: AntoineLatter | Owner: joeyadams
Type: bug | Status: new
Priority: normal| Milestone:
Component: libraries/base|Version: 7.6.1
Resolution:| Keywords:
Os: Unknown/Multiple | Architecture: Unknown/Multiple
Failure: Runtime crash | Difficulty:
Testcase:| Blockedby:
Blocking:|Related:
---+
Changes (by joeyadams):
* owner: = joeyadams
Comment:
I plan to fix this by adding the following methods to BufferedIO:
{{{
readBuffered :: dev - Buffer Word8 - Ptr Word8 - Int - IO (Int,
Buffer Word8)
readBuffered0 :: dev - Buffer Word8 - Ptr Word8 - Int - IO (Maybe
Int, Buffer Word8)
writeBuffered :: dev
- Bool -- ^ If 'True', flush after writing.
- Buffer Word8
- Ptr Word8
- Int
- IO (Buffer Word8)
writeBuffered0 :: dev
- Bool - Buffer Word8 - Ptr Word8 - Int
- IO (Int, Buffer Word8)
}}}
The default implementations will use the fill and flush methods of
BufferedIO, but an instance may provide more efficient substitutes.
RawIO-based methods will be provided, which will go directly to the device
when possible and efficient.
This will move a lot of the complexity of the hGetBuf, hPutBuf, etc.
functions out of GHC.IO.Handle.Text and into GHC.IO.BufferedIO (in the
form of default methods and RawIO drop-ins).
I will be changing the behavior of hGetBufSome slightly: if the buffer is
not empty, it will fetch its contents and return immediately, rather than
trying to squeeze more bytes out of the device with non-blocking fills and
reads. See #5843
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:5
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler
___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: Type: bug | Status: new Priority: normal| Milestone: Component: libraries/base|Version: 7.6.1 Resolution:| Keywords: Os: Unknown/Multiple | Architecture: Unknown/Multiple Failure: Runtime crash | Difficulty: Testcase:| Blockedby: Blocking:|Related: ---+ Changes (by joeyadams): * status: closed = new * cc: joeyadams3.14159@… (added) * resolution: fixed = * component: Compiler = libraries/base * version: 6.12.3 = 7.6.1 * architecture: x86_64 (amd64) = Unknown/Multiple * os: MacOS X = Unknown/Multiple Comment: The fix for this is incomplete. I implemented a custom handle to make network I/O interruptible on Windows, but Holger Reinhardt revealed [https://github.com/haskell/network/pull/79#issuecomment-11637676 a case where `Todo: hPutBuf` comes up]. It looks like `bufWrite` (in GHC.IO.Handle.Text), when told to write a chunk that is larger than the buffer size, does a RawIO `writeChunk`, which only works for `FD`. `hGetBufSome` has the same problem. Another problem is that the RawIO methods are sometimes called with a size argument of zero. This can lead to failed system calls and unexpected side effects unless the implementation filters out zero-length calls. Also, if `RawIO.read buf 0` returns 0, the callee might treat it as EOF. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
+---
Reporter: AntoineLatter | Owner:
Type: bug| Status: closed
Priority: normal | Milestone:
Component: Compiler |Version: 6.12.3
Resolution: fixed | Keywords:
Difficulty: | Os: MacOS X
Testcase: | Architecture: x86_64 (amd64)
Failure: Runtime crash |
+---
Changes (by simonmar):
* status: new = closed
* resolution: = fixed
Comment:
This should fix it:
{{{
Thu Jun 24 14:04:25 BST 2010 Simon Marlow marlo...@gmail.com
* make the hGetBuf/hPutBuf family work with non-FD Handles (#4144)
}}}
if it doesn't find an FD in the Handle, it uses the normal buffered
operations. Perhaps we could improve matters by allowing Handles to
optionally provide `RawIO`, but that's something for another day.
{{{
Prelude ByteStringHandle bsHandle test fake file =
Data.ByteString.Char8.hGetContents
Loading package bytestring-0.9.1.7 ... linking ... done.
test
}}}
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:3
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler
___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
---+
Reporter: AntoineLatter | Owner:
Type: bug | Status: new
Priority: normal | Component: Compiler
Version: 6.12.3 |Keywords:
Os: MacOS X |Testcase:
Architecture: x86_64 (amd64) | Failure: Runtime crash
---+
Comment(by AntoineLatter):
Code to demonstrate crash, given the above module and -XOverloadedStrings:
{{{bsHandle test fake file = Data.ByteString.Char8.hGetContents}}}
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:1
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler
___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure
#4144: Exception: ToDo: hGetBuf - when using custom handle infrastructure ---+ Reporter: AntoineLatter | Owner: Type: bug | Status: new Priority: normal | Component: Compiler Version: 6.12.3 |Keywords: Os: MacOS X |Testcase: Architecture: x86_64 (amd64) | Failure: Runtime crash ---+ Changes (by AntoineLatter): * cc: AntoineLatter (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/4144#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
