Re: base package (Was: GHC 7.8 release?)
Comparing hash, ptr, str gives you a pretty good acceptance/rejection test. hash for the quick rejection, ptr for quick acceptance, str for accuracy. Especially since the particular fingerprints for Typeable at least are usually made up of 3 bytestrings that were just stuffed in and forgotten about. That said, this would seem to bring ByteString or at least Ptr in at a pretty low level for the level of generality folks seem to be suddenly seeking. -Edward On Wed, Feb 20, 2013 at 12:12 PM, Ian Lynagh i...@well-typed.com wrote: On Fri, Feb 15, 2013 at 02:45:19PM +, Simon Marlow wrote: Remember that fingerprinting is not hashing. For fingerprinting we need to have a realistic expectation of no collisions. I don't think FNV is suitable. I'm sure it would be possible to replace the C md5 code with some Haskell. Performance *is* important here though - Typeable is in the inner loop of certain generic programming libraries, like SYB. We currently just compare hash(str) for equality, right? Could we instead compare (hash str, str) ? That would be even more correct, even if a bad/cheap hash function is used, and would only be slower for the case where the types match (unless you're unlucky and get a hash collision). In fact, we may be able to arrange it so that in the equal case the strings are normally exactly the same string, so we can do a cheap pointer equality test (like ByteString already does) to make the equal case fast too (falling back to actually checking the strings are equal, if they aren't the same string). Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On 20/02/13 17:12, Ian Lynagh wrote: On Fri, Feb 15, 2013 at 02:45:19PM +, Simon Marlow wrote: Remember that fingerprinting is not hashing. For fingerprinting we need to have a realistic expectation of no collisions. I don't think FNV is suitable. I'm sure it would be possible to replace the C md5 code with some Haskell. Performance *is* important here though - Typeable is in the inner loop of certain generic programming libraries, like SYB. We currently just compare hash(str) for equality, right? Could we instead compare (hash str, str) ? That would be even more correct, even if a bad/cheap hash function is used, and would only be slower for the case where the types match (unless you're unlucky and get a hash collision). In fact, we may be able to arrange it so that in the equal case the strings are normally exactly the same string, so we can do a cheap pointer equality test (like ByteString already does) to make the equal case fast too (falling back to actually checking the strings are equal, if they aren't the same string). So it's not a single string, a TypeRep consists of a TyCon applied to some arguments, which themselves are TypeReps etc. You could do pointer equality, and maybe that would work for the common cases. But I don't see why we have to do that when fingerprinting works well and we already have it. Why add a potential performance pitfall when we don't have to? One other thing: it's useful to be able to use the fingerprint as an identifier for the contents, e.g. when sending Typeable values across the network. If you can't do this with the fingerprint, then you need another unique Id, which is the situation we used to have before fingerprints. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Fri, Feb 15, 2013 at 02:45:19PM +, Simon Marlow wrote: Remember that fingerprinting is not hashing. For fingerprinting we need to have a realistic expectation of no collisions. I don't think FNV is suitable. I'm sure it would be possible to replace the C md5 code with some Haskell. Performance *is* important here though - Typeable is in the inner loop of certain generic programming libraries, like SYB. We currently just compare hash(str) for equality, right? Could we instead compare (hash str, str) ? That would be even more correct, even if a bad/cheap hash function is used, and would only be slower for the case where the types match (unless you're unlucky and get a hash collision). In fact, we may be able to arrange it so that in the equal case the strings are normally exactly the same string, so we can do a cheap pointer equality test (like ByteString already does) to make the equal case fast too (falling back to actually checking the strings are equal, if they aren't the same string). Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, Am Donnerstag, den 14.02.2013, 21:41 -0500 schrieb brandon s allbery kf8nh: On Thursday, February 14, 2013 at 8:14 PM, Johan Tibell wrote: On Thu, Feb 14, 2013 at 2:53 PM, Joachim Breitner m...@joachim-breitner.de wrote: I don't think having FFI far down the stack is a problem. There are lots of pure data types we'd like in the pure data layer (e.g. bytestring) that uses FFI. As long as the I/O layer itself (System.IO, the I/O manager, etc) doesn't get pulled in there's no real problem in depending on the FFI. I think it would be nice, also to other Haskell implementations that might have not have FFI, to separate the really basic stuff from pure-but-impurely-implemented stuff. At least as long as it does not caues trouble. GHC.Fingerprint does not need to be crippled when it is going to use a pure hashing; I quickly added some simple fingerpriting found via Wikipedia that was easier than MD5. https://github.com/nomeata/packages-base/commit/b7f80066a03fd296950e0cafa2278d43a86f82fc The choice is of course not final, maybe something with more bits is desirable. Doesn't the FFI pull in some part of the I/O layer, though? In particular threaded programs are going to end up using forkOS? Another good reason to try to have a pure ground library. Based on base-pure, the next step would be to check if FFI can be provided without IO (but I doubt that is difficult), so there would be an base-io package on top of base-pure, and then bytestring can depend on that base-io and base-pure, while users of bytestring of course don’t have to depend on base-io (as long as they are not using the IO-related functions of bytestring). Questions: * Does anyone have a tool to compare package APIs? It would be interesting to diff base’s API with the combined APIs of the package we are creating right now. * Currently, base-pure exports lots of modules that should not be part of its “pure” API (all the GHC.* packages). But I assume they need to be exported for the benefit of packages like base-io built on top. Should we provide another package that re-exports those that are for public consumption and is likely to have a more stable API? Again I feel the need for packages re-exporting modules without incurring a conflict. * What to do with Prelude. Should it be in base-io (which is potentially the lowest package containing everything needed) or rather in a package of its own? Or should it live in a package of its own? Or can we use the haskell2010 package for that somehow? * Should base-io provide just the IO monad and all, say, file-related stuff in a separate package or is this going too far? Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: base package (Was: GHC 7.8 release?)
| Doesn't the FFI pull in some part of the I/O layer, though? In | particular threaded programs are going to end up using forkOS? | | Another good reason to try to have a pure ground library. Remember that we have UNSAFE ffi calls and SAFE ones. The SAFE ones may block, cause GC etc. They involve a lot of jiggery pokery and I would not be surprised if that affected the I/O manager. But UNSAFE ones are, by design, no more than fat machine instructions that are implemented by taking an out-of-line call. They should not block. They should not cause GC. Nothing. Think of 'sin' and 'cos' for example. Fingerprinting is a classic example, I would have thought. So my guess is that it should not be hard to allow UNSAFE ffi calls in the core (non-IO-ish) bits, leaving SAFE calls for higher up the stack. Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, more progress: On top of base-pure, I created base-io involving GHC/IO and everything required to build it (which pulled in ST, some of Foreign and unfortunately some stuff related to Handles and Devices, because it is mentioned in IOException). This is the list of modules: Foreign.C.Types, Foreign.ForeignPtr, Foreign.ForeignPtr.Imp, Foreign.ForeignPtr.Safe, Foreign.ForeignPtr.Unsafe, Foreign.Ptr, Foreign.Storable, GHC.ForeignPtr, GHC.IO.BufferedIO, GHC.IO.Buffer, GHC.IO.Device, GHC.IO.Encoding.Types, GHC.IO.Exception, GHC.IO.Handle.Types, GHC.IO, GHC.IORef, GHC.MVar, GHC.Ptr, GHC.Stable, GHC.ST, GHC.Storable, GHC.STRef It is on a different branch on my github repo: https://github.com/nomeata/packages-base/tree/base-io GHC would complain that the CInt type is not valid in a ffi call (probably due to the different package name), so I replaced foreign declarations by regular ones defined using “undefined” – after all I’m just trying to discover how things can be split at all and just work towards building stuff. ST can probably be pulled below this package, after all it is quite pure. Either a package of its own, or in base-pure. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On 15/02/13 09:36, Simon Peyton-Jones wrote: | Doesn't the FFI pull in some part of the I/O layer, though? In | particular threaded programs are going to end up using forkOS? | | Another good reason to try to have a pure ground library. Remember that we have UNSAFE ffi calls and SAFE ones. The SAFE ones may block, cause GC etc. They involve a lot of jiggery pokery and I would not be surprised if that affected the I/O manager. But UNSAFE ones are, by design, no more than fat machine instructions that are implemented by taking an out-of-line call. They should not block. They should not cause GC. Nothing. Think of 'sin' and 'cos' for example. Fingerprinting is a classic example, I would have thought. So my guess is that it should not be hard to allow UNSAFE ffi calls in the core (non-IO-ish) bits, leaving SAFE calls for higher up the stack. Actually as far as the Haskell-level API goes, there's no difference between safe and unsafe FFI calls, the difference is all in the codegen. I don't think safe calls cause any more difficulties for splitting up the base. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On 15/02/13 08:36, Joachim Breitner wrote: Hi, Am Donnerstag, den 14.02.2013, 21:41 -0500 schrieb brandon s allbery kf8nh: On Thursday, February 14, 2013 at 8:14 PM, Johan Tibell wrote: On Thu, Feb 14, 2013 at 2:53 PM, Joachim Breitner m...@joachim-breitner.de wrote: I don't think having FFI far down the stack is a problem. There are lots of pure data types we'd like in the pure data layer (e.g. bytestring) that uses FFI. As long as the I/O layer itself (System.IO, the I/O manager, etc) doesn't get pulled in there's no real problem in depending on the FFI. I think it would be nice, also to other Haskell implementations that might have not have FFI, to separate the really basic stuff from pure-but-impurely-implemented stuff. At least as long as it does not caues trouble. GHC.Fingerprint does not need to be crippled when it is going to use a pure hashing; I quickly added some simple fingerpriting found via Wikipedia that was easier than MD5. https://github.com/nomeata/packages-base/commit/b7f80066a03fd296950e0cafa2278d43a86f82fc The choice is of course not final, maybe something with more bits is desirable. Remember that fingerprinting is not hashing. For fingerprinting we need to have a realistic expectation of no collisions. I don't think FNV is suitable. I'm sure it would be possible to replace the C md5 code with some Haskell. Performance *is* important here though - Typeable is in the inner loop of certain generic programming libraries, like SYB. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On 15/02/13 12:22, Joachim Breitner wrote: Hi, more progress: On top of base-pure, I created base-io involving GHC/IO and everything required to build it (which pulled in ST, some of Foreign and unfortunately some stuff related to Handles and Devices, because it is mentioned in IOException). This is the list of modules: Foreign.C.Types, Foreign.ForeignPtr, Foreign.ForeignPtr.Imp, Foreign.ForeignPtr.Safe, Foreign.ForeignPtr.Unsafe, Foreign.Ptr, Foreign.Storable, GHC.ForeignPtr, GHC.IO.BufferedIO, GHC.IO.Buffer, GHC.IO.Device, GHC.IO.Encoding.Types, GHC.IO.Exception, GHC.IO.Handle.Types, GHC.IO, GHC.IORef, GHC.MVar, GHC.Ptr, GHC.Stable, GHC.ST, GHC.Storable, GHC.STRef You have a random collection of modules here :) I think you want to have the IO *monad* (GHC.IO) live in a lower layer, separate from the IO *library* (GHC.IO.Device and so on). Every Haskell implementation will need the IO monad, but they might want to replace the IO library with something else. Things like GHC.IORef, GHC.MVar can all live in a low-down layer because they're just wrappers over the primops. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thu, Feb 14, 2013 at 03:48:51PM +0100, Joachim Breitner wrote: Yesterday, I experimented a bit with base’s code, [...] Maybe the proper is to reverse the whole approach: Leave base as it is, and then build re-exporting smaller packages (e.g. a base-pure) on top of it. The advantage is: * No need to rewrite the tightly intertwined base. * Libraries still have the option to have tighter dependencies. * Base can evolve with lots of breaking changes, as long as they do not affect the API by the smaller packages. * Development of this collection can happen outside the GHC tree. * Alternative implementations for (some of) these packages can be created, if the reason why they could not be moved out of base is one of implementation, not of API How does that sound? Essentially good to me... One might consider instead (as has been proposed before, I believe), to rename the current ``base'' to something like ``ghc-base'' which is not intended to be depended on by packages not shipped with GHC (that is, by default ``hidden'' in ghc-pkg), and instead export: base with a very stable interface io with a very stable interface GHCwith a probably rapidly evolving interface. * possibly other packages giving access to internals Most packages that currently depend on ``base'' would then depend only on ``base'' and possibly ``io'', and by virtue of the stability of these two interfaces would therefore not be affected by most GHC releases. This would effectively be ``splitting the interfaces GHC and io out from base'' instead of ``deprecating base and replacing it with the three new interfaces base-pure, io, and GHC''. That choice is possibly mostly a matter of taste --- I think that the name ``base'' is good for a user-facing interface, and the name ``ghc-base'' more indicative of its implementation-dependent character. Wolfram ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thu, Feb 14, 2013 at 6:48 AM, Joachim Breitner m...@joachim-breitner.dewrote: Maybe the proper is to reverse the whole approach: Leave base as it is, and then build re-exporting smaller packages (e.g. a base-pure) on top of it. The advantage is: * No need to rewrite the tightly intertwined base. * Libraries still have the option to have tighter dependencies. * Base can evolve with lots of breaking changes, as long as they do not affect the API by the smaller packages. * Development of this collection can happen outside the GHC tree. * Alternative implementations for (some of) these packages can be created, if the reason why they could not be moved out of base is one of implementation, not of API How does that sound? I'm not in favor of this approach as it precludes pushing any data types down the stack. In particular, we want text and bytestring to be below the I/O layer, so we can defined Handles that work with those in base itself. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
As a super obvious aside, we could just switch code paths based on platform/environment. That lets you keep the fast code path and have a pure fallback for the javascripters. Just propagate an FFI_AVAILABLE flag into the compilation unit. We're going to have a number of these points which force a tension between generality And speed as we go, and it'd nice to have the ability to gracefully fall back. On Feb 15, 2013, at 9:45 AM, Simon Marlow marlo...@gmail.com wrote: On 15/02/13 08:36, Joachim Breitner wrote: Hi, Am Donnerstag, den 14.02.2013, 21:41 -0500 schrieb brandon s allbery kf8nh: On Thursday, February 14, 2013 at 8:14 PM, Johan Tibell wrote: On Thu, Feb 14, 2013 at 2:53 PM, Joachim Breitner m...@joachim-breitner.de wrote: I don't think having FFI far down the stack is a problem. There are lots of pure data types we'd like in the pure data layer (e.g. bytestring) that uses FFI. As long as the I/O layer itself (System.IO, the I/O manager, etc) doesn't get pulled in there's no real problem in depending on the FFI. I think it would be nice, also to other Haskell implementations that might have not have FFI, to separate the really basic stuff from pure-but-impurely-implemented stuff. At least as long as it does not caues trouble. GHC.Fingerprint does not need to be crippled when it is going to use a pure hashing; I quickly added some simple fingerpriting found via Wikipedia that was easier than MD5. https://github.com/nomeata/packages-base/commit/b7f80066a03fd296950e0cafa2278d43a86f82fc The choice is of course not final, maybe something with more bits is desirable. Remember that fingerprinting is not hashing. For fingerprinting we need to have a realistic expectation of no collisions. I don't think FNV is suitable. I'm sure it would be possible to replace the C md5 code with some Haskell. Performance *is* important here though - Typeable is in the inner loop of certain generic programming libraries, like SYB. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, Am Freitag, den 15.02.2013, 14:50 + schrieb Simon Marlow: On 15/02/13 12:22, Joachim Breitner wrote: Hi, more progress: On top of base-pure, I created base-io involving GHC/IO and everything required to build it (which pulled in ST, some of Foreign and unfortunately some stuff related to Handles and Devices, because it is mentioned in IOException). This is the list of modules: You have a random collection of modules here :) I think you want to have the IO *monad* (GHC.IO) live in a lower layer, separate from the IO *library* (GHC.IO.Device and so on). Every Haskell implementation will need the IO monad, but they might want to replace the IO library with something else. Things like GHC.IORef, GHC.MVar can all live in a low-down layer because they're just wrappers over the primops. Right, that is my aim, and I started with GHC.IO. But unfortunately, the IO monad calls failIO, which is an IOError which has a field of type ioe_handle :: Maybe Handle (and one of type CInt) which pulls in all the rest there, and so far I did not have a good idea how to untangle that. What would break if fail would not raise an IOError, but a separate exception type, e.g. IOFailError? Probably too much, as users expect to catch the exception raised by fail with an exception handler that matches IOError. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, Am Donnerstag, den 14.02.2013, 02:21 + schrieb Ian Lynagh: On Wed, Feb 13, 2013 at 07:32:06PM +0100, Joachim Breitner wrote: I have started a wikipage with the list of all modules from base, for a first round of shuffling, grouping and brainstorming: http://hackage.haskell.org/trac/ghc/wiki/SplitBase Great, thanks for taking the lead on this! lets see how far that leads goes... Yesterday, I experimented a bit with base’s code, first beginning with as few modules as possible and adding what’s required; then starting with the whole thing and trying to remove e.g. IO. But clearly it is not easy: 1. Int requires throw DivideByZero; Monad requires error. That pulls in Exceptions. 2. Exceptions require Typeable. 3. Typeable is implemented with GHC.Fingerprint. 4. GHC.Fingerprint is implemented with Foreign and IO. 5. Foreign clearly needs Int and the like. so it is not clear how to pull out a pure base without IO and Foreign. Are there any good tricks how to break these interdependencies? Maybe it is possible to have a pure base without the Monad class and without some of the operations on Int that throw exceptions, but that wold hardly be useable. There are other issues, some avoidable (such as the hard-coded base:Num constraint on literals); I collected a list on http://hackage.haskell.org/trac/ghc/wiki/SplitBase Maybe the proper is to reverse the whole approach: Leave base as it is, and then build re-exporting smaller packages (e.g. a base-pure) on top of it. The advantage is: * No need to rewrite the tightly intertwined base. * Libraries still have the option to have tighter dependencies. * Base can evolve with lots of breaking changes, as long as they do not affect the API by the smaller packages. * Development of this collection can happen outside the GHC tree. * Alternative implementations for (some of) these packages can be created, if the reason why they could not be moved out of base is one of implementation, not of API How does that sound? Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thu, Feb 14, 2013 at 03:48:51PM +0100, Joachim Breitner wrote: Yesterday, I experimented a bit with base’s code, first beginning with as few modules as possible and adding what’s required; then starting with the whole thing and trying to remove e.g. IO. But clearly it is not easy: 1. Int requires throw DivideByZero; Monad requires error. That pulls in Exceptions. 2. Exceptions require Typeable. 3. Typeable is implemented with GHC.Fingerprint. 4. GHC.Fingerprint is implemented with Foreign and IO. 5. Foreign clearly needs Int and the like. so it is not clear how to pull out a pure base without IO and Foreign. Are there any good tricks how to break these interdependencies? We'll probably end up with a package (let's say ghc-bottom for now) right at the bottom of the hierarchy which contains a minimal set of definitions for Typeable, throw, etc, in modules like GHC.Typeable, GHC.IO (or, if it's small and the dependencies are circular, might be easier to have it all in a single module). It might be possible to merge this into ghc-prim later, but don't worry about that for now. These definitions would then be re-exported by Data.Typeable, Control.Exception etc higher up. Other libraries would be discouraged from depending on ghc-bottom, either socially or by needing to jump through extra hoops in a .cabal file fo allow it. However, this is the hard end to start from, because the modules right at the bottom depend on things much higher up, with cyclic imports. It'll be a lot easier to pull modules off the top instead, as nothing imports them. They also tend to be less magical (e.g. don't have wired-in names). Also, don't worry about necessarily pulling out /all/ the modules you want for a package all at once. Once you have smaller chunks it'll be easier to see what changes are necessary to move modules up/down. You may also need to leave e.g. some of IO lower down than the io package in a GHC.* module, and then to re-export it from io. Once the top has been pulled at as much as possible, it'll be a lot easier to see what's going on with the remainder, and work out what needs to be done to break the biggest loops. There are other issues, some avoidable (such as the hard-coded base:Num constraint on literals); I collected a list on http://hackage.haskell.org/trac/ghc/wiki/SplitBase Right, for things with built-in syntax you'll have to update GHC's wired-in names as you go. This will mostly just mean changing the modules, e.g. gHC_ENUM = mkBaseModule (fsLit GHC.Enum) in compiler/prelude/PrelNames.lhs might become gHC_ENUM = mkGhcPureModule (fsLit GHC.Enum) with mkGhcPureModule defined analogously to mkBaseModule, but occasionally you might want to move a definition to another module. If you get stuck, just yell. Maybe the proper is to reverse the whole approach: Leave base as it is, and then build re-exporting smaller packages (e.g. a base-pure) on top of it. The advantage is: * No need to rewrite the tightly intertwined base. * Libraries still have the option to have tighter dependencies. * Base can evolve with lots of breaking changes, as long as they do not affect the API by the smaller packages. * Development of this collection can happen outside the GHC tree. * Alternative implementations for (some of) these packages can be created, if the reason why they could not be moved out of base is one of implementation, not of API Disadvantages: * No-one would use the new packages unless they come with GHC; e.g. not a perfect analogy, but compare the number of rev-deps according to http://packdeps.haskellers.com/reverse of the various *prelude* packages vs base: 4831 base 6 basic-prelude 8 classy-prelude 4 classy-prelude-conduit 2 custom-prelude 1 general-prelude 1 modular-prelude 17 numeric-prelude 2 prelude-extras * If it comes with GHC, it would mean us permanently maintaining the two levels * base would still be an opaque blob, with too many modules and cyclic imports, which makes development tricky Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, I made a little progress after crippling GHC.Fingerprint: The package at https://github.com/nomeata/packages-base/tree/base-pure (Branch base-pure) builds and contains just these modules: ./Control/Applicative.hs ./Control/Arrow.hs ./Control/Category.hs ./Control/Monad/Fix.hs ./Control/Monad.hs ./Data/Bits.hs ./Data/Bool.hs ./Data/Either.hs ./Data/Eq.hs ./Data/Foldable.hs ./Data/Function.hs ./Data/Functor.hs ./Data/Int.hs ./Data/List.hs ./Data/Maybe.hs ./Data/Monoid.hs ./Data/Ord.hs ./Data/Ratio.hs ./Data/Traversable.hs ./Data/Tuple.hs ./Data/Typeable.hs ./Data/Typeable.hs-boot ./Data/Typeable/Internal.hs ./Data/Typeable/Internal.hs-boot ./dist/build/autogen/Paths_base_pure.hs ./GHC/Base.lhs ./GHC/Char.hs ./GHC/Enum.lhs ./GHC/Err.lhs ./GHC/Err.lhs-boot ./GHC/Exception.lhs ./GHC/Fingerprint.hs ./GHC/Fingerprint/Type.hs ./GHC/Int.hs ./GHC/List.lhs ./GHC/Num.lhs ./GHC/Real.lhs ./GHC/Show.lhs ./GHC/Word.hs ./Prelude.hs (contains just $!) ./Unsafe/Coerce.hs The crippling can be seen here: https://github.com/nomeata/packages-base/blob/base-pure/GHC/Fingerprint.hs So if there were a magic way of getting a working GHC.Fingerprint module without pulling in Foreign, this would be a good start for a base free of any trace of * Foreign * IO * Floating point arithmetic * Read * ST * Array Alternative, a magic way of providing the functions error and divZeroError without having to define Exceptions would do well. I guess it is not possible to define the data ErrorCall without the Exception class and somehow call the primop raise# in a way that the error can be caught, as catch will expect something of type SomeException that has the Exception dictionary bundled. Any idea how to achieve that, without having resort to ghc-bottom as suggested by Ian (which would be ok, but not as nice as genuine small packages to start with)? Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thu, Feb 14, 2013 at 8:45 AM, Joachim Breitner m...@joachim-breitner.dewrote: ./Control/Applicative.hs ./Control/Arrow.hs ./Control/Category.hs ./Control/Monad/Fix.hs ./Control/Monad.hs ./Data/Bits.hs ./Data/Bool.hs ./Data/Either.hs ./Data/Eq.hs ./Data/Foldable.hs ./Data/Function.hs ./Data/Functor.hs ./Data/Int.hs ./Data/List.hs ./Data/Maybe.hs ./Data/Monoid.hs ./Data/Ord.hs ./Data/Ratio.hs ./Data/Traversable.hs ./Data/Tuple.hs ./Data/Typeable.hs ./Data/Typeable.hs-boot ./Data/Typeable/Internal.hs ./Data/Typeable/Internal.hs-boot ./dist/build/autogen/Paths_base_pure.hs ./GHC/Base.lhs ./GHC/Char.hs ./GHC/Enum.lhs ./GHC/Err.lhs ./GHC/Err.lhs-boot ./GHC/Exception.lhs ./GHC/Fingerprint.hs ./GHC/Fingerprint/Type.hs ./GHC/Int.hs ./GHC/List.lhs ./GHC/Num.lhs ./GHC/Real.lhs ./GHC/Show.lhs ./GHC/Word.hs ./Prelude.hs (contains just $!) ./Unsafe/Coerce.hs That's great. I'm curious I was under the impression that it was hard to split out a pure subset as functions might call 'error' (e.g. due to incomplete pattern matches) and that would pull in the whole I/O subsystem. How did you avoid that? -- Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, Am Donnerstag, den 14.02.2013, 13:19 -0800 schrieb Johan Tibell: That's great. I'm curious I was under the impression that it was hard to split out a pure subset as functions might call 'error' (e.g. due to incomplete pattern matches) and that would pull in the whole I/O subsystem. How did you avoid that? as mentioned before: By crippling GHC.Fingerprint. error foo just calls raise (ErrorCall foo), which calls raise# (SomeException (ErrorCall foo). The problem is that the definition of SomeException pulls in the Exception type class, which pulls in Typeable, which pulls in GHC.Fingerprint, which uses FFI and IO code to to fingerprinting... Looking at the code it seems that the FFI is only required for MD5. Maybe a pure implementation there is worth it, if it is not performance-critical code. Or, another work-around, would be primops for these commands that can be used without the FFI types and IO. I have also removed GHC.Unicode and functions like lines that require the FFI calls there. These seem to be the largest open issues. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thu, Feb 14, 2013 at 2:53 PM, Joachim Breitner m...@joachim-breitner.dewrote: Hi, Am Donnerstag, den 14.02.2013, 13:19 -0800 schrieb Johan Tibell: That's great. I'm curious I was under the impression that it was hard to split out a pure subset as functions might call 'error' (e.g. due to incomplete pattern matches) and that would pull in the whole I/O subsystem. How did you avoid that? as mentioned before: By crippling GHC.Fingerprint. error foo just calls raise (ErrorCall foo), which calls raise# (SomeException (ErrorCall foo). The problem is that the definition of SomeException pulls in the Exception type class, which pulls in Typeable, which pulls in GHC.Fingerprint, which uses FFI and IO code to to fingerprinting... I don't think having FFI far down the stack is a problem. There are lots of pure data types we'd like in the pure data layer (e.g. bytestring) that uses FFI. As long as the I/O layer itself (System.IO, the I/O manager, etc) doesn't get pulled in there's no real problem in depending on the FFI. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Thursday, February 14, 2013 at 8:14 PM, Johan Tibell wrote: On Thu, Feb 14, 2013 at 2:53 PM, Joachim Breitner m...@joachim-breitner.de (mailto:m...@joachim-breitner.de) wrote: I don't think having FFI far down the stack is a problem. There are lots of pure data types we'd like in the pure data layer (e.g. bytestring) that uses FFI. As long as the I/O layer itself (System.IO, the I/O manager, etc) doesn't get pulled in there's no real problem in depending on the FFI. Doesn't the FFI pull in some part of the I/O layer, though? In particular threaded programs are going to end up using forkOS? -- brandon s allbery kf8nh Sent with Sparrow (http://www.sparrowmailapp.com/?sig) ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
base package (was: GHC 7.8 release?)
* Simon Marlow marlo...@gmail.com [2013-02-13 09:00:15+] It's feasible to split base, but to a first approximation what you end up with is base renamed to ghc-base, and then the new base contains just stub modules that re-export stuff from ghc-base. It would be great to have a portable base, without any GHC-specific stuff in it. After all, modules like Control.Monad or Data.Foldable are pure Haskell2010. The only obstacle I see is that ghc-base, as you call it, uses some amount of base definitions, and so we have a loop. How hard would it be to break this loop? That is, either make GHC.* modules self-contained, or make base not to depend on GHC.*? Roman ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (was: GHC 7.8 release?)
Hi, Am Mittwoch, den 13.02.2013, 11:34 +0200 schrieb Roman Cheplyaka: It would be great to have a portable base, without any GHC-specific stuff in it. After all, modules like Control.Monad or Data.Foldable are pure Haskell2010. while you are considering to split base, please also consider separating IO out. We can expect compiling Haskell to, say JavaScript or other targets that are not processes in the usual sense. For these IO might not make sense. Having something below base that provides the pure stuff (common data structures etc.) would enable libraries to easily say: „My algorithm can be used in normal programs as well as in programs that are compiled to JS“ by not depending on base, but on, say, pure-base. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (was: GHC 7.8 release?)
Somebody claiming to be Roman Cheplyaka wrote: * Simon Marlow marlo...@gmail.com [2013-02-13 09:00:15+] It's feasible to split base, but to a first approximation what you end up with is base renamed to ghc-base, and then the new base contains just stub modules that re-export stuff from ghc-base. It would be great to have a portable base, without any GHC-specific stuff in it. After all, modules like Control.Monad or Data.Foldable are pure Haskell2010. +1 -- Stephen Paul Weber, @singpolyma See http://singpolyma.net for how I prefer to be contacted edition right joseph signature.asc Description: Digital signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
base package (Was: GHC 7.8 release?)
Hi, Am Mittwoch, den 13.02.2013, 13:58 + schrieb Ian Lynagh: If we go this route, then we would probably want to end up without a package called 'base', and then to make a new package called 'base' that just re-exports modules from all the new packages. can you transparently re-export a module from another package? I.e. if base depends on io, IO provides System.IO, is there a way for base to tell ghc to pretend that System.IO is in base, but that there is no conflict if io happens to be un-hidden as well. It seems that something like this would be required to move modules from base to something below it without breaking existing code. Also, if it works that smooth, this would not have to be one big reorganization, but could be done piece by piece. The disadvantage is that, at some point between the first release and the release that removes base, each package will have to have its dependencies updated. Why remove base? If it is just a list of dependencies and list of modules to be re-exported, then keeping it (but advocate that it should not be used) should not be too much a burden. (This is assuming that the reorganizing should not change existing module names. If your plan was to give the modules new names, this problem does not exist, but I’d rather prefer the less intrusive approach.) Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Wed, Feb 13, 2013 at 06:28:22PM +0100, Joachim Breitner wrote:
Am Mittwoch, den 13.02.2013, 13:58 + schrieb Ian Lynagh:
If we go this route, then we would probably want to end up without a
package called 'base', and then to make a new package called 'base'
that just re-exports modules from all the new packages.
can you transparently re-export a module from another package? I.e. if
base depends on io, IO provides System.IO, is there a way for base to
tell ghc to pretend that System.IO is in base, but that there is no
conflict if io happens to be un-hidden as well.
No. But there are currently no packages that depend on both base and io,
and anyone adding a dependency on io would remove the base dependency at
the same time.
It seems that something like this would be required to move modules from
base to something below it without breaking existing code.
I don't see why that's necessary. base would end up containing a load of
modules that look something like
{-# LANGUAGE PackageImports #-}
module System.IO (module X) where
import io System.IO as X
Also, if it works that smooth, this would not have to be one big
reorganization, but could be done piece by piece.
It's tricky to do it piece by piece. It's hard to remove individual
sensible pieces in the first place, and it means that you can't
subsequently move modules between packages later without breaking code
depending on the new packages.
The disadvantage is that, at some point between the first release and
the release that removes base, each package will have to have its
dependencies updated.
Why remove base? If it is just a list of dependencies and list of
modules to be re-exported, then keeping it (but advocate that it should
not be used) should not be too much a burden.
* Any package using it doesn't benefit from the reduced version bumps,
so we do actually want packages to move away from it
* Even though base (probably) wouldn't require a lot of work at any one
time, it would require a little work every now and again, and that
adds up to a lot of work
* Any time a module is added to one of the new packages, either we'd
have to spend time adding it to base too, or packages continuing to
use base wouldn't (easily) be able to use that new module.
(This is assuming that the reorganizing should not change existing
module names. If your plan was to give the modules new names, this
problem does not exist, but I’d rather prefer the less intrusive
approach.)
The odd module might be renamed, and there will probably be a handful of
definitions that move from one module to another, but for the most part
I was envisaging that we'd end up with the same modules exporting the
same things.
Thanks
Ian
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Hi, I have started a wikipage with the list of all modules from base, for a first round of shuffling, grouping and brainstorming: http://hackage.haskell.org/trac/ghc/wiki/SplitBase Am Mittwoch, den 13.02.2013, 18:09 + schrieb Ian Lynagh: On Wed, Feb 13, 2013 at 06:28:22PM +0100, Joachim Breitner wrote: Am Mittwoch, den 13.02.2013, 13:58 + schrieb Ian Lynagh: If we go this route, then we would probably want to end up without a package called 'base', and then to make a new package called 'base' that just re-exports modules from all the new packages. can you transparently re-export a module from another package? I.e. if base depends on io, IO provides System.IO, is there a way for base to tell ghc to pretend that System.IO is in base, but that there is no conflict if io happens to be un-hidden as well. No. But there are currently no packages that depend on both base and io, and anyone adding a dependency on io would remove the base dependency at the same time. hmm, that reminds me of how haskell98 was handled, and it was slightly annoying when haskell98 and base eventually were made to conflict, and we had to patch some unmaintained packages. Ok, in this case io would be introduced with the intention of being used exclusive from base. So as long as we make sure that the set of modules exported by base is always the union of all modules provided by package that have any module in common with base this would be fine. (Why this condition? Imagine io adding IO.GreatModule without base also providing the module. Then a program that still uses base cannot use IO.GreatModule without fixing the dependencies _now_ (or using package imports everywhere). It would be nice if library authors allowed to do the change whenever convenient.) Also, if it works that smooth, this would not have to be one big reorganization, but could be done piece by piece. It's tricky to do it piece by piece. It's hard to remove individual sensible pieces in the first place, and it means that you can't subsequently move modules between packages later without breaking code depending on the new packages. Agreed. The disadvantage is that, at some point between the first release and the release that removes base, each package will have to have its dependencies updated. Why remove base? If it is just a list of dependencies and list of modules to be re-exported, then keeping it (but advocate that it should not be used) should not be too much a burden. * Any package using it doesn't benefit from the reduced version bumps, so we do actually want packages to move away from it We want them to do so. We should not force them (most surely will...) * Even though base (probably) wouldn't require a lot of work at any one time, it would require a little work every now and again, and that adds up to a lot of work Hopefully it is just updating the set of modules to be exported, sounds like it could be automated, given a list of packages. * Any time a module is added to one of the new packages, either we'd have to spend time adding it to base too, or packages continuing to use base wouldn't (easily) be able to use that new module. Hence we should add them; shouldn’t be too much work. After every larger change to base I am forced to touch old, hardly maintained code that I do not know, to get the packages working in Debian again. Hence my plea for staying compatible as long as feasible. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
Somebody signing messages as Joachim Breitner wrote: I have started a wikipage with the list of all modules from base, for a first round of shuffling, grouping and brainstorming: http://hackage.haskell.org/trac/ghc/wiki/SplitBase Looks like a good start! Here's an idea: why not use the `haskell2010` package as one of the groupings? It seems like this sort of reorganisation could help solve the problem we currently have where one cannot using any of the features of `base` along with the `haskell2010` modules. -- Stephen Paul Weber, @singpolyma See http://singpolyma.net for how I prefer to be contacted edition right joseph signature.asc Description: Digital signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Wed, Feb 13, 2013 at 4:32 PM, Joachim Breitner m...@joachim-breitner.de wrote: No. But there are currently no packages that depend on both base and io, and anyone adding a dependency on io would remove the base dependency at the same time. hmm, that reminds me of how haskell98 was handled, and it was slightly annoying when haskell98 and base eventually were made to conflict, and we had to patch some unmaintained packages. Ok, in this case io would be introduced with the intention of being used exclusive from base. So as long as we make sure that the set of modules exported by base is always the union of all modules provided by package that have any module in common with base this would be fine. (Why this condition? Imagine io adding IO.GreatModule without base also providing the module. Then a program that still uses base cannot use IO.GreatModule without fixing the dependencies _now_ (or using package imports everywhere). It would be nice if library authors allowed to do the change whenever convenient.) There should also be a condition stating that base should only re-export modules, and that those re-exports need to have the same name on another package. This condition guarantees that the only thing you need to change is the import list, and even this change could be (at least partially) automated via a tool take took all your imports and decided which new packages export them. -- Felipe. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: base package (Was: GHC 7.8 release?)
On Wed, Feb 13, 2013 at 07:32:06PM +0100, Joachim Breitner wrote: I have started a wikipage with the list of all modules from base, for a first round of shuffling, grouping and brainstorming: http://hackage.haskell.org/trac/ghc/wiki/SplitBase Great, thanks for taking the lead on this! The disadvantage is that, at some point between the first release and the release that removes base, each package will have to have its dependencies updated. Why remove base? If it is just a list of dependencies and list of modules to be re-exported, then keeping it (but advocate that it should not be used) should not be too much a burden. * Any package using it doesn't benefit from the reduced version bumps, so we do actually want packages to move away from it We want them to do so. We should not force them (most surely will...) A lot of packages won't react until something actually breaks. (and I suspect many are unmaintained and unused, and won't react even once it does break). * Even though base (probably) wouldn't require a lot of work at any one time, it would require a little work every now and again, and that adds up to a lot of work Hopefully it is just updating the set of modules to be exported, sounds like it could be automated, given a list of packages. * Any time a module is added to one of the new packages, either we'd have to spend time adding it to base too, or packages continuing to use base wouldn't (easily) be able to use that new module. Hence we should add them; shouldn’t be too much work. I realised that there's actually no reason that the new 'base' package has to come with GHC (even immediately after the break-up); it can just be a package on Hackage (and, if desired, in the Haskell Platform). So it could easily be maintained by someone else, and thus be not much work for you, and 0 work for me :-) Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
