Re[2]: [Haskell-cafe] Re: [Haskell] Re: compiler-independent core libraries infrastructure
Hello John, (i'm moving thread to the ghc-users where this discussion continues in ghc-related aspects) Thursday, September 28, 2006, 3:30:09 AM, you wrote: So, just to confirm in my mind what you are proposing: Compiler/Version specific Core: Yhc.Core, Hugs.Core, GHC.Core With a different version for each compiler version. Tied intimately to the compiler. A large issue with this, is that what needs to be tied intimately to the complier is very different for different compilers. for instance Data.Typeable and Data.Dynamic are fairly portableish haskell on ghc, but are primitives provided by the compiler in jhc as a very simple example. module JHC.Dynamic: module Data.Dynamic: #ifdef JHC import JHC.Dynamic #else -- make portable definitions #endif in general, compiler-specific code should go into *hc-core libraries while the default implementations into the common core package i have experience of multi-OS development. it is not the solution to write program tied to API of some OS or two and then make complex translation layer for all other OSes. instead, i select the common subset of APIs and try to use only it. if that's impossible, i make the translation layers that gives OS-neutral names to common functionality and masks OS-specific details It leads to the current mess with the jhc libraries where it has bits and pieces of base. I can't just use base as is, because it implements a lot that jhc needs to implement natively or expects things that GHC.* provides, but Jhc.* doesn't, or at least provides with a different interface/semantics (and #ifdefs or tieing jhc's development with the fptools repos are not acceptable solutions) However, a lot of stuff depends on libraries provided by base, so I can't just ignore it. so I end up syncing some code from base, modifying some, having a big mess that is somewhat tricky to maintain. yes, jhc will get the largest benefit if this project succeeds - and even partial success will mean that jhc may get some better libraries than now. i don't imagine that i can develop some theoretical scheme in which each compiler will fit - vice versa, the plan is to look at all the compilers that are worth to support (now it's ghc, hugs, nhc, yhc and jhc) and realize what the common API for all them may be. if support for one more original compiler will be added in future to Core then API may need to change again. In any case, haskell-prime will clean this up some by giving a more complete compiler-provided set of libraries, and it looks like the fptools repos are moving in the right direction with modularization. i don't understand how H' will help jhc - with any library report you will need to implement it yourself :) -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: [Haskell-cafe] Re: [Haskell] Re: compiler-independent core libraries infrastructure
On Tue, Oct 03, 2006 at 06:06:02PM +0400, Bulat Ziganshin wrote: (i'm moving thread to the ghc-users where this discussion continues in ghc-related aspects) I don't see how compiler-independence is a GHC-specific topic. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re[2]: [Haskell-cafe] Re: [Haskell] Re: compiler-independent core libraries infrastructure
Hello Ross, Tuesday, October 3, 2006, 10:33:46 PM, you wrote: On Tue, Oct 03, 2006 at 06:06:02PM +0400, Bulat Ziganshin wrote: (i'm moving thread to the ghc-users where this discussion continues in ghc-related aspects) I don't see how compiler-independence is a GHC-specific topic. i mean that we discussed here splitting the base into several packages and and reordering it in order to simplify splitting GHC.* into compiler-dependent and independent bits i think that the best place for this discussion is libraries list and i continue to write here just to put together all the letters for this topic if you are interested in this project, i will be glad to hear your critics/suggestions -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: compiler-independent core libraries infrastructure
On Fri, Sep 15, 2006 at 05:20:36PM +0100, Ian Lynagh wrote: As it happens I was working on getting GHC to use cabal to build base et al on the plane the other day, and I had a brief look at this. See my comment in http://hackage.haskell.org/trac/ghc/ticket/710 for the results of my longer look at this. Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re[2]: compiler-independent core libraries infrastructure
Hello Neil, Thursday, September 14, 2006, 6:14:30 PM, you wrote: then, a base library may be written against virtual Haskell compiler, which provides uniform set of low-level features while 'base' decorates these features with user-friendly interfaces Nice idea. There are a few practical issues - for example does this virtual Haskell copmiler support higher rank types? Multi-parameter type classes? Bang patterns? It quickly gets a lot more complicated. i mean by virtual Haskell compiler set of _library_ functions, not the language features, pragmas and other differences. problem of availability of language features for standard libraries typically solved in conservative way - i.e. we try to use as less language features as possible and separate modules that use non-standard features in separate packages. for example, modern array libraries should go into separate package because they use MPTC and therefore not available for many compilers. shebang patters, of course, should be no used - it's a feature for applications, not for writers of standard packages - ghc-base/hugsbase/.. libs to implement _subset_ of common low-level API Sounds like a very good idea. - base lib to equalize several compilers and compiler versions, Yes, some operations might be implemented in the base library, but have more efficient versions in the ghc-base library. For example, map for base should be defined the obvious way, for ghc it should be defined with foldr. How can you accomodate this? i'm pragmatic and don't think that base package should hide _all_ the compiler differences and especially all compiler-specific optimizations. going this way, we should put to ghc-base byte strings and many other things my proposal is to extract from 'base' package all high-level, written in pure Haskell algorithms and put it in ~10 'application' packages, say ByteString, Array, DataStructures, FFI... 'map' definitely should be defined here, even if its definition will have ghc-specific version 'base' package should provide common API to compiler libraries. for example, it should provide functions integerMult, arrayCreate, type IO and so on *hc-base packages should provide implementations of these functions, specific for concrete compiler, i.e. arrayCreate = arrayCreate# integerMult (S# a) (S# b) = intMult# a รจ ... newtype IO = IO (...) 'base' package should then analyze compiler name and version and import appropriate modules or define operations itself if there is no implementation: module ArrayOps where #if GHC import GHC.Arr #elseif Hugs import Hugs.Arr #else type Array a b = [(a,b)] arrayCreate = [] ... #endif all functions/types implemented in pure Haskell, all complex algorithms, all class definitions should go away from this package! it just provides common set of low-level operations and of no interest for end users. it's just a tool which provides virtual Haskell compiler API, which allows to write all other libraries in rather portable way last line: i have some experience of writing compiler-independent code with Haskell and C++ and believe that this plan is realistic The differences between Haskell compilers may well be bigger than those between C++ compilers! I wish you the best of luck, and think this would be really nice to hvae - unfortunately I think its unobtainable - but if we could just get some of this goodness that would be fantastic! you've missed one point - we _already_ have working solution, the 'base' library itself. all that we need is just to split it carefully to modules which may be independently upgraded, plus add compatibility with previous compiler version that 'base' currently lacks. so, it's more moving code around and careful planning task than a real technological challenge :) i think you just misunderstood me - i don't plan to make ultimate solution, just to solve some current meaningless problems - say, that we can't use old MArray interface with ghc 6.6 or new implementation of HashTable with ghc 6.2. cabal provided us with all the instruments required - all that is remain is to refactor base library to make it compiler-version-independent -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re[2]: compiler-independent core libraries infrastructure
Hello Ian, Friday, September 15, 2006, 8:20:36 PM, you wrote: what is a 'base' library now? it is the library that implements common set of operations for latest versions of ghc, hugs and nhc. it contains low-level implementation for ghc, but relies on separate hugsbase package for hugs (the same for nhc, afaiu). so, first step is obvious - separate ghc-base library from the rest. hugsbase, ghc-base and nhc-base packages should provide common set of low-level operations, As it happens I was working on getting GHC to use cabal to build base et al on the plane the other day, and I had a brief look at this. Unfortunately there is a tangled web of dependencies, e.g. you need the low level Int# stuff in ghc-base, then Int in base, but then any other GHC-specific stuff can't use Int because it's in base. We could put everything into ghc-base and just re-export the common stuff in base, but then we can't share any code between ghc, hugs etc. I haven't looked in detail to see just how bad the problem is, but I agree it would be really good if we could split things up somehow so that base (or whatever base gets split into) is the same everywhere. yes, it is one of problems that i was overlooked (and i expect that discussing my plan will show other problems i skipped by ignorance) first, let's specify that i propose (my today letter in haskell list contains more detailed plan): ghc-base should export Int operations. why? because it can't export Int# operations, they are not supported by other compilers (as the whole unboxed type concept), so they are useless to export. 'core' library should provide some common API. *hc-core libs should provide _subset_ of this API with hope that 'core' will emulate missing features but problem your mentioned still remains - while ghc-base defines operations on Int, it don't contains class Num definition, so that (*) or (+) operations can't be used. so that can we do? we should use intMul, intAdd and other operations directly. we can even define (*) and (+) operations for _internal_ use inside our ghc-base package, but not export them. while this seems a little Draconic, it will allow us to share Num defining code with other compilers and even introduce libraries with alternative Num/(*) definitions while idea of using some internal (*), (+) ... definitions may seem like work duplication, my experience says that it's much better to define duplicate operations for internal use only rather than try to implement whole Num class inside each compiler-specific library - because this definition should be a high-quality code and we don't want to copy such code over and over again and i hope that *hc-base libraries will not use Num operations too much because their main purpose is to give standard interface to compiler-specific functions, not to implement any algorithms. for example, looking to GHC.* modules in my own ArrayRef lib (which implements boxed and unboxed arrays), i don't see any arithmetic in _process_ of rewriting base lib, we should not have problems with GHC, because we can use recursive imports. but in order to retain compatibility with Hugs we may need to move Hugs.* modules inside 'base' package (the same for nhc). well, i don't know the best plan for intermediate versions. one possible but slow variant is to introduce intAdd/... operations, then rewrite ghc.*/hugs.*/... using these operations, then move out non-core stuff and then rewrite it back... -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: compiler-independent core libraries infrastructure
Hi Bulat, Just a partial answer for now: On Wed, Sep 13, 2006 at 12:29:58PM +0400, Bulat Ziganshin wrote: Friday, September 8, 2006, 5:52:57 AM, you wrote: what is a 'base' library now? it is the library that implements common set of operations for latest versions of ghc, hugs and nhc. it contains low-level implementation for ghc, but relies on separate hugsbase package for hugs (the same for nhc, afaiu). so, first step is obvious - separate ghc-base library from the rest. hugsbase, ghc-base and nhc-base packages should provide common set of low-level operations, As it happens I was working on getting GHC to use cabal to build base et al on the plane the other day, and I had a brief look at this. Unfortunately there is a tangled web of dependencies, e.g. you need the low level Int# stuff in ghc-base, then Int in base, but then any other GHC-specific stuff can't use Int because it's in base. We could put everything into ghc-base and just re-export the common stuff in base, but then we can't share any code between ghc, hugs etc. I haven't looked in detail to see just how bad the problem is, but I agree it would be really good if we could split things up somehow so that base (or whatever base gets split into) is the same everywhere. Thanks Ian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
compiler-independent core libraries infrastructure
(i crosspost my letter into main haskell list because i hope that proposed solution is of great interest for (discussion with) many developers) Hello Ian, Friday, September 8, 2006, 5:52:57 AM, you wrote: and last question - i don't like inclusion of unix and win32 in a list of core libs. why they are here? may be it's possible to include small modules with functionality required for compiler itself in GHC.* hierarchy and then move the rest into extra libraries? This doesn't work well for any parts shared with the other compilers. core libs is possibly the wrong name for it - bootstrapping libs might be more appropriate. Note also that unix and win32 haven't been included in anything they weren't already, it's just that other libraries (those that are now called extra libs) have been taken out. We can always take more out for 6.8 if we want. thank you - you have cleaned up the situation for me. thanks to Cabal, now ghc is much more modular than 6.4. but it is not yet the ultimate solution and i propose to discuss what we can do in future, possibly even in 6.6.1. i will become devil advocate for a little :) i am, John de MacLee programmer, never planned to build ghc itself and i don't need any bootstrapping libs in my download. please sell it as separate ghc-for-ghc package :) the core-ghc package should then contain only libraries that are dependent on GHC compiler internals (GHC.* part of base, stm and th - and nothing more!) moreover, i want to be able to upgrade even these libraries without upgrading compiler proper. or, to be exact, i may need to install newer versions of these libraries which contains new features and therefore not 100% compatible with libraries shipped at the moment when GHC 6.6.1 was released why this may be impossible? first, because existing libs and programs may rely on older interfaces. i think that Cabal should eventually solve this problem so that multiple version of any lib can be installed on computer (to be exact, on concrete GHC installation) and proper version of library selected for any project. but that's another discussion.. what is a 'base' library now? it is the library that implements common set of operations for latest versions of ghc, hugs and nhc. it contains low-level implementation for ghc, but relies on separate hugsbase package for hugs (the same for nhc, afaiu). so, first step is obvious - separate ghc-base library from the rest. hugsbase, ghc-base and nhc-base packages should provide common set of low-level operations, hiding from other libraries implementation details, differences between compilers, and differences between compiler versions. they should provide _incremental_ interfaces so that old code will continue to work with newer compilers. eventually compiler-specific code for stm and th should also go into these libraries but that is not the immediate goal then, a base library may be written against virtual Haskell compiler, which provides uniform set of low-level features while 'base' decorates these features with user-friendly interfaces even more interesting variant is to allow ghc-base and other compiler-specific base packages to export non-incremental interfaces and use 'base' solely to equalize all compilers to some common interface, providing emulation of all missing features (i think that such emulation will be compiler-independent that means that it's better to put it into compiler-independent package). then, _all_ other libs should rely on version of base package version instead of version of compiler they are use. so: ghc 6.2 ghc 6.2.2 ghc 6.4.3 hugs 2003 all supported in base 1.0 package which expose stable interface independent on compiler used. all other libraries relies on this interface and therefore works with any compiler whose support included in base 1.0. as ghc 6.6 rolls out, we add its support to 'base' library, rolling out base-1.0.1. and all libraries written against base 1.0, now will work with ghc 6.6, although they can ignore some features what was not included in base 1.0 API. at the same time, base-2.0 rolled out which includes new APIs (but don't omit old ones!), supporting new features of ghc 6.6. but base 2.0 continues to support existing compilers, providing emulation of new features for old compilers. those developers that need these new features upgrade their cabal files to require base 2.0. those users of old compilers that go to compile these apps download (automatically) and install base-2.0 lib so, i propose: - ghc-base/hugsbase/.. libs to implement _subset_ of common low-level API - base lib to equalize several compilers and compiler versions, providing _full_ common low-level API. when we need to include new API, we roll out new major version of base and work hard to support old compilers by providing some emulation of new feature. base lib versions should be independent on compiler versions and as much backward-compatible as possible - all other libs to just
