Re: dataflow rewriting engine
Manuel M T Chakravarty wrote: Deborah Goldsmith: Has there been any thought about working with the LLVM project? I didn't find anything on the wiki along those lines. I have only had a rather brief look at LLVM, but my understanding at the moment is that LLVM would not be able to support one of GHC's current code layout optimisations. More precisely, with LLVM, it would not be possible to enforce that the meta data for a closure is placed right before (in terms of layout in the address space) the code executing the eval method of that same closure. GHC uses that to have the closure code pointer point directly to the eval code (and hence also by an appropriate offset) to the various fields of the meta data. If that layout cannot be ensured, GHC needs to take one more indirection to execute evals (which is a very frequent operation) - this is what an unregistered build does btw. However, I am not convinced that this layout optimisation is really gaining that much extra performance these days. In particular, since dynamic pointer tagging, very short running evals (for which the extra indirection incurs the largest overhead) have become less frequent. Even if there is a slight performance regression, I think, it would be worthwhile to consider giving up on the described layout constraint. It is the Last Quirk that keeps GHC from using standard compiler back-ends (such as LLVM), and I suspect, it is not worth it anymore. When we discussed this last, Simon Marlow planned to run benchmarks to determine how much performance the layout optimisation gains us these days. Simon, did you ever get around to that? I didn't get around to benchmarking it, but since the layout optimisation is easily switched off (it's called tablesNextToCode inside GHC) there's really nothing stopping someone from building a backend that doesn't rely on it. Everything works without this optimisation, including GHCi, the debugger, and the FFI. My guess is you'd pay a few percent on average for not doing it. You're quite right that pointer tagging makes it less attractive, but like most optimisations there are programs that fall outside the common case. Programs that do a lot of thunk evals will suffer the most. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Build system idea
On Wed, Aug 13, 2008 at 01:31:55PM +1000, Roman Leshchinskiy wrote: This makes me wonder, though. Wouldn't this model make more sense for Cabal in general than the current approach of duplicating the functionality of autoconf, make and other stuff? If it works ghc, it ought to work for other projects, too. Cabal as a preprocessor seems much more attractive to me than as a universal build system. I can't tell you how much I agree with this. the fact that cabal wants to be my build system as well as my configuration system means it is pretty much unusable to me in my projects. Features are something that _hurts_ a system such as this. between a build system, a configuration manager, a packaging system, etc, it is rare for any large project that at least one isn't imposed on you by some external constrant or just a better choice for the job. I would much rather see cabals functionality split among a variety of different programs so the pieces can be used when appropriate, not as an all or nothing thing. (bring back hmake! :) ). John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Build system idea
John Meacham [EMAIL PROTECTED] wrote: (bring back hmake! :) ). It never went away... http://www.cs.york.ac.uk/fp/hmake I even have the idea to allow hmake to read the .cabal file format for configuration data (although that is waiting for a delivery of round tuits). Regards, Malcolm ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Build system idea
On Wed, 2008-08-27 at 03:04 -0700, John Meacham wrote: On Wed, Aug 13, 2008 at 01:31:55PM +1000, Roman Leshchinskiy wrote: This makes me wonder, though. Wouldn't this model make more sense for Cabal in general than the current approach of duplicating the functionality of autoconf, make and other stuff? If it works ghc, it ought to work for other projects, too. Cabal as a preprocessor seems much more attractive to me than as a universal build system. I can't tell you how much I agree with this. the fact that cabal wants to be my build system as well as my configuration system means it is pretty much unusable to me in my projects. Features are something that _hurts_ a system such as this. between a build system, a configuration manager, a packaging system, etc, it is rare for any large project that at least one isn't imposed on you by some external constrant or just a better choice for the job. I would much rather see cabals functionality split among a variety of different programs so the pieces can be used when appropriate, not as an all or nothing thing. (bring back hmake! :) ). People are of course still free to use autoconf and make to implement their own build system and have it still be a Cabal package (which has the advantage of presenting the same meta-data and command interface to packaging tools). It's been that way since the original design. Quite a few packages to use autoconf though the use seems to be slightly on the decline as people try and make their packages portable to Windows. Very few packages use make as it involves re-implementing their own build system which is a lot of work. That's partly a self-fulfilling prophecy of course because nobody uses that interface so it does not get improved so nobody uses it etc. Also, as far as I'm aware hmake still works, at least for nhc, I've not used it recently for building with ghc. So there's nothing stopping people from using that (except hard work), even as part of a cabal package. The different parts of the system are relatively separated. The declarative bits that deal with package meta-data (.cabal files) are available through the Cabal library (Distribution.*) and many tools make use of this. Then the 'Simple' build system is in the same library but fairly cleanly separated (Distribution.Simple.*). As I mentioned, you do not have to use the 'Simple' build system, but the vast majority of packages do. Then there are the packaging tools like the tools for converting to native packages and cabal-install which use the Cabal library and the command line interface that Cabal packages present. I'm not saying it's perfect, but it's not as monolithic as some would suggest. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: dataflow rewriting engine
| I think we're all rather excited about seeing this stuff land. | What's the expected timeline, wrt. ghc 6.10's release? Good question. I've updated the overview here http://hackage.haskell.org/trac/ghc/wiki/Commentary/Compiler/NewCodeGen to say what we plan. Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Build system idea
The problem with the way cabal wants to mix with make/autoconf is that it is the wrong way round. make is very good at managing pre-processors, dependency tracking and calling external programs in the right order, in parallel, and as needed. cabal is generally good at building a single library or executable given relatively straightforward haskell source. (I know it _can_ do more, but this is mainly what it is good at). The way this should work is that make determines what haskell libraries need to be built, and what haskell files need to be generated to allow cabal to run and calls cabal to build just the ones needed. cabal as a build tool that make calls is much more flexible and in tune with each tools capabilities. The other issue is with cabal files themselves which are somewhat conflicted in purpose. on one hand, you have declarative stuff about a package. name, version, etc... information you want before you start to build something. but then you have build-depends, which is something that you cannot know until after your configuration manager (whatever it may be, autoconf being a popular one) is run. What packages you depend on are going to depend on things like what compiler you have installed, your configuration options, which packages are installed, what operating system you are running on, which kernel version you are running, which c libraries you have installed. etc. things that cannot be predicted before the configuration is actually run. Then you have cabal as a packaging system (or perhaps hackage/cabal considered together). Which has its own warts, if it is meant to live in the niche of package managers such as rpm or deb, where are the 'release' version numbers that rpms and debs have for one example? If it is meant to be a tarball like format, where is the distinction between 'distribution' and 'source' tarballs? For instance, jhc from darcs for developers requires perl,ghc,DrIFT,pandoc,autotools, and happy. however the jhc tarball requires _only_ ghc. nothing else. This is because the make dist target is more interesting than just taring up the source. (and posthooks/prehooks don't really help. they are sort of equivalent to saying 'write your own build system'.) One of the biggest sources of conflict arise from using cabal as a configuration manager. A configuration managers entire purpose is to examine the system and figure out how to adapt your programs build to the system. this is completely 100% at odds with the idea of users having to 'upgrade' cabal. Figuring out how to adapt your build to whatever cabal is installed or failing gracefully if you can't is exactly the job of the configuration manager. something like autoconf. This is why _users_ need not install autoconf, just developers. since autoconf generates a portable script is so that users are never told to upgrade their autoconf. if a developer wants to use new features, he gets the new autoconf and reruns 'autoreconf'. The user is never asked to update anything that isn't actually needed for the project itself. This distinction is key fora configuration manager and really conflicts with cabal wanting to also be a build system and package manager. It is also what is needed for forwards and backwards compatibility. All in all, I think these conflicting goals of cabal make it hard to use in projects and have led to very odd design choices. I think external tools should not be the exception but rather the rule. Not that cabal shouldn't come with a full set of said tools. But as long as they are integrated I don't see cabal's design problems being fixed, meerly augmented with various work-arounds. John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: dataflow rewriting engine
Wow, lots of great information. We'll take a look at the papers and get back if there's any remaining confusion. Thanks! Chad ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Build system idea
On Wed, 2008-08-27 at 06:13 -0700, John Meacham wrote: The problem with the way cabal wants to mix with make/autoconf is that it is the wrong way round. make is very good at managing pre-processors, dependency tracking and calling external programs in the right order, in parallel, and as needed. cabal is generally good at building a single library or executable given relatively straightforward haskell source. (I know it _can_ do more, but this is mainly what it is good at). The way this should work is that make determines what haskell libraries need to be built, and what haskell files need to be generated to allow cabal to run and calls cabal to build just the ones needed. cabal as a build tool that make calls is much more flexible and in tune with each tools capabilities. I'd say if you're using make for all that, then use it to build the haskell modules too. That gives the advantage of incremental and parallel builds, which Cabal does not do yet (though we've got a GSoC project just coming to an end which does this). The other issue is with cabal files themselves which are somewhat conflicted in purpose. on one hand, you have declarative stuff about a package. name, version, etc... information you want before you start to build something. but then you have build-depends, which is something that you cannot know until after your configuration manager (whatever it may be, autoconf being a popular one) is run. Ah, but that's where the autoconf and Cabal models part ways. What packages you depend on are going to depend on things like what compiler you have installed, your configuration options, which packages are installed, what operating system you are running on, which kernel version you are running, which c libraries you have installed. etc. things that cannot be predicted before the configuration is actually run. So Cabal takes the view that the relationship between features and dependencies should be declarative. autoconf is essentially a function from a platform environment to maybe a configuration. That's a very flexible approach, the function is opaque and can do whatever feature tests it likes. The downside is that it is not possible to work out what the dependencies are. It might be able to if autoconf explained the result of its decisions, but even then, it's not possible to work out what dependencies are required to get a particular feature enabled. With the Cabal approach these things are explicit. The conditionals in a .cabal file can be read in either direction so it is possible for a package manager to automatically work out what deps would be needed for that optional libcurl feature, or GUI. The other principle is that the packager, the environment is in control over what things the package 'sees'. With autoconf, the script can take into account anything it likes, even if you'd rather it did not. Eg it's important to be able to build a package that does not have that optional dependency, even though the C lib is indeed installed on the build machine, because I may be configuring it for a machine without the C lib. Sure, some good packages allow those automagic decisions to be overridden, but many don't and of course there is no easy way to tell if it's picking up deps it should not. So one of the principles in Cabal configuration is that all decisions about how to configure the package are transparent to the packager and can be overridden. Now currently, Cabal only has a partial implementation of the concept because when it tries to find a configuration that works in the current environment (which it only does if the configuration is not already fully specified by the packager) it only considers dependencies on haskell packages. Obviously there are a range of other dependencies specified in the .cabal file and it should use them all, in particular external C libs. So I accept that we do not yet cover the range of configuration choices that are needed by the more complex packages (cf darcs), but I think that we can and that the approach is basically sound. The fact that we can automatically generate distro packages for hundreds of packages is not insignificant. This is just not possible with the autoconf approach. Then you have cabal as a packaging system (or perhaps hackage/cabal considered together). Which has its own warts, if it is meant to live in the niche of package managers such as rpm or deb, where are the 'release' version numbers that rpms and debs have for one example? If it is meant to be a tarball like format, where is the distinction between 'distribution' and 'source' tarballs? Right, it's supposed to be the upstream release format, tarballs. Distro packages obviously have their additional revision numbers. For instance, jhc from darcs for developers requires perl,ghc,DrIFT,pandoc,autotools, and happy. however the jhc tarball requires _only_ ghc. nothing else. This is because the make dist target is more interesting than just
Re: Build system idea
On Wed, Aug 27, 2008 at 10:18:59PM +0100, Duncan Coutts wrote: On Wed, 2008-08-27 at 06:13 -0700, John Meacham wrote: The problem with the way cabal wants to mix with make/autoconf is that it is the wrong way round. make is very good at managing pre-processors, dependency tracking and calling external programs in the right order, in parallel, and as needed. cabal is generally good at building a single library or executable given relatively straightforward haskell source. (I know it _can_ do more, but this is mainly what it is good at). The way this should work is that make determines what haskell libraries need to be built, and what haskell files need to be generated to allow cabal to run and calls cabal to build just the ones needed. cabal as a build tool that make calls is much more flexible and in tune with each tools capabilities. I'd say if you're using make for all that, then use it to build the haskell modules too. That gives the advantage of incremental and parallel builds, which Cabal does not do yet (though we've got a GSoC project just coming to an end which does this). So, don't use cabal at all? that is the solution I have been going with so far and am trying to remedy. The other issue is with cabal files themselves which are somewhat conflicted in purpose. on one hand, you have declarative stuff about a package. name, version, etc... information you want before you start to build something. but then you have build-depends, which is something that you cannot know until after your configuration manager (whatever it may be, autoconf being a popular one) is run. Ah, but that's where the autoconf and Cabal models part ways. What packages you depend on are going to depend on things like what compiler you have installed, your configuration options, which packages are installed, what operating system you are running on, which kernel version you are running, which c libraries you have installed. etc. things that cannot be predicted before the configuration is actually run. So Cabal takes the view that the relationship between features and dependencies should be declarative. autoconf is essentially a function from a platform environment to maybe a configuration. That's a very flexible approach, the function is opaque and can do whatever feature tests it likes. The downside is that it is not possible to work out what the dependencies are. It might be able to if autoconf explained the result of its decisions, but even then, it's not possible to work out what dependencies are required to get a particular feature enabled. With the Cabal approach these things are explicit. unfortunately the cabal approach doesn't work. note, I am not saying a declarative configuration manager won't work. in fact, I have sketched a design for one on occasion. but cabal's particular choices are broken. It is treading the same waters that made 'imake' fail. the ideas of forwards and backwards compatability are _the_ defining features of a configuration manager. Think about this, I can take my old sunsite CD, burned _ten years_ ago and take the unchanged tarballs off that CD and ./configure make and in general most will work. many were written before linux even existed, many were written with non gcc compilers, yet they work today. The cabal way wasn't able to handle a single release of ghc and keep forwards or backwards compatability. That any project ever had to be changed to use the flag 'split-base' is a travesty. What about all the projects on burnt cds or that don't have someone to update them? 20 years from now when we are all using 'fhc' (Fred's Haskell Compiler) will we still have this reference to 'split-base' in our cabal files? how many more flags will have accumulated by then? Sure it's declarative, but in a language that doesn't make sense without the rule-book. autoconf tests things like 'does a library named foo exist and export bar'. 'is char signed or unsigned on the target system'. those are declarative statement and have a defined meaning through all time. (though, implemented in a pretty ugly imperative way) That is what allows autoconfed packages to be compiled by compilers on systems that were never dreamed of when the packages were written. The conditionals in a .cabal file can be read in either direction so it is possible for a package manager to automatically work out what deps would be needed for that optional libcurl feature, or GUI. In the cabal framework Will cabal be able to do things like cross compile a c file to an object file, and deconstruct the generated ELF file to determine parameters needed for an unknown embedded platform _and_ do so without me requiring the user to upgrade their cabal? This is an example of the type of autoconf test that comes up in the real world. You can never come up with a language that will have every needed primitive, any restricted set will ultimately not be enough for someone. and
