Re: Understanding the -A and the -H flags
On 25/02/2012 16:51, Johan Tibell wrote: Hi! I'm trying to understand the interaction between the -A and -H RTS flags. The documentation at http://www.haskell.org/ghc/docs/7.4.1/html/users_guide/runtime-control.html says that if you use -H (with or without an argument) it implicitly implies some value of -A. However, it's not clear to me what value -A will get and how that value is related to the value of -H. For example, if I set the suggested heap size to 1G, using -H1G, surely the size of the nursery (-A) won't be whatever is left over, but something more reasonable e.g. the size of the L2 cache? Perhaps it would make sense to document the actual algorithm used to set -A given -H (with and without argument.) Hmm, I took a look at the docs and to me it seems clear (but it would do, since I wrote the docs :-) Think of -Hsize as a variable -A option. It says: I want to use at least size bytes, so use whatever is left over to increase the -A value. Doesn't that describe exactly what it means? Well, actually it's a bit more complicated than that, and there are some heuristics involved. But the basic idea is to use all of the memory granted by -H for the heap, by increasing -A to fill any free space. The complications arise because we don't know how much of the nursery will need to be copied at the next GC, and the worst case (all of it) very often leaves a lot of memory unused, so we make an approximation. Sometimes this is an underestimate, and we end up using more memory than the -H value for a while. Whether -H is a good idea is not clear. When I added it, it was for backwards compatibilty with the previous GC, which had a fixed-size heap and required that you specify the heap size with -H. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Removal of #include HsFFI.h from template-hsc.h breaks largefile support on 32bit Linux
On 17/02/2012 22:51, John Meacham wrote: On Fri, Feb 17, 2012 at 2:12 PM, Simon Marlowmarlo...@gmail.com wrote: On 17/02/12 19:36, John Meacham wrote: It isn't local to a file though because it changes the ABI, for instance void foo(off_t *x); it will blow up if called from a file with a differently sized off_t. But we're talking about Haskell code here, not C code. There's no way for something to blow up, the typechecker will catch any discrepancies. Perhaps I don't understand what problem you're thinking of - can you give more detail? Someone writes a C function that returns an off_t * that is foreign imported by a haskell program using Ptr COff, the haskell program then writes to the output pointer with the COff Storable instance. However the imported function was compiled without 64 bit off_t's so it only allocated 32 bits to be written into so some other memory gets overwritten with garbage. 64 bit off_t's change the ABI of called C functions much like passing -m32 or -mrtd does so should be considered a all-or-nothing sort of thing. In particular, when ghc compiles C code, it should make sure it does it with the same ABI flags as the rest of the thing being compiled. So I'm not sure I agree with this. If someone is writing some C code and some Haskell code that calls it, it is their responsibility to make sure they get it right. Furthermore, GHC should not be imposing any choices on end users - what if you need to call an API that uses the 32-bit off_t, for example? We've had all kinds of trouble in the past with GHC's configuration details leaking into user's code, which is why we try to be as clean as possible now. There are no ABI issues here - it's just a matter of the contract between the Haskell code and the C code, which is completely under the control of the user. The only place where there the choices we make in GHC affect the user's code is the definition of the COff type. We should make it clear in the documentation for COff which version of off_t it corresponds to (or maybe even have two versions), but IMO that's all we should do. Cheers, Simon John environment as the rest of the code. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Unpack primitive types by default in data
On 17/02/2012 20:10, Johan Tibell wrote: nofib probably has, to a first approximation, zero strictness annotations. Because most of the programs in there predate the addition of strictness annotations to Haskell. That's good for us. In what way? The nofib programs have no strictness annotations, so they won't be affected by the optimisation, so the results won't tell us anything at all. (am I missing something here?) Cheers, Simon The downside of nofib is that it probably doesn't represent real world Haskell programs well, as they tend to use more packed types, such as ByteString, Text, and Vector. Still, it's a good start. GHC itself would be a good benchmark, incidentally... Indeed, that's what I thought as well. How do I test the build time of GHC? By building my modified GHC in one build tree and then use it to build another GHC in another (clean) build tree? -- Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: parallelizing ghc
On 17/02/2012 18:12, Evan Laforge wrote: Sure, except that if the server is to be used by multiple clients, you will get clashes in the PIT when say two clients both try to compile a module with the same name. The PIT is indexed by Module, which is basically the pair (package,modulename), and the package for the main program is always the same: main. This will work fine if you spin up a new server for each program you want to build - maybe that's fine for your use case? Yep, I have a new server for each CPU. So compiling one program will start up (say) 4 compilers and one server. Then shake will start throwing source files at the server, in the proper dependency order, and the server will distribute the input files among the 4 servers. Each server is single-threaded so I don't have to worry about calling GHC functions reentrantly. But --make is single-threaded as well, so why doesn't it just call compileFile repeatedly and instead bother with all that HPT stuff? Is it just for ghci? That might be true, but I'm not completely sure. The HPT stuff was added with a continuous edit-recompile cycle in mind (i.e. for GHCi), and we added --make at the same time because it fitted nicely. It might be that just calling compileFile repeatedly works, and it would end up storing the interfaces for the home-package modules in the PackageIfaceTable, but we never considered this use case. One thing that worries me: will it be reading the .hi file for a module off the disk after compiling it? I suspect it might, whereas the HPT method will be caching the iface in the HPT. The 'user' is low for the server because it doesn't count time spent by the subprocesses on the other end of the socket, but excluding linking it looks like I can shave about 25% off compile time. Unfortunately it winds up being just about the same as ghc --make, so it seems too low. But that's what you expect, isn't it? It's surprising to me that the serial --make is just about the same speed as a parallelized one. The whole point was to compile faster! Ah, so maybe the problem is that the compileFile method is re-reading .hi files off the disk (and typechecking them), and that is making it slower. Granted, each interface has to be loaded for each processor while --make only needs to do it once, but once loaded they should stay loaded and I'd expect the benefit from two processors would win out pretty quickly. --make has a slight advantage for linking in that it knows which packages it needs to link against, whereas plain ghc will link against all the packages on the command line. Ohh, so maybe with --make it can omit some packages and do less work. Let me try minimizing the -packages and see if that helps. As an aside, it would be handy to be able to ask ghc given this main module, which -packages should the final program get? but not actually compile anything. Is there a way to do that, short of writing my own with the ghc api? Would it be a reasonable ghc flag, along the lines of -M but for packages? I don't think we can calculate the package dependencies without knowing the ModIface, which is generated by compiling (or at least typechecking) each module. Cheers, Simon BTW, in case anyone is interested, a darcs repo is at http://ofb.net/~elaforge/ghc-server/ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Unpack primitive types by default in data
When you said strict my brain read UNPACK. If there are no strictness annotations nofib isn't of much use in this experiment. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Understanding the -A and the -H flags
Hi Simon, On Mon, Feb 27, 2012 at 12:25 AM, Simon Marlow marlo...@gmail.com wrote: Think of -Hsize as a variable -A option. It says: I want to use at least size bytes, so use whatever is left over to increase the -A value. Doesn't that describe exactly what it means? Maybe. Let me start with the mental model I approach this with: the allocation area (i.e. the nursery) should have a size in the order of megabytes, often around the size of the L2 cache. Given this model, I read the above as: * if you set e.g. -H1G, you'll get an allocation area which is in the order of 1Gb large. That makes no sense to me. * The suggested size of the total heap (-H) has something to do with the size of the allocation area (-A). This makes no sense to me either. So either I do understand what -H does, but it makes no sense to me, or I don't understand what -H does, but what it does makes sense. Perhaps the confusion lies in the phrase left over. Left over from what? Cheers, Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: ghci 7.4.1 no longer loading .o files?
Evan Laforge wrote: Is there something that changed in 7.4.1 that would cause it to decide to interpret .hs files instead of loading their .o files? E.g.: Brandon Allbery wrote: I thought this was deliberate because the debugger won't work with object files? Oh I hope not. I almost never use the debugger, and it's so much slower to re-interpret all those modules. I am surprised about this complaint. I have never noticed any significant delay for re-interpreting, even when working on projects that are quite large, with tens of thousands of LOC. I have always found the behavior of using .o files by default surprising and annoying. The i in GHCi stands for interactive. I expect work in GHCi to be as interactive as possible in every way, including having access to the debugger. I expect performance to be similar to the usual performance of interpreter shells in any language; I don't mind if it doesn't match the speed of compiled Haskell. It's nice if there is a way for experts to load .o files in GHCi, e.g., for the rare case where the performance difference for some specific module is so great that you can't work effectively interactively in some other module that imports it. There could be something to set in .ghci for people who do like that behavior all the time, perhaps. But it should not be the default. GHCi is headed in that direction in many ways, and I think that's great. I don't think more flags should be excluded from the fingerprint by default if that would detract in any way from the interactive experience. In particular, it is especially important for -XNoMonomorphismRestriction to be the default in GHCi. See also these tickets: http://hackage.haskell.org/trac/ghc/ticket/3217 http://hackage.haskell.org/trac/ghc/ticket/3202 Thanks, Yitz ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: ghci 7.4.1 no longer loading .o files?
On Mon, Feb 27, 2012 at 9:56 AM, Yitzchak Gale g...@sefer.org wrote: Evan Laforge wrote: Is there something that changed in 7.4.1 that would cause it to decide to interpret .hs files instead of loading their .o files? E.g.: Brandon Allbery wrote: I thought this was deliberate because the debugger won't work with object files? Oh I hope not. I almost never use the debugger, and it's so much slower to re-interpret all those modules. I am surprised about this complaint. I have never noticed any significant delay for re-interpreting, even when working on projects that are quite large, with tens of thousands of LOC. Really really? I'm even more surprised! Maybe you have a really fast computer? I just tested a module moderately high in the app, which drags in 134 modules (out of 300 or so). When it re-interprets them all, it takes around 16 seconds. When it can load them all from .o files, it's around 1 second. Usually what happens is I'll be testing, and some test will fail. If I edit some module and reload only the dependent modules get reloaded, which is usually ok. But if I switch to another module with :l instead of :r, or if there was a particularly bad syntax error, it will want to reload everything. There's a big difference between waiting under a second to try a change or experiment with a function and waiting 16 seconds! It doesn't feel interactive anymore. This is on a new MacBook Pro with an SSD, which is a relatively high end laptop. The other thing is that I use hint to insert a REPL into the app. This may be a result of me using hint wrong, or how hint uses the ghc API, but every expression apparently does the equivalent of a complete reload. It's ok when each expression takes about 1s, but not when they all take 16s each! I need to to some research here to figure out if I can implement something more ghci-like, specifically :reload. I have always found the behavior of using .o files by default surprising and annoying. The i in GHCi stands for interactive. I expect work in GHCi to be as interactive as possible in every way, including having access to the debugger. I expect performance to be similar to the usual performance of interpreter shells in any language; I don't mind if it doesn't match the speed of compiled Haskell. Oh, I don't care so much about the speed of the code being executed, it's how long it takes that code to load into ghci. GHCi is headed in that direction in many ways, and I think that's great. I don't think more flags should be excluded from the fingerprint by default if that would detract in any way from the interactive experience. In particular, it is especially important for -XNoMonomorphismRestriction to be the default in GHCi. Yeah, I used to have that on for ghci, but it in 7.4.1 it makes it not load the .o files. I agree it's the right choice for ghci, but to me it's not worth the no-longer-interactive caveat it comes with. One other point about the fingerprint thing, is that it also affects --make. That can be the difference between a 1m build and a 15m build. I'm insulated from this because I don't use --make anymore, but it's still very convenient for pure haskell projects. Not to mention cabal uses it. Of course getting mysterious link errors due to inconsistent flags can waste much more than 15m of debugging time, so we should definitely avoid those. But I think the current settings are too conservative. See also these tickets: http://hackage.haskell.org/trac/ghc/ticket/3217 http://hackage.haskell.org/trac/ghc/ticket/3202 Ya, I agree with both of those tickets, but I think they're orthogonal. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: ghci 7.4.1 no longer loading .o files?
On Monday 27 February 2012, 18:56:47, Yitzchak Gale wrote: It's nice if there is a way for experts to load .o files in GHCi, e.g., for the rare case where the performance difference for some specific module is so great that you can't work effectively interactively in some other module that imports it. Is that so rare? For me it's pretty standard that the core modules _have_ to be loaded as object files, interpreting them would make things orders of magnitude slower (100× - 1000×), they'd be unusably slow. So in my opinion it's absolutely essential that modules can be loaded as object files. There could be something to set in .ghci for people who do like that behavior all the time, perhaps. And that too, if it's no longer the default. But it should not be the default. But with it not being the default, I could live well. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Records in Haskell
Greg Weber greg at gregweber.info writes: What on earth do you mean by not automatically abstract over fields? Abstraction over fields is the ability to write a function that works on two records with the same field label. Thanks Greg, I see you've put something on the wiki about abstract over fields. The example code doesn't seem to be right, so I'm still confused what you mean ... The example has: getA = r.a I think that should be: getA r = r.a (Otherwise how does `getA` know which record to extract the `a` from?) And you give no type signature for getA so (at a wild guess) by abstract you mean that `getA` extracts the `a` from any record `r` with an `a` field(?) Under SORF: r.a desugars to a call to the `get` method. Under DORF: r.a desugars to (a r), then we can eta-reduce: getA r = r.a == (a r) getA = a Either way, I don't see that `getA` is adding anything over plain field `a`. There _is_ a difference between DORF and SORF here. In DORF I can declare: getF f r = r.f -- works for any field of any record -- RHS desugars to (f r), so getF === ($) And can use it, for example: getF lastName cust1 getF fullName person2 I don't think you can do this is SORF (but please check with SPJ). In particular, I don't think you could call this function and pass an argument into it for the field name. That's because in SORF the dot notation is desugarred at the point it occurs (according to the wiki on SORF), and the `f` appearing on the RHS is a bound variable, not a field name as such. (In fact, I wonder if SORF would take the dot notation and try to desugar it to field f, then either get a type failure or (by accident) extract the wrong field.) Please correct this on the wiki. AntC ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Records in Haskell
On Mon, Feb 27, 2012 at 4:52 PM, AntC anthony_clay...@clear.net.nz wrote: And can use it, for example: getF lastName cust1 getF fullName person2 I don't think you can do this is SORF (but please check with SPJ). In particular, I don't think you could call this function and pass an argument into it for the field name. You only need some way to write a value of a type of kind String; the section should Get have a proxy argument of the SORF wiki page discusses approaches to this problem. This example demonstrates the flexibility of the Proxy-based approach, but does not distinguish between DORF and SORF. /g P.S. Perhaps we should find record proposals that don't sound like noises my cat makes when coping with a hairball? ;) -- Would you be so kind as to remove the apricots from the mashed potatoes? ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Records in Haskell -- updating higher-ranked fields: required?
Greg Weber greg at gregweber.info writes: No, I don't think anybody has a satisfactory approach to updating polymorphic/higher-ranked fields. (DORF mentions one, but it's a ghastly hack. So are the proposals dead until this is tackled, or should SORF/DORF propose not to allow that? That's an important question. I've asked in the DORF proposal how big is the demand to update h-r fields? Note that is something you can do currently with H98 records/fields. Is it good enough to be able to extract and use h-r fields in polymorphic contexts? (Both SORF and DORF can do that.) Is it good enough to be able to create records with h-r fields (using the data constructor)? (Both SORF and DORF can do that.) AntC ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
