RE: Plugins, dynamic loading and GHCi
Austin is dead right. In principle you could compile a plug-in with the exact same compiler (not just version but which build was used) as the one used to build stage-1. But that's delicate and we've never thought about it, so I doubt the other ancillary things Austin mentions would work. Sorry Simon | -Original Message- | From: cvs-ghc-boun...@haskell.org [mailto:cvs-ghc-boun...@haskell.org] | On Behalf Of Austin Seipp | Sent: 17 December 2012 23:26 | To: Daniel Vainsencher | Cc: Cvs-ghc@haskell.org | Subject: Re: Plugins, dynamic loading and GHCi | | We discussed this on IRC the other day but since I have a copy of GHC | open I dug through it to refresh my memory. I don't think it would be | currently possible with the way things currently are. Someone is free to | correct me. | | To explain the reasoning, I'll parrot that paragraph and specialize it | to this problem: When you build a stage1 compiler, that stage1 compiler | is built using symbols and components of the *host* compiler. | Let's say I build GHC HEAD stage1, with 7.6.1. So the stage1 compiler is | built by the *host* compiler. It links to the host compiler libraries | and runtime. Now, let's say we hypothetically compile a plugin for | stage1 (somehow.) But now we have a conundrum if we could even load it: | that stage1 plugin requires APIs from stage1 itself (for example, maybe | the definition of Core changed in HEAD, and your plugin depends on that) | but at runtime, when those object files are loaded into stage1, those | changed APIs are definitely not there, because | stage1 was built with the host, not with itself. So there is a link-time | mismatch. | | So, when stage2 is built, we are sure at runtime it is linked to the | same APIs that comprise the stage1 compiler. In this way, stage2 can | safely dynamically link packages it has built. This same line of | thinking extends to TH and GHCi: if you had a stage1 GHCi, bytecode | would be linked into it at runtime, but it would run against the host | compiler runtime, not the same runtime the bytecode was compiled for. | | You could theoretically do some loading-multiple-package-versioning | nonsense here, but considering there are plenty of non-versioned symbols | (especially C functions,) you're not going to get very far. | | And the bootstrapping issues aside, there are other direct aspects that | are organizational: DynamicLoading (which is used to grab the 'plugin' | declaration in a GHC plugin) depends directly on GHCi's Linker module, | since it has to link packages into the compiler dynamically anyway. | Obviously, Linker.lhs is only built during stage2, provided GHCi is | enabled, for all the prior reasons. So DynamicLoading must also depend | on #ifdef GHCI in this same way. | | I don't know how relevant dynamic-by-default here is, but I see how it's | going to get you around the first issue, which is the major one. | | On Mon, Dec 17, 2012 at 5:01 PM, Daniel Vainsencher | daniel.vainsenc...@gmail.com wrote: | In SimplCore.lhs, addPlugin* is #ifdef GHCI. Is this dependency really | necessary? plugins can be useful independently of the GHCi UI. | | From the wiki [1]: | Stage 1 does not support interactive execution (GHCi) and Template | Haskell. The reason being that when running byte code we must | dynamically link the packages, and only in stage 2 and later can we | guarantee that the packages we dynamically link are compatible with | those that GHC was built against (because they are the very same | packages). | This does not seem to be a reason to not support plugins and dynamic | linking in general in stage though, or am I missing something? | | Besides finding the dependency in this direction a bit strange, | building stage 2 to debug plugin mechanism problems is a bit slow. | | Thanks, | Daniel Vainsencher | | ___ | Cvs-ghc mailing list | Cvs-ghc@haskell.org | http://www.haskell.org/mailman/listinfo/cvs-ghc | | | | -- | Regards, | Austin | | ___ | Cvs-ghc mailing list | Cvs-ghc@haskell.org | http://www.haskell.org/mailman/listinfo/cvs-ghc ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc
Re: Plugins, dynamic loading and GHCi
So first, IIUC, the dependency on GHCi is incidental, but the dependency on dynamic loading and the linker (which is currently part of GHCi) isn't. Plugins without bootstrapping may be possible, but seem a likely fiddly business. I know currently plugins are rarely used and so this question seems a bit esoteric, the context for it was the following thought. Suppose that we tried to take as much of GHC as possible out of the bootstrap loop. For example, make all of core-2-core into one or more plugins. One benefit is obvious: if you change core2core, you compile the plugins and don't bootstrap at all. If you change the bootstrapped part, but not the plugins, your bootstrap is shorter: plugins are compiled only once at the end, (or maybe not at all if you are not using them). This does mean that -O isn't available for stage1, though, apparently. Daniel On 12/18/2012 01:26 AM, Austin Seipp wrote: We discussed this on IRC the other day but since I have a copy of GHC open I dug through it to refresh my memory. I don't think it would be currently possible with the way things currently are. Someone is free to correct me. To explain the reasoning, I'll parrot that paragraph and specialize it to this problem: When you build a stage1 compiler, that stage1 compiler is built using symbols and components of the *host* compiler. Let's say I build GHC HEAD stage1, with 7.6.1. So the stage1 compiler is built by the *host* compiler. It links to the host compiler libraries and runtime. Now, let's say we hypothetically compile a plugin for stage1 (somehow.) But now we have a conundrum if we could even load it: that stage1 plugin requires APIs from stage1 itself (for example, maybe the definition of Core changed in HEAD, and your plugin depends on that) but at runtime, when those object files are loaded into stage1, those changed APIs are definitely not there, because stage1 was built with the host, not with itself. So there is a link-time mismatch. So, when stage2 is built, we are sure at runtime it is linked to the same APIs that comprise the stage1 compiler. In this way, stage2 can safely dynamically link packages it has built. This same line of thinking extends to TH and GHCi: if you had a stage1 GHCi, bytecode would be linked into it at runtime, but it would run against the host compiler runtime, not the same runtime the bytecode was compiled for. You could theoretically do some loading-multiple-package-versioning nonsense here, but considering there are plenty of non-versioned symbols (especially C functions,) you're not going to get very far. And the bootstrapping issues aside, there are other direct aspects that are organizational: DynamicLoading (which is used to grab the 'plugin' declaration in a GHC plugin) depends directly on GHCi's Linker module, since it has to link packages into the compiler dynamically anyway. Obviously, Linker.lhs is only built during stage2, provided GHCi is enabled, for all the prior reasons. So DynamicLoading must also depend on #ifdef GHCI in this same way. I don't know how relevant dynamic-by-default here is, but I see how it's going to get you around the first issue, which is the major one. On Mon, Dec 17, 2012 at 5:01 PM, Daniel Vainsencher daniel.vainsenc...@gmail.com wrote: In SimplCore.lhs, addPlugin* is #ifdef GHCI. Is this dependency really necessary? plugins can be useful independently of the GHCi UI. >From the wiki [1]: "Stage 1 does not support interactive execution (GHCi) and Template Haskell. The reason being that when running byte code we must dynamically link the packages, and only in stage 2 and later can we guarantee that the packages we dynamically link are compatible with those that GHC was built against (because they are the very same packages). " This does not seem to be a reason to not support plugins and dynamic linking in general in stage though, or am I missing something? Besides finding the dependency in this direction a bit strange, building stage 2 to debug plugin mechanism problems is a bit slow. Thanks, Daniel Vainsencher ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc
Re: Plugins, dynamic loading and GHCi
We discussed this on IRC the other day but since I have a copy of GHC open I dug through it to refresh my memory. I don't think it would be currently possible with the way things currently are. Someone is free to correct me. To explain the reasoning, I'll parrot that paragraph and specialize it to this problem: When you build a stage1 compiler, that stage1 compiler is built using symbols and components of the *host* compiler. Let's say I build GHC HEAD stage1, with 7.6.1. So the stage1 compiler is built by the *host* compiler. It links to the host compiler libraries and runtime. Now, let's say we hypothetically compile a plugin for stage1 (somehow.) But now we have a conundrum if we could even load it: that stage1 plugin requires APIs from stage1 itself (for example, maybe the definition of Core changed in HEAD, and your plugin depends on that) but at runtime, when those object files are loaded into stage1, those changed APIs are definitely not there, because stage1 was built with the host, not with itself. So there is a link-time mismatch. So, when stage2 is built, we are sure at runtime it is linked to the same APIs that comprise the stage1 compiler. In this way, stage2 can safely dynamically link packages it has built. This same line of thinking extends to TH and GHCi: if you had a stage1 GHCi, bytecode would be linked into it at runtime, but it would run against the host compiler runtime, not the same runtime the bytecode was compiled for. You could theoretically do some loading-multiple-package-versioning nonsense here, but considering there are plenty of non-versioned symbols (especially C functions,) you're not going to get very far. And the bootstrapping issues aside, there are other direct aspects that are organizational: DynamicLoading (which is used to grab the 'plugin' declaration in a GHC plugin) depends directly on GHCi's Linker module, since it has to link packages into the compiler dynamically anyway. Obviously, Linker.lhs is only built during stage2, provided GHCi is enabled, for all the prior reasons. So DynamicLoading must also depend on #ifdef GHCI in this same way. I don't know how relevant dynamic-by-default here is, but I see how it's going to get you around the first issue, which is the major one. On Mon, Dec 17, 2012 at 5:01 PM, Daniel Vainsencher daniel.vainsenc...@gmail.com wrote: In SimplCore.lhs, addPlugin* is #ifdef GHCI. Is this dependency really necessary? plugins can be useful independently of the GHCi UI. From the wiki [1]: Stage 1 does not support interactive execution (GHCi) and Template Haskell. The reason being that when running byte code we must dynamically link the packages, and only in stage 2 and later can we guarantee that the packages we dynamically link are compatible with those that GHC was built against (because they are the very same packages). This does not seem to be a reason to not support plugins and dynamic linking in general in stage though, or am I missing something? Besides finding the dependency in this direction a bit strange, building stage 2 to debug plugin mechanism problems is a bit slow. Thanks, Daniel Vainsencher ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc -- Regards, Austin ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc
Re: Plugins, dynamic loading and GHCi
Minor nitpick: On Mon, Dec 17, 2012 at 5:26 PM, Austin Seipp mad@gmail.com wrote: I don't know how relevant dynamic-by-default here is, but I see how it's going to get you around the first issue, which is the major one. s/but I see/but I can't see/ -- Regards, Austin ___ Cvs-ghc mailing list Cvs-ghc@haskell.org http://www.haskell.org/mailman/listinfo/cvs-ghc
