[O] bug#10125: RFE: require and load-path-shadowing
Kevin Rodgers kevin.d.rodg...@gmail.com writes: (defun run-emacs (command) Run the Emacs COMMAND in the background via `shell-command'. (interactive (let ((program (expand-file-name invocation-name invocation-directory))) (list (read-string Emacs command: (cons (concat program (if (cdr command-line-args) (mapconcat 'identity (cdr command-line-args) You need to use shell-quote-argument to properly shell-quote each argument. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different.
[O] bug#10125: RFE: require and load-path-shadowing
On 1/15/13 12:34 PM, Achim Gratz wrote: Achim Gratz writes: + (concat invocation-directory invocation-name) Better make that + (expand-file-name invocation-name invocation-directory) FWIW here's what I use: (defun run-emacs (command) Run the Emacs COMMAND in the background via `shell-command'. (interactive (let ((program (expand-file-name invocation-name invocation-directory))) (list (read-string Emacs command: (cons (concat program (if (cdr command-line-args) (mapconcat 'identity (cdr command-line-args) ) -Q) ) (1+ (length program))) (shell-command command)) -- Kevin Rodgers Denver, Colorado, USA
[O] bug#10125: RFE: require and load-path-shadowing
On 1/16/13 3:06 AM, Andreas Schwab wrote: Kevin Rodgerskevin.d.rodg...@gmail.com writes: (defun run-emacs (command) Run the Emacs COMMAND in the background via `shell-command'. (interactive (let ((program (expand-file-name invocation-name invocation-directory))) (list (read-string Emacs command: (cons (concat program (if (cdr command-line-args) (mapconcat 'identity (cdr command-line-args) You need to use shell-quote-argument to properly shell-quote each argument. Thanks, Andreas! -- Kevin Rodgers Denver, Colorado, USA
[O] bug#10125: RFE: require and load-path-shadowing
Achim Gratz writes: + (concat invocation-directory invocation-name) Better make that + (expand-file-name invocation-name invocation-directory) Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
[O] bug#10125: RFE: require and load-path-shadowing
Eli Zaretskii writes: Sorry, I don't follow: new packages can only affect emacs -Q if you re-dump Emacs in between. Am I missing something? No, for package manager emacs -Q should work well enough. The fork/spawn thing might still be useful to make this more efficient or (if you allow to create such fork points programmatically) to enable Emacs to easily backtrack to some earlier state. But that should be discussed in emacs-devel under a different headline. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
[O] bug#10125: RFE: require and load-path-shadowing
From: Stefan Monnier monn...@iro.umontreal.ca Date: Fri, 11 Jan 2013 17:52:33 -0500 Cc: 10...@debbugs.gnu.org I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. I've been thinking about something like this for a while… if it worked at least as well as starting a new Emacs instance on all platforms, I'd favor this approach. IIUC fork is not really an option for w32. For the intended application spawn should work as well? Could be: depends on the precise semantics of spawn, which I don't know. I'm barely following this thread, so please tell what semantics do you need, and I will then try figuring out whether spawn does what you need.
[O] bug#10125: RFE: require and load-path-shadowing
Achim Gratz writes: Stefan Monnier writes: Yes, this subroutine is never directly called from C, so placing an advice should work just fine. OK, I'll give it a try. Here's what I've come up with: --8---cut here---start-8--- ;; some parts of Org might already have been used from a different ;; place, try to reload these parts from the current load-path (defadvice require (before org-require-reload-when-shadowed (feature optional filename noerror) activate compile preactivate) Check whether a required feature has been shadowed by changing `load-path' after it has been loaded and reload that feature from current load-path in this case. (when (featurep feature) (let ((feature-name (or filename (symbol-name feature (when (string-match ^\\(org\\|ob\\)[.-] feature-name) (let ((feature-lib (file-name-directory (or (locate-library feature-name) ))) (feature-dir (file-name-directory (feature-file feature ;(message require-reload-when shadowed %s\n\t%s\n\t%s feature-name feature-lib feature-dir) (when (not (string= feature-lib feature-dir)) (message Reloading %s feature-name) (unload-feature feature t))) (require 'org-macs) (require 'org-compat) (require 'org-entities) (require 'org-faces) (require 'org-list) (require 'org-pcomplete) (require 'org-src) (require 'org-footnote) ;; babel (require 'ob) (eval-and-compile (ad-unadvise 'require)) --8---cut here---end---8--- This takes care of a situation when Org is activated after some parts of it have already been loaded (maybe via autoload) and the load-path has been changed inbetween. This uses unload-feature although it would probably work to just (setq features (delq feature features)). For package manager, a slightly more general version of the above (not conditionalized on the feature starting with org or ob) could be used around the package compilation. I haven't yet tried this, though. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
[O] bug#10125: RFE: require and load-path-shadowing
Eli Zaretskii writes: Could be: depends on the precise semantics of spawn, which I don't know. I'm barely following this thread, so please tell what semantics do you need, and I will then try figuring out whether spawn does what you need. The idea IIUC is to fork/spawn Emacs after it has initialized to be able to use a pristine instance to spawn off again for byte compilation. emacs-pristine +--- emacs-user | +--- emacs-bytecompile-1 | +--- emacs-bytecompile-2 | +--- emacs-bytecompile-3 Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
[O] bug#10125: RFE: require and load-path-shadowing
From: Achim Gratz strom...@nexgo.de Date: Sat, 12 Jan 2013 11:20:58 +0100 Eli Zaretskii writes: Could be: depends on the precise semantics of spawn, which I don't know. I'm barely following this thread, so please tell what semantics do you need, and I will then try figuring out whether spawn does what you need. The idea IIUC is to fork/spawn Emacs after it has initialized to be able to use a pristine instance to spawn off again for byte compilation. emacs-pristine +--- emacs-user | +--- emacs-bytecompile-1 | +--- emacs-bytecompile-2 | +--- emacs-bytecompile-3 How is this different from invoking emacs-bytecompile-N instances in parallel to invoking emacs-pristine? IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q?
[O] bug#10125: RFE: require and load-path-shadowing
IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q? Yes, because the current Emacs may be a different executable than the one the shell would run in response to emacs -Q. Stefan
[O] bug#10125: RFE: require and load-path-shadowing
Achim Gratz strom...@nexgo.de writes: Achim Gratz writes: Stefan Monnier writes: Yes, this subroutine is never directly called from C, so placing an advice should work just fine. I thought coding conventions prevented advising primitives? I does not look clean to advise `require' here, just for Org. Check whether a required feature has been shadowed by changing `load-path' after it has been loaded and reload that feature from current load-path in this case. I don't understand why we need this. This is at best a workaround to fix dirty installations. I updated Org's installation process yesterday: http://orgmode.org/org.html#Installation I made it clear that the user, if she wants to install Org on top of the pre-bundled version, must choose between on of the three methods. For each method, I explain how to set up the load-path (if needed) and org.el will load the correct org-loaddefs.el, resetting autoloads correctly. In org.el, no Org package is required before org-loaddefs.el is loaded. This takes care of a situation when Org is activated after some parts of it have already been loaded (maybe via autoload) and the load-path has been changed inbetween. I don't see why this should happen. Setting the correct load-paths is the first thing user should do in their .emacs.el. -- Bastien
[O] bug#10125: RFE: require and load-path-shadowing
From: Stefan Monnier monn...@iro.umontreal.ca Cc: Achim Gratz strom...@nexgo.de, 10...@debbugs.gnu.org Date: Sat, 12 Jan 2013 08:28:29 -0500 IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q? Yes, because the current Emacs may be a different executable than the one the shell would run in response to emacs -Q. And if we make sure the same executable is run (easy on Windows)? Are there any other differences?
[O] bug#10125: RFE: require and load-path-shadowing
Yes, this subroutine is never directly called from C, so placing an advice should work just fine. I thought coding conventions prevented advising primitives? Nothing prevents advising primitive functions (advising special forms is another matter altogether). But primitive functions can be called from C directly (rather than via looking up the symbols' function cell) in which case the advice will be ignored. I does not look clean to advise `require' here, just for Org. It's just an experiment (AFAIK). Check whether a required feature has been shadowed by changing `load-path' after it has been loaded and reload that feature from current load-path in this case. I don't understand why we need this. For the case where Org is installed via package.el rather than by manually downloading and following some installation instructions. For package manager, a slightly more general version of the above (not conditionalized on the feature starting with org or ob) could be used around the package compilation. I haven't yet tried this, though. Maybe the better way to do it is for package.el to compare the set of files of the new package, with the set of currently loaded `features' and unload the intersection. Stefan
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier monn...@iro.umontreal.ca writes: It's just an experiment (AFAIK). Okay. Check whether a required feature has been shadowed by changing `load-path' after it has been loaded and reload that feature from current load-path in this case. I don't understand why we need this. For the case where Org is installed via package.el rather than by manually downloading and following some installation instructions. In that case, isn't it enough to call (package-initialize) before any Org configuration? -- Bastien
[O] bug#10125: RFE: require and load-path-shadowing
In that case, isn't it enough to call (package-initialize) before any Org configuration? Here's the scenario: start Emacs use Org use package.el to install a newer version of Org Stefan
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier writes: In that case, isn't it enough to call (package-initialize) before any Org configuration? Here's the scenario: start Emacs use Org use package.el to install a newer version of Org Exactly. Thanks for putting it so succinctly. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf Blofeld V1.15B11: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
[O] bug#10125: RFE: require and load-path-shadowing
Eli Zaretskii writes: IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q? Yes, because the current Emacs may be a different executable than the one the shell would run in response to emacs -Q. And if we make sure the same executable is run (easy on Windows)? Are there any other differences? There may have been changes in the way Emacs starts up in the meantime, since we're considering the case where new packages get installed. Starting Emacs with -Q shouldn't be affected by this, but there's no guarantee (yes, this is a fringe case). Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier monn...@iro.umontreal.ca writes: In that case, isn't it enough to call (package-initialize) before any Org configuration? Here's the scenario: start Emacs use Org use package.el to install a newer version of Org I see, thanks. -- Bastien
[O] bug#10125: RFE: require and load-path-shadowing
From: Achim Gratz strom...@nexgo.de Date: Sat, 12 Jan 2013 18:01:36 +0100 Eli Zaretskii writes: IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q? Yes, because the current Emacs may be a different executable than the one the shell would run in response to emacs -Q. And if we make sure the same executable is run (easy on Windows)? Are there any other differences? There may have been changes in the way Emacs starts up in the meantime, since we're considering the case where new packages get installed. Starting Emacs with -Q shouldn't be affected by this, but there's no guarantee (yes, this is a fringe case). Sorry, I don't follow: new packages can only affect emacs -Q if you re-dump Emacs in between. Am I missing something?
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier wrote: IOW, do you expect the byte-compile instances to be different in any way from a fresh Emacs session invoked from the shell as emacs -Q? Yes, because the current Emacs may be a different executable than the one the shell would run in response to emacs -Q. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125#50 Turns out I was looking for invocation-directory and invocation-name. If you want to worry about wacky things like a new Emacs having been installed on top of the old in the meantime, more power to you.
[O] bug#10125: RFE: require and load-path-shadowing
Glenn Morris writes: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125#50 Turns out I was looking for invocation-directory and invocation-name. Indeed, thanks. So using your earlier patch as a template, this should be good for emacs-24: --8---cut here---start-8--- diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el index 6059f03..df194b8 100644 --- a/lisp/emacs-lisp/package.el +++ b/lisp/emacs-lisp/package.el @@ -632,11 +632,20 @@ untar into a directory named DIR; otherwise, signal an error. Generate autoloads and do byte-compilation for package named NAME. PKG-DIR is the name of the package directory. (package-generate-autoloads name pkg-dir) - (let ((load-path (cons pkg-dir load-path))) -;; We must load the autoloads file before byte compiling, in -;; case there are magic cookies to set up non-trivial paths. -(load (expand-file-name (concat name -autoloads) pkg-dir) nil t) -(byte-recompile-directory pkg-dir 0 t))) + (with-current-buffer (get-buffer-create *package-compile*) +(goto-char (point-max)) +(pop-to-buffer (current-buffer)) +(or (zerop (call-process + (concat invocation-directory invocation-name) + nil t t --batch -Q --eval + (format +(let ((pkg-dir \%s\)(name \%s\)) +(progn (setq load-path (cons pkg-dir load-path)) +(load (expand-file-name (concat name \-autoloads\) pkg-dir) nil t) +(batch-byte-recompile-directory 0))) +pkg-dir name) + pkg-dir)) + (error Compiling the package gave an error (defun package--write-file-no-coding (file-name) (let ((buffer-file-coding-system 'no-conversion)) --8---cut here---end---8--- I've confirmed that Emacs 24.2 fails to install current Org from ELPA without that patch when trying to install after having opened an Org file. It installs Org correctly in that same situation with the patch that does the package compilation in an external Emacs process. Regards Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier writes: I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. I've been thinking about something like this for a while… if it worked at least as well as starting a new Emacs instance on all platforms, I'd favor this approach. - outdated .elc file taking precedence over the new .el file can do the same. Yes, but you get a warning and can already arrange for this (by binding the appropriate variables) to be no problem in practise. See the way org-reload works in current master (of the Org repo). - bytecompiling a file affects the running session by side-effects such as requiring packages. If that problem was finally solved that would be very welcome. I suggested a quickdirty solution: E.g. we could add to bytecomp.el the ability to force `require' to reload a package if it's not already loaded from the file that locate-library returns. I still think it's not a bad option. Would an advice work in this situation (given that require is a primitive)? If yes, I'd like to give it a try over the weekend. If not, I don't really see why require, more specifically the part that checks features needs to be a primitive, so maybe it could be moved partly to elisp. Of course, we'd still get trouble when the loading is not performed via `require' but via autoload (maybe we could try and attack this problem by allowing `autoload' to override an already existing definition, but that could be delicate). That I'd like to split off from the discussion about require. I don't see why that would introduce a difficulty. As long as the package is properly namespaced, why not allow for removing all definitions pertaining to that entire namespace (features, autoloads, definitions, …)? Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf Blofeld V1.15B11: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
[O] bug#10125: RFE: require and load-path-shadowing
I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. I've been thinking about something like this for a while… if it worked at least as well as starting a new Emacs instance on all platforms, I'd favor this approach. IIUC fork is not really an option for w32. I suggested a quickdirty solution: E.g. we could add to bytecomp.el the ability to force `require' to reload a package if it's not already loaded from the file that locate-library returns. I still think it's not a bad option. Would an advice work in this situation (given that require is a primitive)? Yes, this subroutine is never directly called from C, so placing an advice should work just fine. If yes, I'd like to give it a try over the weekend. If not, I don't really see why require, more specifically the part that checks features needs to be a primitive, so maybe it could be moved partly to elisp. AFAICT the only part of `require' which can't be written in Elisp right now is the part that handles Vautoload_queue because that variable is not exposed to Elisp (IIRC this variable is used to undo the effects of a partially loaded file when the load bumps into an error midway through; FWIW I'm not convinced this feature works reliably nowadays). I don't see why that would introduce a difficulty. As long as the package is properly namespaced, why not allow for removing all definitions pertaining to that entire namespace (features, autoloads, definitions, …)? We could try that, as well, but it would only work for those packages that are properly namespaced (and there's no way to detect that AFAIK). Along the same lines, we could try to use unload-feature. Stefan
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier writes: I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. I've been thinking about something like this for a while… if it worked at least as well as starting a new Emacs instance on all platforms, I'd favor this approach. IIUC fork is not really an option for w32. For the intended application spawn should work as well? Yes, this subroutine is never directly called from C, so placing an advice should work just fine. OK, I'll give it a try. As long as the package is properly namespaced, why not allow for removing all definitions pertaining to that entire namespace (features, autoloads, definitions, …)? We could try that, as well, but it would only work for those packages that are properly namespaced (and there's no way to detect that AFAIK). True, but a package might indicate if it is one of those and get the appropriate treatment from package manager. Along the same lines, we could try to use unload-feature. I thought this was potentially dangerous, but reading the docstring again maybe not. Let me try that as well. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
[O] bug#10125: RFE: require and load-path-shadowing
I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. I've been thinking about something like this for a while… if it worked at least as well as starting a new Emacs instance on all platforms, I'd favor this approach. IIUC fork is not really an option for w32. For the intended application spawn should work as well? Could be: depends on the precise semantics of spawn, which I don't know. Along the same lines, we could try to use unload-feature. I thought this was potentially dangerous, but reading the docstring again maybe not. Let me try that as well. It's fundamentally tricky just in the same way as your proposed namespace cleanup: if you undefine a function that's still registered on some hook, process filter, ... you may get subsequent errors, some of which may render Emacs completely unusable. So it's risky to call unload-feature on a random package, but it's not too hard for a package to make sure it survives unload-feature. Tho currently, there are some significant shortcomings (IIRC there are cases where the package's autoloads aren't re-instated, for example). Stefan
[O] bug#10125: RFE: require and load-path-shadowing
There is currently a problem with package manager when a package is installed from a package archive,that package is already installed either in Emacs core or site-lisp, and when (parts of) said package have already been loaded when ELPA tries to install: the byte-compiler will use the already loaded definitions rather than the new ones from the package to install. I guess this would be http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10125 Indeed, and as I mentioned back then I think it's a general enough problem that we should try and think up a good solution. Currently, we have two proposals: 1- run a separate Emacs instance: this gives you a clean slate, and lets you compile in parallel, but runs into the difficulty of figuring out exactly which clean slate to use. I guess we could fork Emacs early on and keep this second process around as a process from which to generate new clean slates. 2- improve bytecomp.el to try and better isolate the compiled file from the previously loaded packages. I don't see a clear winner, but since I'm biased in favor of the second (not sure why, to tell you the truth), I'll add a few points related to it. The current behavior of bytecomp.el leads to various related problems: - outdated but already loaded packages can lead to mis-compilation. for the bootstrap we try to workaround this with byte-compile-refresh-preloaded, tho it only takes care of some particular cases. - outdated .elc file taking precedence over the new .el file can do the same. - bytecompiling a file affects the running session by side-effects such as requiring packages. - if a package calls `byte-compile' during its own compilation, this sub-compilation will tend to complain about undeclared variables because it doesn't know about the vars that have been defvar'd in the outer compilation. That's one of the main reasons for cc-bytecomp's hideous gymnastics. Maybe we should (similarly to the fork idea above) keep a clean obarray, and run byte-compilations in a fresh copy of this clean obarray. I suggested a quickdirty solution: E.g. we could add to bytecomp.el the ability to force `require' to reload a package if it's not already loaded from the file that locate-library returns. I still think it's not a bad option. Of course, we'd still get trouble when the loading is not performed via `require' but via autoload (maybe we could try and attack this problem by allowing `autoload' to override an already existing definition, but that could be delicate). That will probably work fine most of the time, but what if a package is restructed so that the feature names are different? Or a feature is removed? I don't see why that would introduce a difficulty. Stefan
[O] bug#10125: RFE: require and load-path-shadowing
Stefan Monnier monn...@iro.umontreal.ca writes: - outdated .elc file taking precedence over the new .el file can do the same. I find this warning quite useful. Consider this, all happening because of what is in .emacs. 1. Something in .emacs causes org-x to be loaded from Vanilla Emacs. 2. Load path changed to point to a git checkout. 3. User does something which loads org-y. org-y comes from (2). The chain of events ends up with requiring an org-x. org-x WASN'T RE-LOADED because the symbol is provided. Ofcourse, it is provided from the wrong place. In case of (3), a warning that says I refuse to have mutiple personality disorder. A warning during Emacs initing could be missed. So if there is a command line option to trigger an abrupt-end, we could advise the user to enable it and say You have got a problem in your init file. Not the Org-mode that is distributed. This will serve to remove endless amount of confusion. What I am saying, even if we couldn't solve the problem cleanly, a simple warning that says Something fishy will greatly reduce the user complaints in the Org list. I am not sure how much of what I typed makes sense. But I type nevertheless, people read it or ignore it. --
