Re: bug#59882: Multiple versions of Org in load-path problem

[Tom Gillespie]

> The state of our unloading support is not good enough to use it blindly

I have managed to use unload-feature to reload a new version of org
after the builtin version has been previously loaded, but the dance you
have to do to avoid nasty errors (and segfaults if using native comp!)
is about 100 lines of arcane footwork, and if any one step is off you
will encounter one of the many hidden landmines.

unload-feature cannot be used blindly, and there are a very large
number of side effects along the path that are not controlled which
makes any process that tries to use unload-feature brittle and hard
to debug.

https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1010-L1064
https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1073-L1177

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> I have noticed that neither `package--reload-previously-loaded` nor code
> loading already compiled package unload the loaddefs file. E.g. I added

The state of our unloading support is not good enough to use it blindly,
indeed: it's too easy/common to have packages where `unload` leaves you
with an Emacs that has serious problems (including being unusable).

So, instead we limit ourselves to force-reloading files which tends to
be much more harmless (tho in theory of course it's just as bad).
It also tends to be sufficient.


Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 18/12/2022 22:40, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:

- "emacs -l org" to simulate a regular user session with some opened files.
- Update to Org-9.6 from ELPA.
- Close emacs and start it again "emacs -l org"

Result:
byte-code: Invalid function: org-assert-version


Hmm... there's code in `package.el` intended to avoid that problem, but
clearly it's failing here.  Could Someone™ investigate why that code
doesn't do its job.

The code in question is in `package--reload-previously-loaded`


Since I managed to reproduce the issue with emacs-28.2 built from source 
tree, I tried current master (30.0.50) as well. I used the following 
trick to shadow org-assert-version:


sed -i -e 's/org-assert-version/org-assert-version-old/g' lisp/org/*.el

I am unable to reproduce the "Invalid function: org-assert-version" 
issue. Org compilation finishes without warnings.


I have noticed that neither `package--reload-previously-loaded` nor code 
loading already compiled package unload the loaddefs file. E.g. I added 
org-stale-function marked as autoload. It is possible to get help for 
this function, but attempts to call this function causes "Autoloading 
file" error. Ideally emacs should completely forget about such functions 
when newer elpa package is installed.

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 03/02/2023 18:02, Ihor Radchenko wrote:


The problem seems to be real and appears to be some combination of
Debian/Ubuntu + Emacs.


It is not a Debian packaging issue. I have reproduced the "invalid 
function `org-assert-version'" bug using emacs built from emacs-28.2 git 
tag. It is important to execute "make install" and to launch emacs from 
from that directory.


I have no idea why behavior is different when emacs is launched from the 
build tree.


Environment: Debian-12 bookworm (frozen testing).

There is no problem when emacs is built by just "make" from debian 
sources with all patches applied. I admit that this experiment was not 
really pure since I did not use configure options from the 
debian/control file.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi  writes:

> These are init files specific to other packages installed via OS package
> manager. But those are not loaded, it seems in the above steps. I tried to
> run (featurep 'clang-format) and it returned nil, while this returns t
> in my normal Emacs instance.
> ...
> None of these load org-mode.

They just define auto-loading, but I see not how they would affect Org
loading.

Very confusing...

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Wed, 22 Feb 2023 at 16:17, Ihor Radchenko  wrote:
>
> Bhavin Gandhi  writes:
>
> > I'm able to reproduce this on Fedora 37, Emacs 28.2.
> >
> > In Fedora we have:
> > - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> > from /usr/share/emacs/site-lisp/site-start.d).
> > - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).
>
> May I know what is the contents of
> /usr/share/emacs/site-lisp/site-start.d ?

These are init files specific to other packages installed via OS package
manager. But those are not loaded, it seems in the above steps. I tried to
run (featurep 'clang-format) and it returned nil, while this returns t
in my normal Emacs instance.

$ ls -1 /usr/share/emacs/site-lisp/site-start.d/
autoconf-init.el
  https://src.fedoraproject.org/rpms/autoconf/blob/rawhide/f/autoconf-init.el
clang-format.el
clang-include-fixer.el
clang-rename.el
cmake-init.el
  https://src.fedoraproject.org/rpms/cmake/blob/rawhide/f/cmake-init.el
desktop-entry-mode-init.el
  
https://src.fedoraproject.org/rpms/desktop-file-utils/blob/rawhide/f/desktop-entry-mode-init.el
gtypist-init.el
  https://src.fedoraproject.org/rpms/gtypist/blob/rawhide/f/gtypist.spec#_43-46
mercurial-site-start.el
  
https://src.fedoraproject.org/rpms/mercurial/blob/rawhide/f/mercurial-site-start.el

None of these load org-mode.

--
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi  writes:

> I'm able to reproduce this on Fedora 37, Emacs 28.2.
>
> In Fedora we have:
> - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> from /usr/share/emacs/site-lisp/site-start.d).
> - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

May I know what is the contents of
/usr/share/emacs/site-lisp/site-start.d ?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Fri, 3 Feb 2023 at 16:35, Ihor Radchenko  wrote:
>
> Max Nikulin  writes:
>
> > On 27/12/2022 16:47, Ihor Radchenko wrote:
> >> Can you then try to test using Emacs 28?
> >> The main question if whether this has been fixed in newer Emacs releases
> >> or it is also something to do with OS environment.
> >
> > I see quite the same issue with Emacs-28.2 in Debian testing. Compile
> > buffer displays a bit more warnings and usual `org-assert-version'
> > warnings and errors are present as well. It might have another level of
> > complexity due to .eln files. I am unsure what happens with calls to
> > undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.

I'm able to reproduce this on Fedora 37, Emacs 28.2.

In Fedora we have:
- /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
from /usr/share/emacs/site-lisp/site-start.d).
- /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

There are other patches but most of them are backports and seem to be
unrelated to package.el:
https://src.fedoraproject.org/rpms/emacs/tree/rawhide

Steps using with-emacs.sh (https://github.com/alphapapa/with-emacs.sh):

1. mkdir /tmp/emacs-59882
2. with-emacs.sh -d /tmp/emacs-59882 -- -l org
3. M-x package-list-packages
4. Install Org 9.6.1 from ELPA
5. When the package is compiled you get this error at the end (it is
there multiple times)
   ox.el:82:1: Error: Symbol’s function definition is void: org-assert-version
6. Close Emacs and start it again with
   with-emacs.sh -d /tmp/emacs-59882 -- -l org
7. I get byte-code: Invalid function: org-assert-version

-- 
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
> From: Richard Stallman 
> Date: Tue, 21 Feb 2023 00:22:04 -0500
> 
>   > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
>   > >> does use a modified Emacs startup which is not part of the standard
>   > >> Emacs distribution. They do this to enable the ability to have multiple
>   > >> versions of Emacs installed at the same time.
> 
> Would we like to add a feature like that to Emacs?
> I don't know how much work it would be, but I think
> it would be a desirable change if it is easy.

I'm not sure I understand: feature like what?  If you are talking
about having several Emacs versions installed on the same system, then
I think we already support that: I use that all the time.  That's why
we install various files in version-specific directories.

If you mean something else, please elaborate.  (The bug discussion is
very long, and the above quote is from a tangent, so I'm not sure it's
worth my while reading all the discussion from the beginning trying to
understand what is being alluded to here.)

Re: bug#59882: Multiple versions of Org in load-path problem

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
  > >> does use a modified Emacs startup which is not part of the standard
  > >> Emacs distribution. They do this to enable the ability to have multiple
  > >> versions of Emacs installed at the same time.

Would we like to add a feature like that to Emacs?
I don't know how much work it would be, but I think
it would be a desirable change if it is easy.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 15/02/2023 05:53, Gregor Zattler wrote:

* Tim Cross [2023-02-04; 07:01 +11]:


I don't run Debian or Ubuntu anymore. However, I do recall that debian
does use a modified Emacs startup which is not part of the standard
Emacs distribution. They do this to enable the ability to have multiple
versions of Emacs installed at the same time.


that would be /usr/share/emacs/site-lisp/debian-startup.el


This file should be ignored when Emacs is started with --quick (-Q) 
option, see

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882#86

I have not checked patches that may be applied to Emacs core files.

Re: bug#59882: Multiple versions of Org in load-path problem

[Gregor Zattler]

Hi Tim, org-mode and emacs developers,
* Tim Cross  [2023-02-04; 07:01 +11]:
> Ihor Radchenko  writes:
>> Max Nikulin  writes:
>>> On 27/12/2022 16:47, Ihor Radchenko wrote:
 Can you then try to test using Emacs 28?
 The main question if whether this has been fixed in newer Emacs releases
 or it is also something to do with OS environment.
>>> I see quite the same issue with Emacs-28.2 in Debian testing. Compile
>>> buffer displays a bit more warnings and usual `org-assert-version'
>>> warnings and errors are present as well. It might have another level of
>>> complexity due to .eln files. I am unsure what happens with calls to
>>> undefined macros.
>>
>> I asked people around to test using Debian, and we do have a
>> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
>> error.
>>
>> I also installed Debian 11.6.0 on virtual machine, and I was also able to
>> trigger the error, following the provided steps, using the Emacs 27
>> installed via apt-get.
>>
>> The problem seems to be real and appears to be some combination of
>> Debian/Ubuntu + Emacs.
>>
>> Considering the popularity of Debian-based distros, may someone take a
>> closer look on what may be going on? Since the latest Emacs release also
>> suffers from the problem, I am afraid that the issue will be present in
>> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).
>
> I don't run Debian or Ubuntu anymore. However, I do recall that debian
> does use a modified Emacs startup which is not part of the standard
> Emacs distribution. They do this to enable the ability to have multiple
> versions of Emacs installed at the same time.

that would be /usr/share/emacs/site-lisp/debian-startup.el


Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: bug#59882: Multiple versions of Org in load-path problem

[Tim Cross]

Ihor Radchenko  writes:

> Max Nikulin  writes:
>
>> On 27/12/2022 16:47, Ihor Radchenko wrote:
>>> Can you then try to test using Emacs 28?
>>> The main question if whether this has been fixed in newer Emacs releases
>>> or it is also something to do with OS environment.
>>
>> I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
>> buffer displays a bit more warnings and usual `org-assert-version' 
>> warnings and errors are present as well. It might have another level of 
>> complexity due to .eln files. I am unsure what happens with calls to 
>> undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.
>
> Considering the popularity of Debian-based distros, may someone take a
> closer look on what may be going on? Since the latest Emacs release also
> suffers from the problem, I am afraid that the issue will be present in
> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

I don't run Debian or Ubuntu anymore. However, I do recall that debian
does use a modified Emacs startup which is not part of the standard
Emacs distribution. They do this to enable the ability to have multiple
versions of Emacs installed at the same time.

If you are unable to reproduce the issue with an Emacs built from
sources and only from the versions of Emacs installed bia apt, then I
would be looking at the sources for the apt package and the modified
*.el files it uses as it could be something in there which is triggering
the issue (seem unlikely to be coincidence that a customization to allow
running of multiple versions is also triggering a version conflict in
org?)

. 

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: Ihor Radchenko 
> Cc: Stefan Monnier , emacs-orgmode@gnu.org, Eli
>  Zaretskii , 59...@debbugs.gnu.org
> Date: Fri, 03 Feb 2023 11:02:21 +
> 
> Considering the popularity of Debian-based distros, may someone take a
> closer look on what may be going on? Since the latest Emacs release also
> suffers from the problem, I am afraid that the issue will be present in
> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

I don't know enough about package.el's peculiarities, sorry.  And my
plate is too full ATM.  So I'm afraid someone else will have to
investigate this.  (I think Debian downstream maintainers are the
natural candidates, so I've taken the liberty of CC'ing them.)

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin  writes:

> On 27/12/2022 16:47, Ihor Radchenko wrote:
>> Can you then try to test using Emacs 28?
>> The main question if whether this has been fixed in newer Emacs releases
>> or it is also something to do with OS environment.
>
> I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
> buffer displays a bit more warnings and usual `org-assert-version' 
> warnings and errors are present as well. It might have another level of 
> complexity due to .eln files. I am unsure what happens with calls to 
> undefined macros.

I asked people around to test using Debian, and we do have a
confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
error.

I also installed Debian 11.6.0 on virtual machine, and I was also able to
trigger the error, following the provided steps, using the Emacs 27
installed via apt-get.

The problem seems to be real and appears to be some combination of
Debian/Ubuntu + Emacs.

Considering the popularity of Debian-based distros, may someone take a
closer look on what may be going on? Since the latest Emacs release also
suffers from the problem, I am afraid that the issue will be present in
the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 27/12/2022 16:47, Ihor Radchenko wrote:

Can you then try to test using Emacs 28?
The main question if whether this has been fixed in newer Emacs releases
or it is also something to do with OS environment.


I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
buffer displays a bit more warnings and usual `org-assert-version' 
warnings and errors are present as well. It might have another level of 
complexity due to .eln files. I am unsure what happens with calls to 
undefined macros.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin  writes:

> I found just single commit to lisp/emacs-lisp/package.el between 
> emacs-27.1 and emacs-27.2:
>
> 1fc9de4b81c 2020-10-30 19:20:24 -0700 Glenn Morris: Improve 
> reproducibility of generated -pkg.el files
>
> So I am still surprised that you can not reproduce the issue.

Can you then try to test using Emacs 28?
The main question if whether this has been fixed in newer Emacs releases
or it is also something to do with OS environment.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 26/12/2022 18:01, Ihor Radchenko wrote:

Ihor Radchenko writes:

1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org

I was finally able to reproduce using emacs-26.

Maybe your Emacs 27 is older than mine? I have GNU Emacs 27.2


Debian stable and Ubuntu LTS have Emacs-27.1 only, so I am testing 
1:27.1+1-3ubuntu5


Following the with-emacs.sh script I have tried

mkdir /tmp/emcs
emacs --quick --eval '(setq user-emacs-directory "/tmp/emcs/")' --eval 
'(setq user-init-file "/tmp/emcs/init.el")' -l package --eval '(setq 
package-user-dir "/tmp/emcs/elpa/")' --eval '(package-initialize)' 
--eval '(package-refresh-contents)' -l org


I had no problem to select Org-9.6 from the buffer generated by 
`list-packages'. Compilation errors and the error during next startup 
are the same as I reported before.


I found just single commit to lisp/emacs-lisp/package.el between 
emacs-27.1 and emacs-27.2:


1fc9de4b81c 2020-10-30 19:20:24 -0700 Glenn Morris: Improve 
reproducibility of generated -pkg.el files


So I am still surprised that you can not reproduce the issue.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Ihor Radchenko  writes:

> 1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org

I was finally able to reproduce using emacs-26.

Maybe your Emacs 27 is older than mine? I have GNU Emacs 27.2

I also tried with Emacs 28. Not able to reproduce.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin  writes:

>> I tried to follow these steps, but unfortunately I am unable to
>> reproduce. Everything works fine using Emacs 27 on my side. Strange.
>
> I tried it once more in a minimal LXC container with Ubuntu-22.04 LTS 
> jammy. This time I even removed the elpa-org-9.5.2 system package, so 
> built-in version of org is really used. So, emacs-27
>
> - emacs -l org
> - M-x list-packages RET
> - / n org RET
> - move cursor to org, install it

On my side, it will give nothing useful: the package list is not updated
and I get:

Package org is built-in.

 Status: Built-In.
Version: 9.4.4
Summary: Export Framework for Org Mode

Detailed steps I tried (with package refresh) using with-emacs.sh [1]:

1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org
2. M-x package-list-packages RET
3. C-s outline-based RET
4. RET
5. C-x o C-s instal RET
6. RET y
7. The warnings buffer appears, but only contains
   Warning (defvaralias): Overwriting value of ‘org-tab-first-hook’ by aliasing 
to ‘org-cycle-tab-first-hook’

[1] https://github.com/alphapapa/with-emacs.sh

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 18/12/2022 22:40, Stefan Monnier wrote:

Could Someone™ investigate why that code
doesn't do its job.

The code in question is in `package--reload-previously-loaded` and
my understanding is that it should cause `org-macs` to be reloaded


I do not see obvious flaws in `package--reload-previously-loaded' code. 
My guess is that we get reports from users who have installed released 
versions of emacs.


Notice that my complain concerning failure `byte-recompile-directory', 
hidden from its caller, is still valid.


On 25/12/2022 16:48, Ihor Radchenko wrote:

Max Nikulin writes:


I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <=
9.5.


Stefan convinced me that I was too optimistic concerning Emacs-29.


- "emacs -l org" to simulate a regular user session with some opened files.
- Update to Org-9.6 from ELPA.
- Close emacs and start it again "emacs -l org"

Result:
byte-code: Invalid function: org-assert-version


I tried to follow these steps, but unfortunately I am unable to
reproduce. Everything works fine using Emacs 27 on my side. Strange.


I tried it once more in a minimal LXC container with Ubuntu-22.04 LTS 
jammy. This time I even removed the elpa-org-9.5.2 system package, so 
built-in version of org is really used. So, emacs-27


- emacs -l org
- M-x list-packages RET
- / n org RET
- move cursor to org, install it
- i x y
- compilation buffers pops up with warning and errors.

Due to "-l org" some libraries like org, org-macs, ob are loaded. Files 
that `require' already loaded libraries generate warnings


Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-R.el at Sun Dec 18 
04:30:49 2022


In end of data:
ob-R.el:562:1:Warning: the function ‘org-assert-version’ is not known to be
defined.

because `org-assert-version' is defined in org-macs.el, but the old 
version without it is already loaded. Call to `org-assert-version' is 
just added to the .elc file.


E.g. org-keys.el produce the same warning. Another case is

Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-matlab.el at Sun Dec 
18 04:30:50 2022
ob-matlab.el:43:1:Error: Symbol’s function definition is void: 
org-assert-version


This file contains "(require 'ob-octave)". Attempt of load new version 
of ob-octave fails because during execution of it contents causes an 
error due to the `org-assert-version' call.


In particular, compilation of org.el fails because the new version has 
new "(require 'org-fold)" and the latter also tries to call undefined 
`org-assert-version'.


So at this point we have partially compiled broken install. Some files 
are not compiled, others have call to `org-assert-version' instead of 
its expansion.


Let's try to start emacs again and load org. Since org.el is not 
compiled, "(org-assert-version)" inside it is unable to catch any 
problem due to mixed-version compilation. Now `org-release' returns new 
version. The failure happens during "(require 'org-keys)" that was 
compiled with the warning.


Debugger entered--Lisp error: (invalid-function org-assert-version)
  org-assert-version()
  byte-code("\300\301!\210\302 \210\300\303!\210\300\304!\207" [require 
org-macs org-assert-version cl-lib oc] 2)

  require(org-keys)

So we have an obscure load time error.  I would say that compile *error* 
messages are obscure as well because it is unclear that errors happen in 
`require'. E.g. when compiling org-matlab.el the reported line number is 
for "(require 'ob-maxima)" while compiling of ob.maxima.el finished with 
a warning, not an error and it is really confusing. "ob-octave:34:" in 
addition to "ob-matlab:43:" during compiling of ob-matlab.el would be 
more informative.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin  writes:

> I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <= 
> 9.5.
>
> - "emacs -l org" to simulate a regular user session with some opened files.
> - Update to Org-9.6 from ELPA.
> - Close emacs and start it again "emacs -l org"
>
> Result:
> byte-code: Invalid function: org-assert-version

I tried to follow these steps, but unfortunately I am unable to
reproduce. Everything works fine using Emacs 27 on my side. Strange.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> - "emacs -l org" to simulate a regular user session with some opened files.
> - Update to Org-9.6 from ELPA.
> - Close emacs and start it again "emacs -l org"
>
> Result:
> byte-code: Invalid function: org-assert-version

Hmm... there's code in `package.el` intended to avoid that problem, but
clearly it's failing here.  Could Someone™ investigate why that code
doesn't do its job.

The code in question is in `package--reload-previously-loaded` and
my understanding is that it should cause `org-macs` to be reloaded from
the Org-9.6 version of the code before the new code gets compiled, and
that new version does define the `org-assert-version` version, so those
macro calls should then be compiled correctly.

[ The approach used in `package--reload-previously-loaded` has its
  weaknesses, but AFAIK it *should* be able to avoid the above error.  ]


Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 16/12/2022 21:47, Ihor Radchenko wrote:

Can it happen that already loaded version affects compilation of a new
version by package-install.


Yes. It is what appears to be happening:

https://irreal.org/blog/?p=10996
https://irreal.org/blog/?p=10999


I would not call it steps to reproduce.

From my point of view, there is a number of issues with updating of 
packages from emacs session.


1. If compilation of a package failed then it is not withdrawn from the 
load path.


2. There is no reliable way to determine if package compilation failed. 
(byte-recompile-directory DIR 0 t) returns human readable string, not 
signalling an error or returning a value convenient to check in the 
code. It is the cause of the previous issue.


3. Emacs package system is not ready for complex multifile packages 
containing macro definitions. It does not push user to update packages 
just after emacs restart in a clean state. As a result, files may be 
compiled with old or missed macro definitions.


4. Because of the item 2 a package has no chance to implement reliable 
internal sanity check that prevents updating to a broken state.


I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <= 
9.5.


- "emacs -l org" to simulate a regular user session with some opened files.
- Update to Org-9.6 from ELPA.
- Close emacs and start it again "emacs -l org"

Result:
byte-code: Invalid function: org-assert-version

Notice attempt to load Org-9.6 despite warnings and even error during 
update:



Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-C.el at Sun Dec 18 04:30:49 
2022
Entering directory ‘/home/emcs/.emacs.d/elpa/org-9.6/’

In end of data:
ob-C.el:509:1:Warning: the function ‘org-assert-version’ is not known to be
defined. 

...

Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-matlab.el at Sun Dec 18 
04:30:50 2022
ob-matlab.el:43:1:Error: Symbol’s function definition is void: 
org-assert-version


So emacs can not handle update of a package with changed macro 
definitions when they reside in another package file.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin  writes:

> I have impression that there is no recipe how to reproduce this bug yet, 
> just reports that people have faced it. So aspects of autoload may be 
> considered as a part of brainstorm.
>
> Other ideas when built-in Org version may be loaded too early:
> - restoring previous session.
> - user configured Org agenda as startup buffer.
>
> Can it happen that already loaded version affects compilation of a new 
> version by package-install.

Yes. It is what appears to be happening:

https://irreal.org/blog/?p=10996
https://irreal.org/blog/?p=10999

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 15/12/2022 11:17, Stefan Monnier wrote:

Max Nikulin [2022-12-14 23:02:53] wrote:

Completion failure for "org-to" happened because result of
`register-definition-prefixes' calls is not considered as options. I am
aware that it may give false positives, but I still believe they should
be added.


I'm sorry, but I don't understand what you're saying.


My bad. "org-go" is completed to "org--goto" while "org-goto-" is not 
offered. The latter might be a variant because "org-goto" is registered 
as an autoload prefix. Ihor kindly created https://debbugs.gnu.org/60085 
"`help-enable-autoload' is not fully obeyed", so it is better to discuss 
it there.



Though I believe that org mixed version issue happens due to transitional
dependency of some third party package on org or something like (require
'org-protocol) that can not be loaded on demand.


Hmm... indeed the "Subject:" talks about mixed versions, yet I was
discussing something not directly related.  Looks like I got off-topic
somewhere along the way without noticing, sorry.


I have impression that there is no recipe how to reproduce this bug yet, 
just reports that people have faced it. So aspects of autoload may be 
considered as a part of brainstorm.


Other ideas when built-in Org version may be loaded too early:
- restoring previous session.
- user configured Org agenda as startup buffer.

Can it happen that already loaded version affects compilation of a new 
version by package-install.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Stefan Monnier  writes:

>> Though I believe that org mixed version issue happens due to transitional
>> dependency of some third party package on org or something like (require
>> 'org-protocol) that can not be loaded on demand.
>
> Hmm... indeed the "Subject:" talks about mixed versions, yet I was
> discussing something not directly related.  Looks like I got off-topic
> somewhere along the way without noticing, sorry.

And the off-topic is what has been CCed to Org mailing list :)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

Max Nikulin [2022-12-14 23:02:53] wrote:
> On 14/12/2022 21:35, Stefan Monnier wrote:
>>>  Like I said in another message that I sent just before receiving yours
>>>  my conclusion came from the fact that hitting 'C-h v' with the cursor
>>>  on 'org-goto-interface' provided nothing.  It was the first time this
>>>  ever happened to me.  I did try to explicitly enter the variable's
>>>  name by entering 'org-goto', which (like 'org-go') is not
>>>  enough: it is completed to 'org--goto'.
>> AFAICT this is a missing feature: we obey `help-enable-autoload` in
>> `describe-function` but we fail to do the same autoloading dance in
>> `describe-variable`.
>
> I think, the difference is interactive vs. non-interactive calls rather than
> `describe-function' vs. `describe-variable'. What misses this feature is
> `customize-variable'.
>
> Completion failure for "org-to" happened because result of
> `register-definition-prefixes' calls is not considered as options. I am
> aware that it may give false positives, but I still believe they should
> be added.

I'm sorry, but I don't understand what you're saying.

> Though I believe that org mixed version issue happens due to transitional
> dependency of some third party package on org or something like (require
> 'org-protocol) that can not be loaded on demand.

Hmm... indeed the "Subject:" talks about mixed versions, yet I was
discussing something not directly related.  Looks like I got off-topic
somewhere along the way without noticing, sorry.


Stefan

RE: bug#59882: Multiple versions of Org in load-path problem

[Cook, Malcolm]

>On a related note, is it normal that package like org-mouse.el gets
>loaded upon calling C-h f org-mouse-  completion prompt?
>org-mouse.el does not contain a single autoload cookie.

There need not be which is apparently by design:

The autoload happens as a consequence of emacs' documented feature to [Autoload 
by 
Prefix](https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html)
 

Given that:

(equal (gethash "org-mouse-"  definition-prefixes) '("org-mouse"))

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 14/12/2022 21:35, Stefan Monnier wrote:



 Like I said in another message that I sent just before receiving yours
 my conclusion came from the fact that hitting 'C-h v' with the cursor
 on 'org-goto-interface' provided nothing.  It was the first time this
 ever happened to me.  I did try to explicitly enter the variable's
 name by entering 'org-goto', which (like 'org-go') is not
 enough: it is completed to 'org--goto'.


AFAICT this is a missing feature: we obey `help-enable-autoload` in
`describe-function` but we fail to do the same autoloading dance in
`describe-variable`.


I think, the difference is interactive vs. non-interactive calls rather 
than `describe-function' vs. `describe-variable'. What misses this 
feature is `customize-variable'.


Completion failure for "org-to" happened because result of 
`register-definition-prefixes' calls is not considered as options. I am 
aware that it may give false positives, but I still believe they should 
be added.


Though I believe that org mixed version issue happens due to 
transitional dependency of some third party package on org or something 
like (require 'org-protocol) that can not be loaded on demand.

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> similar issues with several important Org components (ol-* and ob-*). In
> particular, ob-*.el packages affect Org simply by defining function
> symbols with specific name patters (by ob-core.el design).
> Achieving no side effects is difficult in practice.

That's true.  It can be a balancing act.

> Also, note that side effects are not uncommon in Elisp packages.
> Including, built-ins, like rect.el.
> dired+ comes to mind among popular third-party packages.

Most of them are plain bugs, tho (typically fixed by introducing an
autoloaded minor-mode, so the package is activated by enabling the
minor mode rather than by `require`ing the package).

> Like I said in another message that I sent just before receiving yours
> my conclusion came from the fact that hitting 'C-h v' with the cursor
> on 'org-goto-interface' provided nothing.  It was the first time this
> ever happened to me.  I did try to explicitly enter the variable's
> name by entering 'org-goto', which (like 'org-go') is not
> enough: it is completed to 'org--goto'.

AFAICT this is a missing feature: we obey `help-enable-autoload` in
`describe-function` but we fail to do the same autoloading dance in
`describe-variable`.


Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

>> On a related note, is it normal that package like org-mouse.el gets
>> loaded upon calling C-h f org-mouse-  completion prompt?

Yes.  See `help-enable-completion-autoload`.

>> I am asking because org-mouse.el advises some functions and its loading
>> may cause unexpected consequences. See
>> https://orgmode.org/list/87r0x6sju1@fastmail.fm
> I think org-mouse.el should be fixed not to cause such effects just by
> loading it.  It should do that only when it is actually used.

Exactly.  Merely loading a file should not affect Emacs's behavior in
any significant way.


Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii  writes:

>> I am asking because org-mouse.el advises some functions and its loading
>> may cause unexpected consequences. See
>> https://orgmode.org/list/87r0x6sju1@fastmail.fm
>
> I think org-mouse.el should be fixed not to cause such effects just by
> loading it.  It should do that only when it is actually used.

Yes, we plan to do exactly this 
https://orgmode.org/list/87zgbqqukg.fsf@localhost
However, it will be a breaking backward-incompatible change. And we have
similar issues with several important Org components (ol-* and ob-*). In
particular, ob-*.el packages affect Org simply by defining function
symbols with specific name patters (by ob-core.el design).

Achieving no side effects is difficult in practice.

Also, note that side effects are not uncommon in Elisp packages.
Including, built-ins, like rect.el.
dired+ comes to mind among popular third-party packages.

>> At the same time, we have users complaining about some defcustoms not
>> being available without loading.
>> https://orgmode.org/list/25496.19258.682537.503...@gargle.gargle.howl
>
> That's too vague: it talks about "whether a variable exists", which is
> ambiguous.  That a variable is not boundp until its definition is
> loaded is perfectly expected, and I don't think defcustom changes that
> in any way (unless you autoload the defcustom, which is considered not
> the best style around here).
>
> So I guess some clarification is in order: what exactly was the
> problem with that variable being "nonexistent"?

Agree. See the quote below:
https://list.orgmode.org/orgmode/25497.41832.663858.836...@gargle.gargle.howl/

Like I said in another message that I sent just before receiving yours
my conclusion came from the fact that hitting 'C-h v' with the cursor
on 'org-goto-interface' provided nothing.  It was the first time this
ever happened to me.  I did try to explicitly enter the variable's
name by entering 'org-goto', which (like 'org-go') is not
enough: it is completed to 'org--goto'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: Ihor Radchenko 
> Cc: David Masterson , emacs-orgmode@gnu.org,
>  mic...@schi.nz, 59...@debbugs.gnu.org
> Date: Wed, 14 Dec 2022 09:51:01 +
> 
> On a related note, is it normal that package like org-mouse.el gets
> loaded upon calling C-h f org-mouse-  completion prompt?
> org-mouse.el does not contain a single autoload cookie.

I think it's expected, because the doc strings need to be loaded.
Stefan, am I right?

> I am asking because org-mouse.el advises some functions and its loading
> may cause unexpected consequences. See
> https://orgmode.org/list/87r0x6sju1@fastmail.fm

I think org-mouse.el should be fixed not to cause such effects just by
loading it.  It should do that only when it is actually used.

> At the same time, we have users complaining about some defcustoms not
> being available without loading.
> https://orgmode.org/list/25496.19258.682537.503...@gargle.gargle.howl

That's too vague: it talks about "whether a variable exists", which is
ambiguous.  That a variable is not boundp until its definition is
loaded is perfectly expected, and I don't think defcustom changes that
in any way (unless you autoload the defcustom, which is considered not
the best style around here).

So I guess some clarification is in order: what exactly was the
problem with that variable being "nonexistent"?

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii  writes:

>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

On a related note, is it normal that package like org-mouse.el gets
loaded upon calling C-h f org-mouse-  completion prompt?
org-mouse.el does not contain a single autoload cookie.

I am asking because org-mouse.el advises some functions and its loading
may cause unexpected consequences. See
https://orgmode.org/list/87r0x6sju1@fastmail.fm

At the same time, we have users complaining about some defcustoms not
being available without loading.
https://orgmode.org/list/25496.19258.682537.503...@gargle.gargle.howl

I feel confused about the situation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii  writes:

>> In my testing, I found a strange case where, in *scratch*, I get:
>> 
>> (message "%s" org-version)
>> ;; Error undefined
>> ;; Do 'C-h v org-version'
>> (message "%s" org-version)
>> 9.3
>> 
>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

Please note that David's recipe is _not_ what I described in the bug
report.

I am referring to the problem people run into when using M-x
package-install.

AFAIU, package-install must not depend on the older version of a package
being loaded/not loaded.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: David Masterson 
> Cc: Eli Zaretskii ,  "mic...@schi.nz" ,
>   "emacs-orgmode@gnu.org" ,  "59...@debbugs.gnu.org"
>  <59...@debbugs.gnu.org>
> Date: Fri, 09 Dec 2022 11:51:29 -0800
> 
> > https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html
> >
> > What you observed will NOT happen if you first run:
> >
> > (clrhash  definition-prefixes)
> 
> Ah! Thank you.  In many ways, I still think of Emacs from what I learned
> about it in the early 80s (yes, pre-Elisp).  I didn't expect this and,
> so, didn't investigate the possibility that it's planned (the Info
> documentation has gotten huge).  Apologies to Eli.

No apologies needed, we all learn something new about Emacs every day.

Re: bug#59882: Multiple versions of Org in load-path problem

[David Masterson]

"Cook, Malcolm"  writes:

>> Eli Zaretskii  writes:
>> 
>> >> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
>> >> From: David Masterson 
>> >> Date: Thu, 08 Dec 2022 13:56:03 -0800
>> >>
>> >> In my testing, I found a strange case where, in *scratch*, I get:
>> >>
>> >> (message "%s" org-version)
>> >> ;; Error undefined
>> >> ;; Do 'C-h v org-version'
>> >> (message "%s" org-version)
>> >> 9.3
>> >>
>> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
>> >
>> > Why is it so surprising?  Is this the first time you see that a Help
>> > command loads something to do its job?
>> 
>> Yes, it is surprising.  How did Emacs know to load Org to find org-version? 
>
> This is provided for via emacs' autoload by prefix mechanism.
>
> See:
>
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html
>
> What you observed will NOT happen if you first run:
>
>   (clrhash  definition-prefixes)

Ah! Thank you.  In many ways, I still think of Emacs from what I learned
about it in the early 80s (yes, pre-Elisp).  I didn't expect this and,
so, didn't investigate the possibility that it's planned (the Info
documentation has gotten huge).  Apologies to Eli.

-- 
David Masterson

RE: bug#59882: Multiple versions of Org in load-path problem

[Cook, Malcolm]

> Eli Zaretskii  writes:
> 
> >> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
> >> From: David Masterson 
> >> Date: Thu, 08 Dec 2022 13:56:03 -0800
> >>
> >> In my testing, I found a strange case where, in *scratch*, I get:
> >>
> >> (message "%s" org-version)
> >> ;; Error undefined
> >> ;; Do 'C-h v org-version'
> >> (message "%s" org-version)
> >> 9.3
> >>
> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
> >
> > Why is it so surprising?  Is this the first time you see that a Help
> > command loads something to do its job?
> 
> Yes, it is surprising.  How did Emacs know to load Org to find org-version? 

This is provided for via emacs' autoload by prefix mechanism.

See:

https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html

What you observed will NOT happen if you first run:

(clrhash  definition-prefixes)

> didn't request it to by this action.  Or did it do a package-activate-all (or
> similar) to ensure that everything was loaded?
> That would be overkill.  Sometimes the best answer is 'nil'.
> 
> --
> David Masterson

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: David Masterson 
> Cc: mic...@schi.nz,  emacs-orgmode@gnu.org,  59...@debbugs.gnu.org
> Date: Thu, 08 Dec 2022 23:43:05 -0800
> 
> Eli Zaretskii  writes:
> 
> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
> >
> > Why is it so surprising?  Is this the first time you see that a Help
> > command loads something to do its job?
> 
> Yes, it is surprising.  How did Emacs know to load Org to find
> org-version?  I didn't request it to by this action.  Or did it do a
> package-activate-all (or similar) to ensure that everything was loaded?
> That would be overkill.  Sometimes the best answer is 'nil'.

I don't think Emacs does package-activate-all.  But Help commands
frequently load packages to provide the documentation of the symbols
you ask about.  If this is surprising, then I suggest to get used to
it.

Re: bug#59882: Multiple versions of Org in load-path problem

[David Masterson]

Eli Zaretskii  writes:

>> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
>> From: David Masterson 
>> Date: Thu, 08 Dec 2022 13:56:03 -0800
>> 
>> In my testing, I found a strange case where, in *scratch*, I get:
>> 
>> (message "%s" org-version)
>> ;; Error undefined
>> ;; Do 'C-h v org-version'
>> (message "%s" org-version)
>> 9.3
>> 
>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

Yes, it is surprising.  How did Emacs know to load Org to find
org-version?  I didn't request it to by this action.  Or did it do a
package-activate-all (or similar) to ensure that everything was loaded?
That would be overkill.  Sometimes the best answer is 'nil'.

-- 
David Masterson

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
> From: David Masterson 
> Date: Thu, 08 Dec 2022 13:56:03 -0800
> 
> In my testing, I found a strange case where, in *scratch*, I get:
> 
> (message "%s" org-version)
> ;; Error undefined
> ;; Do 'C-h v org-version'
> (message "%s" org-version)
> 9.3
> 
> So, 'describe-variable' on org-version causes Org to be loaded?!?

Why is it so surprising?  Is this the first time you see that a Help
command loads something to do its job?