Re: [PATCH v4] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 30/04/2023 17:39, Ihor Radchenko wrote:

Date: Mon, 3 Apr 2023 10:41:50 +0200
Subject: [PATCH v4 4/8] Use compat.el library instead of ad-hoc compatibility
  function set


Ihor, I have not noticed removal of


;; `flatten-tree' was added in Emacs 27.1.
(defalias 'org-protocol-flatten
  (if (fboundp 'flatten-tree) 'flatten-tree


from org-protocol.el

Perhaps I will post more comments, so there is no point to send updated 
patch set in response to this message.

[PATCH v4] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

> I do not like lengthy emacs commands and make functions to generate 
> them. I think, it is better to move such code to a script. A proof of 
> concept is attached, however it is rather rough draft
>
>  ./epm.el -Q --epm-dir $(emacs_pkgdir)/emacs-%e install compat
>  ./epm.el -Q -L ~/src/compat install compat

Good idea. Although, we should not overdo this package management thing.
If we really need complex functionality here, we should better just use
cask/eldev instead of re-inventing the wheel.

Are you willing to improve the draft to be ready for upstream?
I will provide some inline comments below.

> By the way, accordingly to (info "(make) Command Variables") or (info 
> "(standards) Command Variables") "Variables for Specifying Commands"
> https://www.gnu.org/prep/standards/html_node/Command-Variables.html
>
> it should be EMACSFLAGS rather than EFLAGS.

Fair point. I now rearranged the commits as you asked and incorporated
this change. See the attached.

Note that also I changed the way compat.el is loaded, making Org throw
an error for older Emacs versions. This will produce more useful error
if someone attempts to load Org as is, without installing compat.el in
older Emacs.

>> I have no problem with this approach when using system packages.
>> However, it is almost guaranteed that compat.el is absent in global
>> load-path as long as compat.el is not built-in.
>
> I see that installation attempt is not performed when packages are 
> available. However form my point of view it is normal when compilation 
> fails when dependency are not provided. It works so for decades for 
> applications that use make. To be precise, usually I expect detection of 
> missed libraries from configure scrips, but in some cases they are 
> missed. Maybe such experience was formed when access to network was limited.
>
> For me it is quite natural that make does try to pull dependencies (at 
> least by default) and it is my responsibility to ensure availability of 
> necessary libraries.

I think that we need to zoom out a bit and discuss the contexts where
Org build system is used:

1. During Org development, by developers who know what they are doing
2. By ordinary users, not necessarily familiar with GNU make and all the
   associated build process conventions.

Org developers may need to use the whole spectrum of make targets, and
will generally benefit from isolation of Org from the main Emacs
configuration folder. For example, my `package-user-dir' contains a
number of forks with additional patches applied - it is not the
environment I want to develop (and test) Org in.

Org users will likely use make autoloads, make, make docs, and make
repro. Here, it will make sense to re-use default .emacs.d and package
directory when running make, as ordinary users running make are most
likely aiming to build Org for their own usage. However, make repro and
optionally make docs should avoid re-using user packages as it may cause
inconsistent results if the `package-user-dir' is messed up.

One way to handle the above scenarios might be your idea with AUTODEP.
By default, it will be "auto":
 - make compile, docs  :: Re-use default `package-user-dir'
 - make repro:: Auto-download and ignore `package-user-dir'
 - other targets :: Prompt the user
Alternatives will be meant to be used as

AUTODEP=download/user/no make target.

triggering unconditional downloading, using `package-user-dir', and not
using any guess, correspondingly.

WDYT?

>>> I do not like that versions of dependencies are ignored. I have noticed
>>> `package-install-from-buffer'. Perhaps it can be used to generate a stub
>>> package (e.g. org-build-deps) with Package-Requires line obtained from
>>> org.el. The only purpose of this package is to pull dependencies. It is
>>> just an idea, I have not tried such approach.
>> 
>> This sounds fragile. I see no reason to go this far and using so complex
>> approach.
>
> My idea is to ensure that *required* version is installed, not some 
> stale one. I have not tried such approach though.

I think that it is stretching a bit beyond the complexity we should
allow within Org build system. In your scenario, I can simply do
make cleanpkg and re-download the latest dependencies.

Again, package management is not something we want to overdo. If
really necessary, we will use cask or other proper Elisp development
tool.

> #!/bin/sh
> ":"; # -*- mode: emacs-lisp; lexical-binding: t; -*-
> ":"; exec emacs --script "$0" "$@"

Let's not lock to bash. AFAIK, our makefiles can currently work on
Windows. Using /bin/sh will lead to regression.

So, a simple --batch script for Emacs will be better here.

> (defun epm-init ()
>   (unless (epm-nonempty-p epm-dir)
> (setq epm-dir (getenv "EPMDIR")))
>   (when (epm-nonempty-p epm-dir)
> (let* ((fmt-expanded (format-spec epm-dir `((?e . ,emacs-version
>  (dir (directory-file-name (expand-file-name fmt-expanded
>   ;; 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

Ihor,

I do not like lengthy emacs commands and make functions to generate 
them. I think, it is better to move such code to a script. A proof of 
concept is attached, however it is rather rough draft


./epm.el -Q --epm-dir $(emacs_pkgdir)/emacs-%e install compat
./epm.el -Q -L ~/src/compat install compat

or

export EPMDIR="$HOME/.cache/epm/emacs-%e"
./epm.el install compat

to just abort compilation due to absent dependency

./epm.el missing compat

On 20/04/2023 16:27, Ihor Radchenko wrote:

Max Nikulin writes:


Sure. And you will have such option (EFLAGS).


By the way, accordingly to (info "(make) Command Variables") or (info 
"(standards) Command Variables") "Variables for Specifying Commands"

https://www.gnu.org/prep/standards/html_node/Command-Variables.html

it should be EMACSFLAGS rather than EFLAGS.


In my opinion, ideally there should be 3 options for dependency management:
1. Completely disabled. If load from default paths failed than it is a
fatal error.


I have no problem with this approach when using system packages.
However, it is almost guaranteed that compat.el is absent in global
load-path as long as compat.el is not built-in.


I see that installation attempt is not performed when packages are 
available. However form my point of view it is normal when compilation 
fails when dependency are not provided. It works so for decades for 
applications that use make. To be precise, usually I expect detection of 
missed libraries from configure scrips, but in some cases they are 
missed. Maybe such experience was formed when access to network was limited.


For me it is quite natural that make does try to pull dependencies (at 
least by default) and it is my responsibility to ensure availability of 
necessary libraries.



2. Use specified directory outside of Org tree (~/.emacs.d/elpa by
default) or any other directory that you named pkgdir. Only dedicated
target may clean this directory.


This is mostly an equivalent of -L switch.


No, -L is for source directories of package (e.g. git repositories). I 
mean namely alternative `package-user-dir', but not managed by make.



I do not like the idea of
using ~/.emacs.d/elpa default. It is fragile if this default ever
changes.


I still consider it as a reasonable default for a user having just one 
emacs version who is going to build and run org. Both steps may use the 
same package directory. A developer who switches between various emacs 
versions may have set of packages for each emacs version.



3. Install packages to Org source/build directory.

You decided to make 3 the default variant. I believe, it should be
activated by a variable, e.g. AUTODEP = 1 in local.mk or from command
line "make compile AUTODEP=1


It is now activated by EPACKAGES being non-empty.


And it is non-empty by default because it defines list of build 
dependencies, not whether they should be managed by make.



I think, it is better to require an additional command

make autoloads
make fetch-dependencies
make compile


Maybe. Then, also make doc and make install?


In general "make install" may be executed by root while "make all" is a 
task for regular user. "make doc" is an optional step, so I do not see 
any problem. Ideally it might be


make fetch-dependencies # or specify package directory
# or load path in local.mk
make all

followed by optional doc or install


And make repro,


I have not justified my point of view to make repro yet.


I do not like that versions of dependencies are ignored. I have noticed
`package-install-from-buffer'. Perhaps it can be used to generate a stub
package (e.g. org-build-deps) with Package-Requires line obtained from
org.el. The only purpose of this package is to pull dependencies. It is
just an idea, I have not tried such approach.


This sounds fragile. I see no reason to go this far and using so complex
approach.


My idea is to ensure that *required* version is installed, not some 
stale one. I have not tried such approach though.



Subject: [PATCH 3/7] Use compat.el library instead of ad-hoc compatibility
  function set

* mk/default.mk (EPACKAGES): Demand compat library during compile time.


when I asked for more granular commits I expected this change in


Subject: [PATCH 2/7] org-compat: Enable compat.el


To separate adding dependency and replacing org-compat functions to compat.


For me, PATCH 3/7 grouping is more reasonable. So, I disagree.
Splitting EPACKAGES modification would create transient commit with
non-working Org.


I just do not like that a single line change in default.mk (modification 
of build process) is buried in a large patch (changes of the code). My 
idea was that



-EPACKAGES ?=
+EPACKAGES ?= compat


should be in the same commit as


-;; Package-Requires: ((emacs "26.1"))
+;; Package-Requires: ((emacs "26.1") (compat "29.1.4.1"))


Currently patch 2 requires compat, but it is provided till patch 3. 
Despite in commit 2 the package does not do anything useful, I 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

>> Sure. And you will have such option (EFLAGS).
>> However, I decided to enable auto-downloading by default to not break
>> the previous working compilation instructions.
>
> For me adding external dependencies is strong enough reason to change 
> compiling instructions. My vote is for clear separation of dependency 
> management (even if performed through make targets) and 
> compiling/testing/etc.
> ...
> In my opinion, ideally there should be 3 options for dependency management:
> 1. Completely disabled. If load from default paths failed than it is a 
> fatal error.

I have no problem with this approach when using system packages.
However, it is almost guaranteed that compat.el is absent in global
load-path as long as compat.el is not built-in.

> 2. Use specified directory outside of Org tree (~/.emacs.d/elpa by 
> default) or any other directory that you named pkgdir. Only dedicated 
> target may clean this directory.

This is mostly an equivalent of -L switch. I do not like the idea of
using ~/.emacs.d/elpa default. It is fragile if this default ever
changes.

> 3. Install packages to Org source/build directory.
>
> You decided to make 3 the default variant. I believe, it should be 
> activated by a variable, e.g. AUTODEP = 1 in local.mk or from command 
> line "make compile AUTODEP=1

It is now activated by EPACKAGES being non-empty.

> I think, it is better to require an additional command
>
> make autoloads
> make fetch-dependencies
> make compile

Maybe. Then, also make doc and make install?
And make repro, which is dislike in particular - make repro is supposed
to be easy to use for users unfamiliar with Emacs internals or typical
GNU make conventions.

>> +package-install = --eval '(unless (require '"'"'$(package) nil t) (message 
>> "%s" load-path) (package-install '"'"'$(package)))'
>
> I do not like that versions of dependencies are ignored. I have noticed 
> `package-install-from-buffer'. Perhaps it can be used to generate a stub 
> package (e.g. org-build-deps) with Package-Requires line obtained from 
> org.el. The only purpose of this package is to pull dependencies. It is 
> just an idea, I have not tried such approach.

This sounds fragile. I see no reason to go this far and using so complex
approach.

>> +EMACS_VERSION := $(shell $(EMACS) -Q --batch --eval '(message "%s" 
>> emacs-version)' 2>&1)
>
> Ideally $(BATCH) should be used, but it is defined below. (princ 
> emacs-version) is an alternative, but I have not idea which variant is 
> better.

I used $(EMACS) on purpose. $(BATCH) may contain more things, which we
do not want (on purpose) here.

>> Subject: [PATCH 3/7] Use compat.el library instead of ad-hoc compatibility
>>  function set
>> 
>> * mk/default.mk (EPACKAGES): Demand compat library during compile time.
>
> when I asked for more granular commits I expected this change in
>
>> Subject: [PATCH 2/7] org-compat: Enable compat.el
>
> To separate adding dependency and replacing org-compat functions to compat.

For me, PATCH 3/7 grouping is more reasonable. So, I disagree.
Splitting EPACKAGES modification would create transient commit with
non-working Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 13/04/2023 19:42, Ihor Radchenko wrote:

Max Nikulin writes:
My expectation 
is that make does not attempt to manage dependencies. For me it is OK to 
type an additional command to install them and to fail otherwise.


Sure. And you will have such option (EFLAGS).
However, I decided to enable auto-downloading by default to not break
the previous working compilation instructions.


For me adding external dependencies is strong enough reason to change 
compiling instructions. My vote is for clear separation of dependency 
management (even if performed through make targets) and 
compiling/testing/etc.


Last years I rarely use make directly to build software besides the 
cases when I am involved in development. I strongly prefer .deb 
packages. Earlier it was usual practice that "make" or "make all" failed 
if dependencies were not provided. Actually often it was configure that 
reported unavailable libraries. So network requests may be more 
surprising for users than failures due to missed dependencies. It is OK 
for e.g. gradle, but not for make to fetch packages. I expect that 
maintainers of packages from Linux distributions would prefer to avoid 
mixing of compiling and loading dependencies as well.


In my opinion, ideally there should be 3 options for dependency management:
1. Completely disabled. If load from default paths failed than it is a 
fatal error.
2. Use specified directory outside of Org tree (~/.emacs.d/elpa by 
default) or any other directory that you named pkgdir. Only dedicated 
target may clean this directory.

3. Install packages to Org source/build directory.

You decided to make 3 the default variant. I believe, it should be 
activated by a variable, e.g. AUTODEP = 1 in local.mk or from command 
line "make compile AUTODEP=1


I think, it is better to require an additional command

make autoloads
make fetch-dependencies
make compile


+package-install = --eval '(unless (require '"'"'$(package) nil t) (message "%s" 
load-path) (package-install '"'"'$(package)))'


I do not like that versions of dependencies are ignored. I have noticed 
`package-install-from-buffer'. Perhaps it can be used to generate a stub 
package (e.g. org-build-deps) with Package-Requires line obtained from 
org.el. The only purpose of this package is to pull dependencies. It is 
just an idea, I have not tried such approach.



$(FIND) $(pkgdir) -name \*.elc -delete


Looks fine, except that now we have part of the targets using $RM
variable and part of the targets ignoring it. I am not sure if it is a
problem. I am slightly in favour of keeping the existing approach with
$RM.


I have realized that -delete action is not a part of POSIX 
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html
Anyway you removed this line and I like it since I suspected some issues 
when "make compile" is invoked twice.



Is there a way to express (or (compat "29.1.4.1") (emacs "28.1")) to
avoid installing compat in the case of sufficiently new emacs? E.g.
dpkg/apt allows such alternatives.


No, AFAIK:


It is sour. Another deficiency of Emacs package management is that 
errors during compiling of dependencies do not cause immediate failure 
of make.



+EMACS_VERSION := $(shell $(EMACS) -Q --batch --eval '(message "%s" emacs-version)' 
2>&1)


Ideally $(BATCH) should be used, but it is defined below. (princ 
emacs-version) is an alternative, but I have not idea which variant is 
better.



-;; Package-Requires: ((emacs "26.1") (compat "29.1.4.1"))
+;; Package-Requires: ((emacs "26.1") (compat "29.1.3.2"))


thanks


Subject: [PATCH 3/7] Use compat.el library instead of ad-hoc compatibility
 function set

* mk/default.mk (EPACKAGES): Demand compat library during compile time.


when I asked for more granular commits I expected this change in


Subject: [PATCH 2/7] org-compat: Enable compat.el


To separate adding dependency and replacing org-compat functions to compat.

Earlier I used the following procedure: git rebase -i, mark some commit 
for edit, git gui, unstage lines from commit, create new commit, reorder 
and squash/fixup commits on next iteration of rebase -i. Perhaps magit 
can do the same tricks as "git gui". If all changes of some file should 
be moved to another commit it should be easier to use git checkout 
COMMIT FILE.

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

>> +;; Package-Requires: ((emacs "26.1") (compat "29.1.4.1"))
>
> Is there a way to express (or (compat "29.1.4.1") (emacs "28.1")) to 
> avoid installing compat in the case of sufficiently new emacs? E.g. 
> dpkg/apt allows such alternatives.

No, AFAIK:

D.8 Conventional Headers for Emacs Libraries

‘Package-Requires’
 If this exists, it names packages on which the current package
 depends for proper operation.  *Note Packaging Basics::.  This is
 used by the package manager both at download time (to ensure that a
 complete set of packages is downloaded) and at activation time (to
 ensure that a package is only activated if all its dependencies
 have been).

 Its format is a list of lists on a single line.  The ‘car’ of each
 sub-list is the name of a package, as a symbol.  The ‘cadr’ of each
 sub-list is the minimum acceptable version number, as a string that
 can be parsed by ‘version-to-list’.  An entry that lacks a version
 (i.e., an entry which is just a symbol, or a sub-list of one
 element) is equivalent to entry with version "0".  For instance:

  ;; Package-Requires: ((gnus "1.0") (bubbles "2.7.2") cl-lib (seq))

 The package code automatically defines a package named ‘emacs’ with
 the version number of the currently running Emacs.  This can be
 used to require a minimal version of Emacs for a package.

> Early I asked concerning compat-29.1.3. I would prefer to install 
> elpa-compat system package to avoid network activity in response to 
> make. Required version matters for those who builds packages for 
> backport repositories for various linux distributions. It allows to 
> decide if it is necessary to provide newer elpa-compat or it is enough 
> to package elpa-org.

We can put older version require now, indeed. However, later we should
not constrain ourselves about bumping compat version as necessary.

Considering Debian lifecycle, it is pretty much guaranteed that we will
require the compat version that is not installed by Debian, eventually.

In any case, see the attached additional patch.

>> compat.el is required for "compile" target. Compilation will simply fail
>> with older Emacs. And "test" triggers "compile".
>
> There are different types of build systems. Some of them allows to 
> specify which dependencies should be pulled, some do not. My expectation 
> is that make does not attempt to manage dependencies. For me it is OK to 
> type an additional command to install them and to fail otherwise.

Sure. And you will have such option (EFLAGS).
However, I decided to enable auto-downloading by default to not break
the previous working compilation instructions.

It may, however, be worth documenting EFLAGS in WORG. See the attached
patch.

> In my opinion
>
>> +@$(FIND) $(pkgdir) \(  -name \*.elc \) -exec $(RM) {} +
>
> command tells that package management capabilities in Emacs are rather 
> rudimentary (in comparison to e.g. python toolchain). That is why I am 
> in favor to more manual dependency management. Notice that I am not 
> against an option to do it from make.

May you elaborate what you mean by "more manual"? What concrete change
in the patch do you have in mind?

> Untested:
>
> $(FIND) $(pkgdir) -name \*.elc -delete
>
> "@" for silencing is intentionally dropped, parenthesis are unnecessary 
> for single condition.

Looks fine, except that now we have part of the targets using $RM
variable and part of the targets ignoring it. I am not sure if it is a
problem. I am slightly in favour of keeping the existing approach with
$RM.

>>> default.mk is included from top level Makefile only.
>> 
>> Not sure here. I just noticed that GITVERSION got re-calculated many
>> times when looking at debug output of make. (It was slowing things down
>> significantly). GITVERSION is in targets.mk though.
>
> GITVERSION is defined as ?=, so it is a variable with deferred 
> evaluation. Immediately evaluated variable may defined using :=

That's what I did in one of the patches (use :=).

>>> pkgdir = $(top_builddir)/pkg-deps
>> 
>> How to define top_builddir? If also via `pwd`, I see not much difference.
>
> I consider it as self-documenting code. Intermediate variable makes it 
> apparent for readers that the directory is relative to the top of the 
> package file tree.

Agree. Will do.
See the attached.

> Since out of source tree builds are not supported, I would consider 
> adding emacs version to path. The idea is to allow keeping installed 
> packages when switching between several emacs versions.

Done.

>From c5ba5773e69d16930fc12db6fbe0d907fae926cd Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 8 Apr 2023 19:05:37 +0200
Subject: [PATCH 1/4] mk/default.mk (pkgdir): Make it more clear that we indent
 to use top git dir

* mk/default.mk (top_builddir): New variable storing top-level directory.
(pkgdir): Use the new variable.
---
 mk/default.mk | 3 ++-
 1 file changed, 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

+;; Package-Requires: ((emacs "26.1") (compat "29.1.4.1"))


Is there a way to express (or (compat "29.1.4.1") (emacs "28.1")) to 
avoid installing compat in the case of sufficiently new emacs? E.g. 
dpkg/apt allows such alternatives.


Early I asked concerning compat-29.1.3. I would prefer to install 
elpa-compat system package to avoid network activity in response to 
make. Required version matters for those who builds packages for 
backport repositories for various linux distributions. It allows to 
decide if it is necessary to provide newer elpa-compat or it is enough 
to package elpa-org.


On 08/04/2023 18:41, Ihor Radchenko wrote:


I see. Unfortunately, using third-party non-standard packages like
makem.sh or eldev will be more troublesome. We will need to sync Org
repo with them or demand users to install them separately.


I was looking for sources of inspiration. I do not suggest to take these 
tools. Perhaps somebody may suggest a better example of build scripts 
for Emacs packages.



I do not like the idea of network queries on every make.

Any better suggestions?


Do not run install dependencies for regular targets like test or
compile. At least do not do it unless it is explicitly requested by
command line argument or a variable in local.mk


compat.el is required for "compile" target. Compilation will simply fail
with older Emacs. And "test" triggers "compile".


There are different types of build systems. Some of them allows to 
specify which dependencies should be pulled, some do not. My expectation 
is that make does not attempt to manage dependencies. For me it is OK to 
type an additional command to install them and to fail otherwise.


In my opinion


+   @$(FIND) $(pkgdir) \(  -name \*.elc \) -exec $(RM) {} +


command tells that package management capabilities in Emacs are rather 
rudimentary (in comparison to e.g. python toolchain). That is why I am 
in favor to more manual dependency management. Notice that I am not 
against an option to do it from make.


Untested:

$(FIND) $(pkgdir) -name \*.elc -delete

"@" for silencing is intentionally dropped, parenthesis are unnecessary 
for single condition.



Actually, we need pkgdir := $(shell pwd)/pkg-deps.
CURDIR is wrong because default.mk will trigger evaluation in every make
sub-process as well.


default.mk is included from top level Makefile only.


Not sure here. I just noticed that GITVERSION got re-calculated many
times when looking at debug output of make. (It was slowing things down
significantly). GITVERSION is in targets.mk though.


GITVERSION is defined as ?=, so it is a variable with deferred 
evaluation. Immediately evaluated variable may defined using :=



pkgdir = $(top_builddir)/pkg-deps


How to define top_builddir? If also via `pwd`, I see not much difference.


I consider it as self-documenting code. Intermediate variable makes it 
apparent for readers that the directory is relative to the top of the 
package file tree.


Since out of source tree builds are not supported, I would consider 
adding emacs version to path. The idea is to allow keeping installed 
packages when switching between several emacs versions.


A note concerning variable name. For me it is associated for the 
directory where current package should be installed. I do not remember 
origin, but I noticed that such meaning is used in arch 
https://wiki.archlinux.org/title/creating_packages#Defining_PKGBUILD_variables 
Perhaps the same name is in gentoo in other sense that makes it suitable 
for storage of dependencies.

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

>> I know no other Emacs packages that manage dependencies using make.
>
> org-ql uses a helper shell script. org-roam and projectile use eldev 
> that is procedural, not declarative build system. I have not figured out 
> which way magit handles dependencies. I have no more ideas what projects 
> may involve unit tests and dependencies.

I see. Unfortunately, using third-party non-standard packages like
makem.sh or eldev will be more troublesome. We will need to sync Org
repo with them or demand users to install them separately.

Also, eldev is not on ELPA, and we cannot use it at all.

As for makem.sh, AFAIK, it will be even worse wrt network access - it
tries refreshing and downloading some packages by default, unless we
explicitly disable that feature. And under the hood, makem.sh anyway
does the same emacs --eval thing I used in the patch.

>>> I do not like the idea of network queries on every make.
>> Any better suggestions?
>
> Do not run install dependencies for regular targets like test or 
> compile. At least do not do it unless it is explicitly requested by 
> command line argument or a variable in local.mk

compat.el is required for "compile" target. Compilation will simply fail
with older Emacs. And "test" triggers "compile".

>> Actually, we need pkgdir := $(shell pwd)/pkg-deps.
>> CURDIR is wrong because default.mk will trigger evaluation in every make
>> sub-process as well.
>
> default.mk is included from top level Makefile only.

Not sure here. I just noticed that GITVERSION got re-calculated many
times when looking at debug output of make. (It was slowing things down
significantly). GITVERSION is in targets.mk though.

> By the way, it is 
> better to explicitly express that path is relative to top project 
> directory by defining e.g. top_builddir (or abs_top_builddir) at first and
>
> pkgdir = $(top_builddir)/pkg-deps

How to define top_builddir? If also via `pwd`, I see not much difference.

> Today I noticed the following trick in (likely stale) 
> https://github.com/org-roam/org-roam/blob/main/default.mk
>
> TOP := $(dir $(lastword $(MAKEFILE_LIST)))
>
> I have not realized if it is safe enough.

Reading through 6.14 Other Special Variables section of make manual, it
does not look safe - MAKEFILE_LIST may be altered by presence of includes.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 03/04/2023 00:00, Ihor Radchenko wrote:

Max Nikulin writes:


Ihor, do added makefile rules follow best practices used by other Emacs
packages in respect to dependencies?

I know no other Emacs packages that manage dependencies using make.


org-ql uses a helper shell script. org-roam and projectile use eldev 
that is procedural, not declarative build system. I have not figured out 
which way magit handles dependencies. I have no more ideas what projects 
may involve unit tests and dependencies.


Actually Org uses make just as interface, not as a declarative build 
system that is able to minimize work by updating only necessary targets. 
Build rules are effectively procedural: complete rebuild on every 
invocation.



I do not like the idea of network queries on every make.

Any better suggestions?


Do not run install dependencies for regular targets like test or 
compile. At least do not do it unless it is explicitly requested by 
command line argument or a variable in local.mk



+pkgdir = $(shell pwd)/pkg-deps


Make has CURDIR variable, but I am unsure if it is safe to use it in 
this context.


Actually, we need pkgdir := $(shell pwd)/pkg-deps.
CURDIR is wrong because default.mk will trigger evaluation in every make
sub-process as well.


default.mk is included from top level Makefile only. By the way, it is 
better to explicitly express that path is relative to top project 
directory by defining e.g. top_builddir (or abs_top_builddir) at first and


pkgdir = $(top_builddir)/pkg-deps

Today I noticed the following trick in (likely stale) 
https://github.com/org-roam/org-roam/blob/main/default.mk


TOP := $(dir $(lastword $(MAKEFILE_LIST)))

I have not realized if it is safe enough.

[PATCH v3] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Ihor Radchenko  writes:

>> It would be easier to review if this patch was split into 2 parts:
>> - add compat.el dependency (unused)
>> - replace functions to ones from compat.el
>
> Sure, but after spending half an hour trying to decouple this part, I
> gave up and decided to leave it as is. And there is nothing fancy in
> adding compat dependency in 2/3 patch anyway. Just one (require 'compat
> nil t), two macro definitions adviced by compat manual (copy-paste from
> compat code), and adding compat to EPACKAGES in makefiles.

After more fiddling with interactive rebase, I managed to split the
patch further.

See the attached.
Hope it is easier to review now.

>From 313f2b6acdf67f29f4f5f671e8278bb3e566cabf Mon Sep 17 00:00:00 2001
Message-Id: <313f2b6acdf67f29f4f5f671e8278bb3e566cabf.1680511418.git.yanta...@posteo.net>
From: Ihor Radchenko 
Date: Sat, 1 Apr 2023 12:00:48 +0200
Subject: [PATCH 1/7] Upgrade Org build system to handle third-party
 dependencies

* mk/default.mk (pkgdir): New variable holding the location of
third-party packages to be downloaded if necessary during compilation.
(EFLAGS): New variable holding extra flags to be passed to Emacs
executable when running make.
(EPACKAGES): List of packages to be installed (unless already present
in the `load-path') during compilation.
(package-install):
(INSTALL_PACKAGES): New command to download and install missing packages.
(BATCH): Update, setting default package location to pkgdir.
* mk/targets.mk (uppkg): New target to download install missing
packages.
(check test):
(compile compile-dirty): Use the new uppkg target.
(cleanpkg): New target cleaning up the downloaded packages.
(cleanall): Use the new target.
(.PHONY):
(CONF_BASE):
(CONF_DEST):
(CONF_CALL): Update according to the new variables and targets.
* .gitignore: Ignore the downloaded packages.

This commit paves the way towards third-party built-time dependencies
for Org.  In particular, towards including compat.el dependency.

According to EPACKAGES, we can auto-download necessary packages,
unless they are manually specified via -L switches in EFLAGS.

Link: https://orgmode.org/list/87v8ks6rhf.fsf@localhost
---
 .gitignore|  1 +
 mk/default.mk | 24 +++-
 mk/targets.mk | 24 
 3 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/.gitignore b/.gitignore
index 4bb81c359..0d9c5b297 100644
--- a/.gitignore
+++ b/.gitignore
@@ -48,6 +48,7 @@ local*.mk
 .gitattributes
 mk/x11idle
 ChangeLog
+pkg-deps/
 
 # Files generated during `make packages/org` in a clone of `elpa.git`.
 
diff --git a/mk/default.mk b/mk/default.mk
index fa46661e8..997b22b66 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -31,6 +31,15 @@ GIT_BRANCH =
 TMPDIR ?= /tmp
 testdir = $(TMPDIR)/tmp-orgtest
 
+# Where to store Org dependencies
+pkgdir = $(shell pwd)/pkg-deps
+
+# Extra flags to be passed to Emacs
+EFLAGS ?=
+
+# Third-party packages to install when running make
+EPACKAGES ?=
+
 # Configuration for testing
 # Verbose ERT summary by default for Emacs-28 and above.
 # To override:
@@ -72,12 +81,22 @@ REPRO_ARGS ?=
 req-ob-lang = --eval '(require '"'"'ob-$(ob-lang))'
 lst-ob-lang = ($(ob-lang) . t)
 req-extra   = --eval '(require '"'"'$(req))'
+package-install = --eval '(unless (require '"'"'$(package) nil t) (message "%s" load-path) (package-install '"'"'$(package)))'
 BTEST_RE   ?= \\(org\\|ob\\|ox\\)
 BTEST_LOAD  = \
 	--eval '(add-to-list '"'"'load-path (concat default-directory "lisp"))' \
 	--eval '(add-to-list '"'"'load-path (concat default-directory "testing"))'
 BTEST_INIT  = $(BTEST_PRE) $(BTEST_LOAD) $(BTEST_POST)
 
+ifeq (,$(EPACKAGES))
+INSTALL_PACKAGES =
+else
+INSTALL_PACKAGES = \
+	$(BATCH) \
+--eval '(package-refresh-contents)' \
+$(foreach package,$(EPACKAGES),$(package-install))
+endif
+
 BTEST = $(BATCH) $(BTEST_INIT) \
 	  -l org-batch-test-init \
 	  --eval '(setq \
@@ -120,7 +139,10 @@ EMACSQ  = $(EMACS)  -Q
 
 # Using emacs in batch mode.
 BATCH	= $(EMACSQ) -batch \
-	  --eval '(setq vc-handled-backends nil org-startup-folded nil org-element-cache-persistent nil)'
+	  $(EFLAGS) \
+	  --eval '(setq vc-handled-backends nil org-startup-folded nil org-element-cache-persistent nil)' \
+  --eval '(make-directory "$(pkgdir)" t)' \
+	  --eval '(setq package-user-dir "$(pkgdir)")' --eval '(package-initialize)'
 
 # Emacs must be started in toplevel directory
 BATCHO	= $(BATCH) \
diff --git a/mk/targets.mk b/mk/targets.mk
index 0bd293d68..ab8b830bb 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -27,21 +27,21 @@ ifneq ($(GITSTATUS),)
   GITVERSION := $(GITVERSION:.dirty=).dirty
 endif
 
-.PHONY:	all oldorg update update2 up0 up1 up2 single $(SUBDIRS) \
+.PHONY:	all oldorg update update2 up0 up1 up2 uppkg single $(SUBDIRS) \
 	check test install $(INSTSUB) \
 	info html pdf card refcard doc docs \
 	autoloads cleanall clean $(CLEANDIRS:%=clean%) \
 	clean-install cleanelc cleandirs \
-	cleanlisp cleandoc cleandocs 

[PATCH v2] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

I am attaching a slightly tweaked version of the patch that will make
less use of internet connection (use only when really, really
necessary), fixes variable expansion to be done once instead of in every
make sub-process, and adds some extra info messages.

Max Nikulin  writes:

> Ihor, do added makefile rules follow best practices used by other Emacs 
> packages in respect to dependencies?

I know no other Emacs packages that manage dependencies using make.

> I do not like the idea of network queries on every make.

Any better suggestions?

> In some cases I would prefer to specify a directory where compat.el is
> installed, so Makefile should not try to manage this directory.

Sure. See EFLAGS variable. Makefile will not try to manage
-L /path/to/compat directory.

> Originally I expected that either compat.el would be included into Org 
> repository either as a copy of the file or as git submodule.

That will create maintenance nightmare.

> In addition I am afraid of recursive removal of directories. It is too 
> easy to remove too much.
>
>> +pkgdir = $(shell pwd)/pkg-deps
>
> Make has CURDIR variable, but I am unsure if it is safe to use it in 
> this context.

Actually, we need pkgdir := $(shell pwd)/pkg-deps.
CURDIR is wrong because default.mk will trigger evaluation in every make
sub-process as well.

>> +-$(FIND) $(pkgdir) \(  -name \*.elc \) -exec $(RM) {} +
>
> find has -delete action. I see that "-exec $(RM)" is already used.

I just copied the existing code.
I guess we can use -delete as well. I do not have a lot of experience
with this usage of find. May you share the equivalent find call with
-delete that works with this patch?

>> +cleanpkg:
>> +-$(RMR) $(pkgdir)
>
> Perhaps it is impossible to completely avoid recursive deleting of 
> directories, but I still afraid of cases like
> https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123
> https://news.ycombinator.com/item?id=9254876

I do not think that we have any significant risk here.
I would not mind alternative way to implement clean target though. If
you know one.

>> Subject: [PATCH 2/3] Use compat.el library instead of ad-hoc compatibility
>>  function set
>
> It would be easier to review if this patch was split into 2 parts:
> - add compat.el dependency (unused)
> - replace functions to ones from compat.el

Sure, but after spending half an hour trying to decouple this part, I
gave up and decided to leave it as is. And there is nothing fancy in
adding compat dependency in 2/3 patch anyway. Just one (require 'compat
nil t), two macro definitions adviced by compat manual (copy-paste from
compat code), and adding compat to EPACKAGES in makefiles.

>From f95433f53878e8371bb28a045fdb5d06cf0877b9 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 1 Apr 2023 12:00:48 +0200
Subject: [PATCH 1/6] Upgrade Org build system to handle third-party
 dependencies

* mk/default.mk (pkgdir): New variable holding the location of
third-party packages to be downloaded if necessary during compilation.
(EFLAGS): New variable holding extra flags to be passed to Emacs
executable when running make.
(EPACKAGES): List of packages to be installed (unless already present
in the `load-path') during compilation.
(package-install):
(INSTALL_PACKAGES): New command to download and install missing packages.
(BATCH): Update, setting default package location to pkgdir.
* mk/targets.mk (uppkg): New target to download install missing
packages.
(check test):
(compile compile-dirty): Use the new uppkg target.
(cleanpkg): New target cleaning up the downloaded packages.
(cleanall): Use the new target.
(.PHONY):
(CONF_BASE):
(CONF_DEST):
(CONF_CALL): Update according to the new variables and targets.
* .gitignore: Ignore the downloaded packages.

This commit paves the way towards third-party built-time dependencies
for Org.  In particular, towards including compat.el dependency.

According to EPACKAGES, we can auto-download necessary packages,
unless they are manually specified via -L switches in EFLAGS.

Link: https://orgmode.org/list/87v8ks6rhf.fsf@localhost
---
 .gitignore|  1 +
 mk/default.mk | 24 +++-
 mk/targets.mk | 24 
 3 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/.gitignore b/.gitignore
index 4bb81c359..a58670c90 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,7 @@ t
 auto
 tmp
 TODO
+/pkg-deps
 
 # and collateral damage from Emacs
 
diff --git a/mk/default.mk b/mk/default.mk
index fa46661e8..997b22b66 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -31,6 +31,15 @@ GIT_BRANCH =
 TMPDIR ?= /tmp
 testdir = $(TMPDIR)/tmp-orgtest
 
+# Where to store Org dependencies
+pkgdir = $(shell pwd)/pkg-deps
+
+# Extra flags to be passed to Emacs
+EFLAGS ?=
+
+# Third-party packages to install when running make
+EPACKAGES ?=
+
 # Configuration for testing
 # Verbose ERT summary by default for Emacs-28 and above.
 # To override:
@@ -72,12 +81,22 @@ REPRO_ARGS ?=
 

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 01/04/2023 17:31, Ihor Radchenko wrote:


See the attached patch set adding support of compat.el.


Ihor, do added makefile rules follow best practices used by other Emacs 
packages in respect to dependencies? I do not like the idea of network 
queries on every make. In some cases I would prefer to specify a 
directory where compat.el is installed, so Makefile should not try to 
manage this directory. My impression is that package management code in 
Emacs is too oriented for interactive usage. E.g. python's pip caches 
downloaded packages or allows to specify a proxy or an alternative 
source. Moreover precise version may be specified while only last 
version is hosted on ELPA. In other package managers version lock is 
often preferred to avoid unexpected effects of spurious update. 
Fortunately ELPA packages are at least reviewed before publication.


Originally I expected that either compat.el would be included into Org 
repository either as a copy of the file or as git submodule.


In addition I am afraid of recursive removal of directories. It is too 
easy to remove too much.



+pkgdir = $(shell pwd)/pkg-deps


Make has CURDIR variable, but I am unsure if it is safe to use it in 
this context.



+   -$(FIND) $(pkgdir) \(  -name \*.elc \) -exec $(RM) {} +


find has -delete action. I see that "-exec $(RM)" is already used.


+cleanpkg:
+   -$(RMR) $(pkgdir)


Perhaps it is impossible to completely avoid recursive deleting of 
directories, but I still afraid of cases like

https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123
https://news.ycombinator.com/item?id=9254876


Subject: [PATCH 2/3] Use compat.el library instead of ad-hoc compatibility
 function set


It would be easier to review if this patch was split into 2 parts:
- add compat.el dependency (unused)
- replace functions to ones from compat.el

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

> If dependencies may be satisfied by installing system packages then it 
> is easier to run Org from repository clone or to build new version of 
> elpa-org package.

The patch provides EFLAGS variable that will allow supplying flags to
emacs executable. One can just use ~EFLAGS="-L /path/to/local/compat/" make~.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 02/04/2023 15:52, Ihor Radchenko wrote:

And it should not matter anyway. We only need compat when Org mode is
not built-in, but installed by other means. Normally, users will simply
install Org from ELPA where we do not need to worry about older versions
of Compat.


If dependencies may be satisfied by installing system packages then it 
is easier to run Org from repository clone or to build new version of 
elpa-org package.

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Max Nikulin  writes:

> On 01/04/2023 18:38, Daniel Mendler wrote:
>>  From my side, the change looks good. The current Compat version 29.1.4.x
>> is stable with no known issues.
>
> Debian Bookworm and Ubuntu 23.04 (currently frozen testing and beta 
> accordingly) have elpa-compat-29.1.3.4 and 29.1.3.2. Are some issues 
> expected if a bit earlier version is required?

I don't think so. At least, we do not appear to be using anything listed
in NEWS for later Compat versions (https://elpa.gnu.org/packages/compat.html).

And it should not matter anyway. We only need compat when Org mode is
not built-in, but installed by other means. Normally, users will simply
install Org from ELPA where we do not need to worry about older versions
of Compat.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Max Nikulin]

On 01/04/2023 18:38, Daniel Mendler wrote:

 From my side, the change looks good. The current Compat version 29.1.4.x
is stable with no known issues.


Debian Bookworm and Ubuntu 23.04 (currently frozen testing and beta 
accordingly) have elpa-compat-29.1.3.4 and 29.1.3.2. Are some issues 
expected if a bit earlier version is required?

Re: [PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Daniel Mendler]

On 4/1/23 12:31, Ihor Radchenko wrote:
> Ihor Radchenko  writes:
> 
>> I have recently been contacted by the current compat.el maintainer
>> asking if we are willing to adapt compat.el in Org.
> 
> See the attached patch set adding support of compat.el.
> 
> I had to update Org's build system to handle third-party packages.
> Please, give it a close check (first patch).
> 
> The second patch adds the actual Elisp part and replaces some
> compatibility functions with what is provided by compat.el.
> Note that not all the functions are available in compat.el for now.
> Daniel, you might want to take a look.
> 
> The last patch adds the necessary explanation for users who install Org
> from git. Now, they must install compat.el manually to make Org work.

>From my side, the change looks good. The current Compat version 29.1.4.x
is stable with no known issues.

Daniel

[PATCH] Add compat.el support to Org (was: [POLL] Use compat.el in Org? (was: Useful package? Compat.el))

[Ihor Radchenko]

Ihor Radchenko  writes:

> I have recently been contacted by the current compat.el maintainer
> asking if we are willing to adapt compat.el in Org.

See the attached patch set adding support of compat.el.

I had to update Org's build system to handle third-party packages.
Please, give it a close check (first patch).

The second patch adds the actual Elisp part and replaces some
compatibility functions with what is provided by compat.el.
Note that not all the functions are available in compat.el for now.
Daniel, you might want to take a look.

The last patch adds the necessary explanation for users who install Org
from git. Now, they must install compat.el manually to make Org work.

>From f95433f53878e8371bb28a045fdb5d06cf0877b9 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 1 Apr 2023 12:00:48 +0200
Subject: [PATCH 1/3] Upgrade Org build system to handle third-party
 dependencies

* mk/default.mk (pkgdir): New variable holding the location of
third-party packages to be downloaded if necessary during compilation.
(EFLAGS): New variable holding extra flags to be passed to Emacs
executable when running make.
(EPACKAGES): List of packages to be installed (unless already present
in the `load-path') during compilation.
(package-install):
(INSTALL_PACKAGES): New command to download and install missing packages.
(BATCH): Update, setting default package location to pkgdir.
* mk/targets.mk (uppkg): New target to download install missing
packages.
(check test):
(compile compile-dirty): Use the new uppkg target.
(cleanpkg): New target cleaning up the downloaded packages.
(cleanall): Use the new target.
(.PHONY):
(CONF_BASE):
(CONF_DEST):
(CONF_CALL): Update according to the new variables and targets.
* .gitignore: Ignore the downloaded packages.

This commit paves the way towards third-party built-time dependencies
for Org.  In particular, towards including compat.el dependency.

According to EPACKAGES, we can auto-download necessary packages,
unless they are manually specified via -L switches in EFLAGS.

Link: https://orgmode.org/list/87v8ks6rhf.fsf@localhost
---
 .gitignore|  1 +
 mk/default.mk | 24 +++-
 mk/targets.mk | 24 
 3 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/.gitignore b/.gitignore
index 4bb81c359..a58670c90 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,7 @@ t
 auto
 tmp
 TODO
+/pkg-deps
 
 # and collateral damage from Emacs
 
diff --git a/mk/default.mk b/mk/default.mk
index fa46661e8..997b22b66 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -31,6 +31,15 @@ GIT_BRANCH =
 TMPDIR ?= /tmp
 testdir = $(TMPDIR)/tmp-orgtest
 
+# Where to store Org dependencies
+pkgdir = $(shell pwd)/pkg-deps
+
+# Extra flags to be passed to Emacs
+EFLAGS ?=
+
+# Third-party packages to install when running make
+EPACKAGES ?=
+
 # Configuration for testing
 # Verbose ERT summary by default for Emacs-28 and above.
 # To override:
@@ -72,12 +81,22 @@ REPRO_ARGS ?=
 req-ob-lang = --eval '(require '"'"'ob-$(ob-lang))'
 lst-ob-lang = ($(ob-lang) . t)
 req-extra   = --eval '(require '"'"'$(req))'
+package-install = --eval '(unless (require '"'"'$(package) nil t) (message "%s" load-path) (package-install '"'"'$(package)))'
 BTEST_RE   ?= \\(org\\|ob\\|ox\\)
 BTEST_LOAD  = \
 	--eval '(add-to-list '"'"'load-path (concat default-directory "lisp"))' \
 	--eval '(add-to-list '"'"'load-path (concat default-directory "testing"))'
 BTEST_INIT  = $(BTEST_PRE) $(BTEST_LOAD) $(BTEST_POST)
 
+ifeq (,$(EPACKAGES))
+INSTALL_PACKAGES =
+else
+INSTALL_PACKAGES = \
+	$(BATCH) \
+--eval '(package-refresh-contents)' \
+$(foreach package,$(EPACKAGES),$(package-install))
+endif
+
 BTEST = $(BATCH) $(BTEST_INIT) \
 	  -l org-batch-test-init \
 	  --eval '(setq \
@@ -120,7 +139,10 @@ EMACSQ  = $(EMACS)  -Q
 
 # Using emacs in batch mode.
 BATCH	= $(EMACSQ) -batch \
-	  --eval '(setq vc-handled-backends nil org-startup-folded nil org-element-cache-persistent nil)'
+	  $(EFLAGS) \
+	  --eval '(setq vc-handled-backends nil org-startup-folded nil org-element-cache-persistent nil)' \
+  --eval '(make-directory "$(pkgdir)" t)' \
+	  --eval '(setq package-user-dir "$(pkgdir)")' --eval '(package-initialize)'
 
 # Emacs must be started in toplevel directory
 BATCHO	= $(BATCH) \
diff --git a/mk/targets.mk b/mk/targets.mk
index 0bd293d68..ab8b830bb 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -27,21 +27,21 @@ ifneq ($(GITSTATUS),)
   GITVERSION := $(GITVERSION:.dirty=).dirty
 endif
 
-.PHONY:	all oldorg update update2 up0 up1 up2 single $(SUBDIRS) \
+.PHONY:	all oldorg update update2 up0 up1 up2 uppkg single $(SUBDIRS) \
 	check test install $(INSTSUB) \
 	info html pdf card refcard doc docs \
 	autoloads cleanall clean $(CLEANDIRS:%=clean%) \
 	clean-install cleanelc cleandirs \
-	cleanlisp cleandoc cleandocs cleantest \
+	cleanlisp cleandoc cleandocs cleantest cleanpkg \
 	compile compile-dirty uncompiled \
 	config config-test 

Re: [POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Daniel Mendler]

On 1/30/23 20:40, Greg Minshall wrote:
> i see, yes.  i'm just thinking that, for a given release CUR (like i
> know anything about org-mode release procedures!)  we would use whatever
> has been available in compat.el since release CUR-n (for whatever n we
> use -- 2?), and supplement that, in org-compat.el, with whatever other
> compatibility features *we* (org-mode) need to support releases [CUR-n
> .. CUR].
> 
> (and, presumably, contribute whatever might be appropriate from
> org-compat.el to compat.el, so we can prune it out from org-compat.el at
> some future point in time.)

Sounds good. But note that Compat will at some point have sufficient
coverage of the existing APIs, such that Org may not miss anything. From
then on new additions will only be made at the time of a new Emacs
release. The current Compat already supports many Emacs 29 APIs, which
is in time for the upcoming pretest. Compat won't add any APIs which are
still part of the unstable development master branch (before the branch
cut and feature freeze), which reduces the need for coordination.

If you are missing something right now, and want to introduce and use a
new backported function, e.g., something new from Emacs 29, instead of
going via the org-compat indirection, you could also make an addition
directly to Compat, a Compat release can be prepared a short while
after, and then you start using the new function in the Org development
version.

Daniel

Re: [POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Greg Minshall]

Ihor,

> >> 1. If compat.el happens to lack support of some function, we will need
> >>to contribute to compat.el directly and synchronize Org releases with
> >>compat.el releases.
> >
> > would a separate "org-compat.el" (in addition to compat.el) somehow
> > solve this?  (i worry about the synch'ing.)
> 
> That's what we already do.
> Using compat.el means that we can remove some of the functions from
> org-compat.el and instead rely on compat.el where the same functions are
> maintained more carefully.

i see, yes.  i'm just thinking that, for a given release CUR (like i
know anything about org-mode release procedures!)  we would use whatever
has been available in compat.el since release CUR-n (for whatever n we
use -- 2?), and supplement that, in org-compat.el, with whatever other
compatibility features *we* (org-mode) need to support releases [CUR-n
.. CUR].

(and, presumably, contribute whatever might be appropriate from
org-compat.el to compat.el, so we can prune it out from org-compat.el at
some future point in time.)

i'm *only* thinking of trying to de-couple org-mode development from
that of compat.el, in the mindset that "less cross-dependencies" ==
"less complication".  if that makes sense.

cheers, Greg

Re: [POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Ihor Radchenko]

Greg Minshall  writes:

>> 1. If compat.el happens to lack support of some function, we will need
>>to contribute to compat.el directly and synchronize Org releases with
>>compat.el releases.
>
> would a separate "org-compat.el" (in addition to compat.el) somehow
> solve this?  (i worry about the synch'ing.)

That's what we already do.
Using compat.el means that we can remove some of the functions from
org-compat.el and instead rely on compat.el where the same functions are
maintained more carefully.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Greg Minshall]

Ihor,

> Cons:
> 
> 1. If compat.el happens to lack support of some function, we will need
>to contribute to compat.el directly and synchronize Org releases with
>compat.el releases.

would a separate "org-compat.el" (in addition to compat.el) somehow
solve this?  (i worry about the synch'ing.)

cheers, Greg

Re: [POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Kyle Meyer]

Ihor Radchenko writes:

> I have recently been contacted by the current compat.el maintainer
> asking if we are willing to adapt compat.el in Org.

I'm in favor of Org using Compat.  And grepping around emacs.git, there
are already two bundled packages (erc and python) with

  (require 'compat nil 'noerror)

so I don't expect there would be objections/questions from the Emacs
side.

Re: [POLL] Use compat.el in Org?

[Samuel Wales]

what daniel said sgtm

On 1/27/23, Daniel Mendler  wrote:
> On 1/27/23 21:38, Tim Cross wrote:
>>> As long as we keep our promise in terms of backward compatibility with
>>> older Emacs versions, I'm all for it.
>>
>> I would agree. I would also add that even with the use of this package,
>> I don't think we should use it to increase the number of versions we
>> support as support is not as simple as dropping in a compatibility
>> library.
>
> True. The Compat package cannot fix bugs below the Elisp level or
> provide APIs which cannot be backported, e.g., big integer support. If
> Org relies on behavior of the Emacs display engine or the C core of a
> certain Emacs version, Compat cannot help.
>
> The advantage would be that the maintenance burden of org-compat would
> be reduced. Many packages can share the backported functions by
> depending on Compat, which will increase robustness and reduce the risk
> of unexpected bugs. The community only has to maintain a single set of
> backported functions in a single package, instead of scattering
> compatibility code across many packages.
>
>> These libraries come with a cost. Often, compatibility code
>> does not perform as well and/or is much more complicated and more likely
>> to have bugs. The more a version of emacs needs to rely on this library
>> to run org-mode, the higher the likelihood performance will be degraded
>> or unexpected new bugs are found.
>
> To give some context about the stability aspect - many backported
> compatibility functions are copied verbatim from newer Emacs versions.
> Every compatibility function provided by Compat is covered by tests,
> which are executed via CI on all supported Emacs versions (>= 24.4). I
> make sure that no functions are backported which perform much worse such
> that they would introduce performance bugs.
>
> Daniel
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: [POLL] Use compat.el in Org?

[Daniel Mendler]

On 1/27/23 21:38, Tim Cross wrote:
>> As long as we keep our promise in terms of backward compatibility with
>> older Emacs versions, I'm all for it.
> 
> I would agree. I would also add that even with the use of this package,
> I don't think we should use it to increase the number of versions we
> support as support is not as simple as dropping in a compatibility
> library. 

True. The Compat package cannot fix bugs below the Elisp level or
provide APIs which cannot be backported, e.g., big integer support. If
Org relies on behavior of the Emacs display engine or the C core of a
certain Emacs version, Compat cannot help.

The advantage would be that the maintenance burden of org-compat would
be reduced. Many packages can share the backported functions by
depending on Compat, which will increase robustness and reduce the risk
of unexpected bugs. The community only has to maintain a single set of
backported functions in a single package, instead of scattering
compatibility code across many packages.

> These libraries come with a cost. Often, compatibility code
> does not perform as well and/or is much more complicated and more likely
> to have bugs. The more a version of emacs needs to rely on this library
> to run org-mode, the higher the likelihood performance will be degraded
> or unexpected new bugs are found.

To give some context about the stability aspect - many backported
compatibility functions are copied verbatim from newer Emacs versions.
Every compatibility function provided by Compat is covered by tests,
which are executed via CI on all supported Emacs versions (>= 24.4). I
make sure that no functions are backported which perform much worse such
that they would introduce performance bugs.

Daniel

Re: [POLL] Use compat.el in Org?

[Tim Cross]

Bastien Guerry  writes:

> Hi Ihor,
>
> Ihor Radchenko  writes:
>
>> I have recently been contacted by the current compat.el maintainer
>> asking if we are willing to adapt compat.el in Org.
>
> Very nice!
>
>> WDYT?
>
> As long as we keep our promise in terms of backward compatibility with
> older Emacs versions, I'm all for it.

I would agree. I would also add that even with the use of this package,
I don't think we should use it to increase the number of versions we
support as support is not as simple as dropping in a compatibility
library. These libraries come with a cost. Often, compatibility code
does not perform as well and/or is much more complicated and more likely
to have bugs. The more a version of emacs needs to rely on this library
to run org-mode, the higher the likelihood performance will be degraded
or unexpected new bugs are found.

So, use the library, but keep the existing policy of officially
supporting only the previous two major releases. If org does work with
even older versions, that is great, but not supported.

Re: [POLL] Use compat.el in Org?

[Bastien Guerry]

Hi Ihor,

Ihor Radchenko  writes:

> I have recently been contacted by the current compat.el maintainer
> asking if we are willing to adapt compat.el in Org.

Very nice!

> WDYT?

As long as we keep our promise in terms of backward compatibility with
older Emacs versions, I'm all for it.

-- 
 Bastien

[POLL] Use compat.el in Org? (was: Useful package? Compat.el)

[Ihor Radchenko]

Timothy  writes:

> I’ve recently come across an interesting looking library available on ELPA,
> . I’m thinking in future this could allow us 
> to
> both use newer features and also support older versions of Emacs, e.g.
>
> Org 10.X is developed for Emacs 28.1 and newer, but supports Emacs 24.3 and
> newer with compat.el.
>
> Just tossing this out as an idea :)

I have recently been contacted by the current compat.el maintainer
asking if we are willing to adapt compat.el in Org.

Pros:

1. We will have less headache maintaining org-compat.el.
2. We will get an ability to use functions and macros from newer Emacs
   versions almost for free.
   
Cons:

1. If compat.el happens to lack support of some function, we will need
   to contribute to compat.el directly and synchronize Org releases with
   compat.el releases.

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at