Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Ihor Radchenko]

Ihor Radchenko  writes:

> Ok. Here are the tentative patches for Org manual and WORG maintenance
> page.

Applied.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=dd4e06ddc
https://git.sr.ht/~bzg/worg/commit/432828ce

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Ihor Radchenko]

Tim Cross  writes:

> - Adding a section regarding pubic/private API and naming conventions to
>   the Hacking section of the manual. This section could outline what the
>   processes are for adding/changing APIs. 

I think we can add a section to Hacking.

But what should we list there?

At least,

1. prefix--suffix can change any time. prefix-suffix is stable
2. ORG-NEWS mention if we do important changes breaking (1)
3. If we decide to remove or change some function/variable, we first
   obsolete it.

What else?

> - Add the maintainers pledge to the manual as well. It is useful for
>   users to know what the maintainers pledge to try and do. I doubt many
>   users will find the page on worg which already exists. At the very
>   least, add a link to the wrog page.

Should we straight put
https://bzg.fr/en/the-software-maintainers-pledge/ after Installation
section of the manual? Maybe into a new section called "Updating Org"?

Bastien, what do you think?

> - Briefly document the org maintenance and release process or add links
>   to relevant worg pages. . 

Also into Hacking, I think. As extra reference after API and
compatibility conventions.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> I guess we are limited by what the packages we rely on support. For
>> example, if geiser doesn't support Emacs 26 but org is supposed to,
>> there isn't much we can do. We cannot afford to fork geiser and modify
>> it to add the support and even if we provided a patch to add the support
>> to the geiser project, they may not merge it.
>>
>> Best we can do is best effort and reflect this limitation in the manual
>> where we stipulate what our backwards compatibility policy is. 
>
> Ok. Here are the tentative patches for Org manual and WORG maintenance
> page.
>

No objections from me. Just based on reading the patches, they appear to
clarify our support policy and therefore should be helpful. 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Ihor Radchenko]

Tim Cross  writes:

> I guess we are limited by what the packages we rely on support. For
> example, if geiser doesn't support Emacs 26 but org is supposed to,
> there isn't much we can do. We cannot afford to fork geiser and modify
> it to add the support and even if we provided a patch to add the support
> to the geiser project, they may not merge it.
>
> Best we can do is best effort and reflect this limitation in the manual
> where we stipulate what our backwards compatibility policy is. 

Ok. Here are the tentative patches for Org manual and WORG maintenance
page.

>From 15a929176e9c03be6ef3bdb5b6401a8732c188a7 Mon Sep 17 00:00:00 2001
Message-Id: <15a929176e9c03be6ef3bdb5b6401a8732c188a7.1670680001.git.yanta...@posteo.net>
From: Ihor Radchenko 
Date: Sat, 10 Dec 2022 16:46:10 +0300
Subject: [PATCH] org-maintenance.org: Document third-party package
 compatibility

* org-maintenance.org (Org releases are compatible with the latest
three major Emacs releases): Document that we only guarantee
compatibility with the latest stable versions of third-party packages.

Link: https://orgmode.org/list/86iljd3x90@gmail.com
---
 org-maintenance.org | 4 
 1 file changed, 4 insertions(+)

diff --git a/org-maintenance.org b/org-maintenance.org
index 8c7d9492..13f0b30c 100644
--- a/org-maintenance.org
+++ b/org-maintenance.org
@@ -172,6 +172,10 @@ ** Org releases are compatible with the latest three major Emacs releases
 older Emacsen: but maintainers are not bound to fix bugs reported on
 them.
 
+Some Org components also depend on third-party packages available
+through package archives.  Org is only guaranteed to be compatible
+with the last stable versions of these third-party packages.
+
 Org versions that are not yet released (from the main or bugfix
 branch) don't come with any promise regarding compatibility.
 
-- 
2.38.1

>From de2a6b9a924d2c5b135104dc2a383bafdb1995bb Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 10 Dec 2022 16:45:03 +0300
Subject: [PATCH] org-manual: Document third-party package compatibility

* doc/org-manual.org (Installation): Document that we only guarantee
compatibility with the latest stable versions of third-party packages.

Link: https://orgmode.org/list/86iljd3x90@gmail.com
---
 doc/org-manual.org | 4 
 1 file changed, 4 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 63e9d4139..8f4da3149 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -96,6 +96,10 @@ ** Installation
 that Org stable versions are meant to be fully compatible with the
 last three stable versions of Emacs but not with older Emacsen.
 
+Some Org components also depend on third-party packages available
+through package archives.  Org is only guaranteed to be compatible
+with the last stable versions of these third-party packages.
+
 *** Using Emacs packaging system
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.38.1


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

>> It might be worthwhile defining what is meant by 3rd party packages.
>>
>> For example, ob-scheme relying on geiser as a 3rd party package is one
>> thing. Org roam is another type of 3rd party package. I think they need
>> different approaches. The first is about our (org) interface to them and
>> the latter is about their (org roam) interface with org.
>
> Sorry for not being clear.
> I was referring to third-party packages Org optionally depends on.
> Like geiser or citeproc or engrave-faces.
>
>> For the former (e.g. geiser type), I don't think backwards compatibility
>> is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
>> fairly trivial to update to the current version. We just need to support
>> the current version.
>
> Yes, but what if, say, the newest geiser version no longer supports the
> Emacs version Org still supports?
>

I guess we are limited by what the packages we rely on support. For
example, if geiser doesn't support Emacs 26 but org is supposed to,
there isn't much we can do. We cannot afford to fork geiser and modify
it to add the support and even if we provided a patch to add the support
to the geiser project, they may not merge it.

Best we can do is best effort and reflect this limitation in the manual
where we stipulate what our backwards compatibility policy is. 

>> For the latter (e.g. org-roam), backwards compatibility is much more
>> important. Org needs to provide as stable a public API for such packages
>> as possible. It may even be worthwhile separating the API into a
>> public/external API and an internal/private API. The former would be as
>> stable as possible and when changes are made, all efforts to provide
>> backwards compatibility are made. The latter is also stable, but more
>> subject to change. It is the API intended for internal org use. If
>> external packages use it, it is at their own risk. We would still try to
>> keep it stable, but with less of a guarantee and we may not provide
>> backwards compatibility if doing so was going to cause too much
>> complexity or maintenance burden etc.
>
> That's what we already do.
> I mean, https://bzg.fr/en/the-software-maintainers-pledge/ :)
>
> Public/private is the usual org-* vs. org--*.
> We never break org-*. Even if we do, we first deprecate things.
>

Yes, I know. I included it for completeness. However, I'm not convinced
worg is an effective communication channel for this information. We may
need to either add it to the manual or reference it from the
manual. (I'm reminded of the Douglas Adams quote from the Hitch Hikers
Guide [1]).

>> The big thing here is IMO avoiding surprise. We probably should add a
>> section to the 'Hacking' section of the manual which outlines our
>> position wrt backwards compatibility and API (public/private) and change
>> etc. I think most people expect change (even if they might not like
>> it). What they don't like is unexpected change. As long as we are
>> conservative and communicate change when necessary, we will likely be
>> OK.
>
> I hope that we never had unexpected changes. Isn't it for granted? I
> felt like it was always the case in Org and Emacs forever.
>
> It feels so obvious that I cannot even figure out how we could clarify
> it in the manual.

The one thing we can expect is unexpected change!

What is expected by one can be unexpected by others and what may seem
obvious to you or me may not be as obvious to others. We should be
extremely careful about assumptions regarding what can be taken for
granted. Org mode has a very broad user base. We have users with varying
familiarity with Emacs, software development practices and basic
software lifecycle management. For example, I know that symbol names
with '--' in them tend to indicate private API elements. However, that
is because I've been using Emacs for a long time. Someone new or someone
who is not a programmer may not be familiar with this convention.

There has been multiple occasions where I've seen posts on this list
from users complaining about unexpected and breaking changes. Such posts
usually end up in a thread about all the things org developers do to try
and mitigate such things. This would indicate it isn't necessarily taken
for granted by all. Often, those complaining were unaware of changes
being documented in the NEWS file or failed to check it before upgrading
or were unaware of backwards compatibility policy or didn't understand
the extent to which org relies on other external packages/services etc.

Then there is the situation where a change has unforeseen
impact. Consider the change a while ago with electric-indent.

I agree that in general, the org development process is pretty good in
its approach to handling change and maintenance of compatibility. Where
we are perhaps a little weak is in ensuring we communicate this
effectively. 

Perhaps we don't need to do anything extra. However, I feel it would be
beneficial to be very explicit regarding 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Ihor Radchenko]

> It might be worthwhile defining what is meant by 3rd party packages.
>
> For example, ob-scheme relying on geiser as a 3rd party package is one
> thing. Org roam is another type of 3rd party package. I think they need
> different approaches. The first is about our (org) interface to them and
> the latter is about their (org roam) interface with org.

Sorry for not being clear.
I was referring to third-party packages Org optionally depends on.
Like geiser or citeproc or engrave-faces.

> For the former (e.g. geiser type), I don't think backwards compatibility
> is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
> fairly trivial to update to the current version. We just need to support
> the current version.

Yes, but what if, say, the newest geiser version no longer supports the
Emacs version Org still supports?

> For the latter (e.g. org-roam), backwards compatibility is much more
> important. Org needs to provide as stable a public API for such packages
> as possible. It may even be worthwhile separating the API into a
> public/external API and an internal/private API. The former would be as
> stable as possible and when changes are made, all efforts to provide
> backwards compatibility are made. The latter is also stable, but more
> subject to change. It is the API intended for internal org use. If
> external packages use it, it is at their own risk. We would still try to
> keep it stable, but with less of a guarantee and we may not provide
> backwards compatibility if doing so was going to cause too much
> complexity or maintenance burden etc.

That's what we already do.
I mean, https://bzg.fr/en/the-software-maintainers-pledge/ :)

Public/private is the usual org-* vs. org--*.
We never break org-*. Even if we do, we first deprecate things.

> The big thing here is IMO avoiding surprise. We probably should add a
> section to the 'Hacking' section of the manual which outlines our
> position wrt backwards compatibility and API (public/private) and change
> etc. I think most people expect change (even if they might not like
> it). What they don't like is unexpected change. As long as we are
> conservative and communicate change when necessary, we will likely be
> OK.

I hope that we never had unexpected changes. Isn't it for granted? I
felt like it was always the case in Org and Emacs forever.

It feels so obvious that I cannot even figure out how we could clarify
it in the manual.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

> Hi,
>
> Org promises to support the last three Emacs releases.
> However, it is less clear what is our policy wrt third-party packages.
>
> We do need third-party packages, for example, in babel backends.
> Sometimes, we have to make changes to the ob-*.el files in order to
> accommodate changes in the required third-party packages. Like in recent
> changes to ob-scheme where `run-geiser' function is now obsolete
> upstream.
>
> Should we try to support obsolete functions/variables in third-party
> packages? Should we try to work around breaking changes and support both
> before/after package versions?
>
> The answer is not obvious as older Emacs versions might not be supported
> by some third-party packages. Then, logically, we have to support older
> package versions compatible with the oldest Emacs version we support.
> But it might be hard to keep track of such scenarios.
>

It might be worthwhile defining what is meant by 3rd party packages.

For example, ob-scheme relying on geiser as a 3rd party package is one
thing. Org roam is another type of 3rd party package. I think they need
different approaches. The first is about our (org) interface to them and
the latter is about their (org roam) interface with org.

For the former (e.g. geiser type), I don't think backwards compatibility
is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
fairly trivial to update to the current version. We just need to support
the current version.

For the latter (e.g. org-roam), backwards compatibility is much more
important. Org needs to provide as stable a public API for such packages
as possible. It may even be worthwhile separating the API into a
public/external API and an internal/private API. The former would be as
stable as possible and when changes are made, all efforts to provide
backwards compatibility are made. The latter is also stable, but more
subject to change. It is the API intended for internal org use. If
external packages use it, it is at their own risk. We would still try to
keep it stable, but with less of a guarantee and we may not provide
backwards compatibility if doing so was going to cause too much
complexity or maintenance burden etc.

The big thing here is IMO avoiding surprise. We probably should add a
section to the 'Hacking' section of the manual which outlines our
position wrt backwards compatibility and API (public/private) and change
etc. I think most people expect change (even if they might not like
it). What they don't like is unexpected change. As long as we are
conservative and communicate change when necessary, we will likely be
OK.