Re: bug#59882: Multiple versions of Org in load-path problem

[Tim Cross]

Ihor Radchenko  writes:

> Max Nikulin  writes:
>
>> On 27/12/2022 16:47, Ihor Radchenko wrote:
>>> Can you then try to test using Emacs 28?
>>> The main question if whether this has been fixed in newer Emacs releases
>>> or it is also something to do with OS environment.
>>
>> I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
>> buffer displays a bit more warnings and usual `org-assert-version' 
>> warnings and errors are present as well. It might have another level of 
>> complexity due to .eln files. I am unsure what happens with calls to 
>> undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.
>
> Considering the popularity of Debian-based distros, may someone take a
> closer look on what may be going on? Since the latest Emacs release also
> suffers from the problem, I am afraid that the issue will be present in
> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

I don't run Debian or Ubuntu anymore. However, I do recall that debian
does use a modified Emacs startup which is not part of the standard
Emacs distribution. They do this to enable the ability to have multiple
versions of Emacs installed at the same time.

If you are unable to reproduce the issue with an Emacs built from
sources and only from the versions of Emacs installed bia apt, then I
would be looking at the sources for the apt package and the modified
*.el files it uses as it could be something in there which is triggering
the issue (seem unlikely to be coincidence that a customization to allow
running of multiple versions is also triggering a version conflict in
org?)

. 

Re: [POLL] Proposed syntax for timestamps with time zone info (was: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda)

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>>> Either I understand you wrong, or you don't know what you are
>>> talking about. 2023-03-23 02:30 @Europe/Berlin refers to /two/
>>> points in time, thus it /is/ ambiguous. If you use disambiguating
>>> "time zones" (MEZ vs MESZ in this case) you can resolve that.
>>
>> I think the confusion relates to context interpretation. If you see
>> @Europe/Berlin in isolation, then it is ambiguous as it can refer to two
>> different time zone definitions (standard v daylight savings).  However,
>> if you consider it in conjunction with a date and time, as in 2023-03-23
>> 02:30 @Europe/Berlin, then it isn't ambiguous - in that case, it really
>> just says 'Lookup the time zone offset in the databse for Berlin as of
>> that date and time.
>>...
>> Personally, I cannot see the use case of including both a fully
>> qualitifed time zone (as in @Europe/Berlin) and an offset...
>
> Let me try to explain better. Just specifying time zone is ambiguous
> once per year during daylight transition.
>
> [2023-03-29 02:30 @Europe/Berlin] is special.
>
> According to https://www.timeanddate.com/time/zone/germany/berlin,
> 2023-03-29 is the time when the clock is shifted one hour back due to
> the daylight saving transition. The wall time goes like
>
> 2023-03-29 02:30 -> ... -> 02:59 -> (CEST -> CET) 02:00 -> ... -> 2:30 
> (again!)
>
> So, [2023-03-29 02:30 @Europe/Berlin] can mean two time points: before
> and after the transition. Specifying explicit offset is thus necessary
> in this specific scenario to disambiguate the timestamp:
>
> [2023-03-29 02:30+2 @Europe/Berlin] (before transition)
> [2023-03-29 02:30+1 @Europe/Berlin] (after transition)

OK, in that case, I think what we are in danger of here is letting
the perfect be the enemy of good.

The problems of daylight savings transition points is fairly well
understood and I think it is fairly well accepted that there is
ambiguity arising from the use of daylight savings.

The real question is, can the additional complexity associated with
including both a time zone name and an offset be justified in order to
handle the very small number of time stamps which will fall within the
daylight savings transition hour for those locations which have daylight
savings? Keep[ing in mind that the complexity is less to do with the
time stamp format and more to do with using that information in any
meaningful sense. This, combined with the reduced readability of such
time stamps and increased possibility of user confusion leads me to
question if allowing time stamps with both offset and time zone together
in the one time stamp is worthwhile. 

Re: [POLL] Proposed syntax for timestamps with time zone info (was: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda)

[Tim Cross]

 writes:

> [[PGP Signed Part:Undecided]]
> On Tue, Jan 31, 2023 at 11:12:00PM +0300, Jean Louis wrote:
>> * Ihor Radchenko  [2023-01-31 16:46]:
>> > Specifying just @Europe/Berlin is ambiguous around the daylight savings
>> > transition.
>> 
>> Sorry, I cannot see practical example why is it ambiguous. Unless
>> programmer does not take all information in account, it is not
>> ambigous. People on this planet agree on time zones in advance, so
>> there are few changes that people cannot plan in advance.
>
> (snipped the huge CC list)
>
> Either I understand you wrong, or you don't know what you are
> talking about. 2023-03-23 02:30 @Europe/Berlin refers to /two/
> points in time, thus it /is/ ambiguous. If you use disambiguating
> "time zones" (MEZ vs MESZ in this case) you can resolve that.
>

I think the confusion relates to context interpretation. If you see
@Europe/Berlin in isolation, then it is ambiguous as it can refer to two
different time zone definitions (standard v daylight savings).  However,
if you consider it in conjunction with a date and time, as in 2023-03-23
02:30 @Europe/Berlin, then it isn't ambiguous - in that case, it really
just says 'Lookup the time zone offset in the databse for Berlin as of
that date and time. Now that could change - for example, the German
government might make a temporary or permanent change that would change
the offset from UTC for that date+time the day after I look at it (or
export it). 

Personally, I cannot see the use case of including both a fully
qualitifed time zone (as in @Europe/Berlin) and an offset, but I also
don't know all possible use cases - there could well be use cases where
you want/need to record both the location time zone as well as the
offset at the point when you recorded the timestamp. This is why I'm in
favor of a flexible and extensible syntax and strongly disagree with any
argument which says we don't need UTC or we don't need abbreviated TZ
names or .

Re: lesser-than as open paren

[Tim Cross]

Andreas Röhler  writes:

> Hi,
>
> when taking notes in plain org-mode, run into trouble for instance with this
> scala-snippet:
>
>   def scalaFiles =
>     for {
>     file <- filesHere
>     if file.getName.endsWith(".scala")
>   } yield file
>
> With cursor on lesser-than sign, get a type-mismatch.
>
> The culprit resides in org.el:
>
>   (modify-syntax-entry ?< "(>")
>   (modify-syntax-entry ?> ")<")
>
> Maybe use this modification only when a special case requires it?
>

when you say "when taking notes in plain org-mode" do you mean the above
code snippet is just in your org file and not inside either a source or
example block?

If so, I think that would be expected behaviour. Code snippets really
need to be either in an example block or preferably a source code block.

Re: [POLL] Proposed syntax for timestamps with time zone info (was: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda)

[Tim Cross]

Ihor Radchenko  writes:

> Greg Minshall  writes:
>
>> just a thought/reminder.  there are "semantics" and "encoding".  a spec
>> like ISO-8601 specifies both.  the important thing for org-mode is to
>> use an encoding that
>>
>> 1. is easily parsable/understandable by the mere mortal
>>
>> 2. allows expression of all the semantics of the underlying spec/specs
>>(be that ISO-8601, this new IETF spec, the Library of Congress spec,
>>etc.)
>>
>> 3. and, importantly, is designed to *try* to follow updates to the
>>underlying spec/specs (which will inevitably happen)
>
> I agree with these three points.
>
> Following the previous discussion and the various links provided, I have
> reviewed the main discussed timestamp standards and would like to
> propose the new Org timestamp syntax that will allow specifying time
> zone information.
>
> I will not follow the standards fully because the available standards
> are not designed to be easily understood by humans. I will also omit
> the ideas from the standards that are unrelated to time stamps (but
> still outline them below, and keep them in mind for
> forward-compatibility). I will, however, try to use the syntax close to
> the standards where possible.
>
> 1. https://datatracker.ietf.org/doc/draft-ietf-sedate-datetime-extended/
>proposal is extending ISO8601/RFC3339 with time zone information. In
>addition to UTC offset defined in ISO8601, it allows specifying the
>time zone identifier and auxiliary information.
>
>Examples:
>
>2022-07-08T02:14:07+02:00[Europe/Paris]
>(both offset, and time zone are specified)
>Note that we cannot use "[]" symbols because they are incompatible
>with current timestamp syntax that must not contain closing "]>"
>inside the timestamp.
>
>1996-12-19T16:39:57-08:00[America/Los_Angeles][u-ca=hebrew]
>(preferred calendar is specified in addition to time zone)
>Note: calendar spec is out of scope of time zone discussion - if we
>decide to add it in future, we can simply add new parts to
>timestamps, just like repeater interval and warning period.
>
>Further, the draft proposes an idea, which have also been discussed
>in this thread: making use of redundant UTC offset + time zone
>information to detect possible unexpected changes in time zone rules:
>
>2022-07-08T00:14:07+00:00[!Europe/London]
>("!" identifies that +00:00 offset, if not consistent with
>Europe/London at the timestamp time, must be signalled to user or
>generally not ignored)
>
> 2. https://www.loc.gov/standards/datetime/ does not contain any new
>ideas relevant to time zones, although it has many other ideas wrt
>approximate/incomplete timestamps. I will outline them later, but
>they do not directly affect the proposed new Org timestamp syntax.
>
> |---|
> | The proposed new timestamp syntax |
> |---|
>
> I propose two forms of time zone information in Org timestamps
>
> 1. Timestamp with explicit UTC offset
>
>-MM-DD [optional day name] HH:MM[^ \]>]*?[+-−]HH[MM]?
>-MM-DD [optional day name] HH:MM[^ \]>]*?Z[ \]>]
>
>"-" is latin "HYPHEN-MINUS" (code 0x2D)
>"−" is unicode "MINUS SIGN" (code 0x2212), as prescribed by ISO8601
>we will not actually use it when generating timestamps, but allow it
>to keep some compatibility with ISO standard.
>
>"Z" in the second format refers to "Zulu" time (another name for UTC)
>and must be either the last character in the timestamp or the
>last character before space. Not sure if many users are familiar with
>"Z" convention, but it is (1) in ISO; (2) succinct for users who are
>familiar with it. We will prefer +00 number offset in auto-generated
>timestamps.
>
>Examples:
>2022-11-12 12:00+02 # 12:00 UTC+2
>2022-11-12 14:00+0230 # 14:00 UTC+2:30
>2022-11-12 14:00-0230 # 14:00 UTC-2:30
>2022-11-12 14:00Z # 14:00 UTC
>
>The offset is a subset of what is defined by ISO8601.
>
>Note that unlike ISO8601, ":" is not allowed in the offset specifier.
>This is to disambiguate with the time intervals in Org timestamps:
>[2022-11-12 8:00-9:00] in Org is a time range from 8am to 9am.
>
>For time ranges, we will only allow a single offset and time zone
>specifier for both start and end times:
>[2022-11-12 8:00-9:00+08]
>If different time zones are necessary to specify the start/end times,
>users can still use full timestamp range syntax
>[2022-11-12 8:00+03]--[2022-11-12 22:00+08]
>
>The format is also forward-compatible with extending Org timestamps
>for second/sub-second precision: 2022-11-12 14:00:05.0012+0230 will
>remain valid if we decide to adopt such format.
>
> 2. Timestamp with time zone specifier
>
>-MM-DD [optional day name] HH:MM[^ ]* @[!]?<[^ \]>]>
>
>For now, time zone name will only be processed when it follows TZ

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Jean Louis  writes:

> * Tim Cross  [2023-01-29 23:38]:
>> Saying that an offset is a fixed value is very different from saying
>> that a time zone has a fixed offset. I think this is where your
>> confusion is coming from. 
>
> I said neither of those. I never said that UTC offset is fixed. I am
> trying to give you references that you understand what people agreed
> on this planet.
>
> I never said that time zone has fixed offset, some time zones have it
> fixed, some not, as UTC offset changes for daylight savings and
> political reasons.

Jean,

you have a very irritating habit of changing the topic of the discussion
in order to push whatever you feel you want to argue about. My response
to you had nothing to do with all the irrelevant points you insist on
repeating despite numerous attempts by many to explain why what you are
sending is adding little value.

I was simply responding to your response to Sterling's glossary of
definitions where you argue against defining offset (the term) as a
fixed value.

I will not be engaging with you further.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Jean Louis  writes:

> * Tim Cross  [2023-01-28 00:15]:
>> >
>> >> What kinds of representations would a calendar system capable of
>> >> handling timezones require?
>> >> 
>> >> • Instant (fixed)
>> >>   • This is referring to an unambiguous moment in time
>> >>   • e.g., 2007-02-03T05:00:00.000Z
>> >> • Offset (fixed)
>> >>   • This captures the idea of "when did it happen for the person who
>> >> made the observation"
>> >>   • e.g., 2007-02-03T04:00:00.000+01:00
>> >
>> > Offset is not that fixed, maybe from viewpoint of storage as maybe it
>> > is considered fixed in it's representation, but you have to keep in
>> > mind that time offset by it's definition is changing itself, suddenly,
>> > depending of daylight saving and time zone.
>> >
>> 
>> I think your misinterpreting the intent here. If you specify a timestamp
>> with offset, it is fixed.
>
> That is what you say. And I am pointing out to international standard
> references.
>
> If you use offset as "fixed" it means such use would not be by
> standard, and you would confusing users and programmers who are using
> standard for calculations in programs.
>
>> It does not change with daylight savings or any other change in
>> rules for a time zone. It does not even specify a time zone.
>
> And while and before making that decision, did you review the standard
> that time zone offset is influenced and changed by daylight savings?
>
> It does not specify time zone. But it is derived from time zone, and
> is not same from time zone.
>
> Are you aware that time zone offset could have "skipped time" or
> "added time" due to daylight savings?
>
> That implies that by using time offset, you would forget daylight
> savings which are international standard, and make calculations wrong,
> because you started applying own standard in Org.
>

I think your still misunderstanding what is meant by offset.

Yes, a timezone is defined by the offset it has from UTC
Yes, a location time zone may change due to various reasons, such as
daylight savings time, which also means the offset for that timezone
changes. However, it is the time zone definition which has
changed. THink of it as a time zone with a new offset rather than a time
zone with a chagned offset. 

When you specify a date+time wiht an explicit offset, that offset is
fixed. That date+time is fixed. It will not change when daylight davings
comes in or goes out because it isn't a time zone. It is only an offset
and has no location reference and therefore no time zone.

Saying that an offset is a fixed value is very different from saying
that a time zone has a fixed offset. I think this is where your
confusion is coming from. 

Re: UTC or not UTC for timestamps in the past ([FEATURE REQUEST] Timezone support in org-mode)

[Tim Cross]

"Thomas S. Dye"  writes:

> No, I don't think you've missed anything significant.  Thanks very much for 
> your patience
> during a discussion that was interesting for me.  I learned quite a bit from 
> you and the
> other contributors to the thread and look forward now to learning how Org 
> mode evolves to
> handle events and occurrences.
>
> Let me know if you have questions.
>

Agreed. I also think this latest discussion is extremely valuable as
coming up with some clear terminology which can be used to reference
different use cases for timestamps is very likely going to make
documentation and exmaples/tutorials much cleaner and could help define
a more intuitive interface which helps users select the right option in
the right circumstance.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Jean Louis  writes:

> * Sterling Hooten  [2023-01-27 09:06]:
>>   Offset
>> Constant duration difference between times of two time scales
>> (ISO). i.e., a quantity to combine with a time scale to produce
>> a wall time. e.g., Nepal uses a +5:45 offset from the UTC time
>> scale.
>
> I would be careful calling it constant as time offset is changing
> depending of daylight saving time. It is not constant.
>
> Time offset time stamp may be derived from time zone time. I am sure
> about this.
>
> Time zone time stamp cannot be unambiguously derived from time
> offset. I am mostly sure about this.
>
>> What kinds of representations would a calendar system capable of
>> handling timezones require?
>> 
>> • Instant (fixed)
>>   • This is referring to an unambiguous moment in time
>>   • e.g., 2007-02-03T05:00:00.000Z
>> • Offset (fixed)
>>   • This captures the idea of "when did it happen for the person who
>> made the observation"
>>   • e.g., 2007-02-03T04:00:00.000+01:00
>
> Offset is not that fixed, maybe from viewpoint of storage as maybe it
> is considered fixed in it's representation, but you have to keep in
> mind that time offset by it's definition is changing itself, suddenly,
> depending of daylight saving and time zone.
>

I think your misinterpreting the intent here. If you specify a timestamp
with offset, it is fixed. It does not change with daylight savings or
any other change in rules for a time zone. It does not even specify a
time zone. While it is true that a specific location may have time zone
changes or that the offset from UTC might change as a result of daylight
savings, an offset specified in a times stamp does not change and is
fixed.  This is one of the limitaiton with using offset.

I also think it is a mistake (which I've seen others suggest in this
thread) to equate an offset as being an abbreviation for a time
zone. For example, some have suggested things like +1 being the same
as AEST and both being abbreviations for Australia/Sydney outside of
daylight savings periods. I think that is incorrect. +1000 is a fixed
offset from UTC and while it may be the same offset used in a time zone
abbreviation like AEST or in a full time zone specification like
Australia/Sydney, it is not a time zone specification, only an offset
for a specific point in time.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> First of all, thanks for the detailed suggestion!
> I will need more time to look through the provided links and think about
> the ideas.
>
> I will provide one important consideration you missed in the below comment.
>
> Sterling Hooten  writes:
>
>> What format and syntax should Org use?
>>
>> A heretical suggestion: We should abandon the day of week abbreviation
>> and use a new format.
>> ...
>> [2023-01-25 Wed 13:57]–[2023-01-26 Thu 13:57]
>>
>> it would be:
>>
>> [2023-01-25T16:57:42Z/2023-01-26T16:57:42Z].
>
> Following ISO and other standards is indeed a reasonable idea. However,
> the standards are not necessarily designed for human consumption.
> In contrast, Org mode is designed to be read by humans as well, even
> without Emacs - just as plain text.
>
> Design for human consumption is one of the reasons we do provide the
> redundant information like week day (I personally did find it extremely
> useful on multiple occasions) and do use spaces, deviating from ISO. The
> above ISO example is barely readable by humans. Another example from
> wiki page of ISO 8601 is even worse: R5/2008-03-01T13:00:00Z/P1Y2M10DT2H30M
>
> And we need to deviate from ISO 8601 anyway. At least, because it does
> not define time zones, only absolute UTC offsets. So, the ability to
> conform with the existing formats remains questionable.


I strongly agree with Ihor here. We want our timestamps to be easily
read and understood by users. I have also found the redundant day of the
week information very useful.

While we could argue that with overlays or similar, you could get the
best of both worlds i.e. a storage format which is easy for functions to
parse and a display format which is easy for humans to parse, but that
would also work only when you view your org files within org mode. One
of the benefits of org mode is its plain text nature and that you can
read most org mode files 'raw' and they are quite easy to understand. 

Re: [POLL] Use compat.el in Org?

[Tim Cross]

Bastien Guerry  writes:

> Hi Ihor,
>
> Ihor Radchenko  writes:
>
>> I have recently been contacted by the current compat.el maintainer
>> asking if we are willing to adapt compat.el in Org.
>
> Very nice!
>
>> WDYT?
>
> As long as we keep our promise in terms of backward compatibility with
> older Emacs versions, I'm all for it.

I would agree. I would also add that even with the use of this package,
I don't think we should use it to increase the number of versions we
support as support is not as simple as dropping in a compatibility
library. These libraries come with a cost. Often, compatibility code
does not perform as well and/or is much more complicated and more likely
to have bugs. The more a version of emacs needs to rely on this library
to run org-mode, the higher the likelihood performance will be degraded
or unexpected new bugs are found.

So, use the library, but keep the existing policy of officially
supporting only the previous two major releases. If org does work with
even older versions, that is great, but not supported.

Re: Supporting non-free SQL clients in ob-sql (was: [PATCH] ob-sql: Add support for Athena)

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> 3. There is no requirement to install non-free software to use
>> ob-sql.el. The software is fully functional using a free RDMS like
>> postgres.
>
> Yes, but there is requirement to install in order to use ob-sql.el
> __with :engine set to non-free option__.
>
> So, I can envision that someone who decided to use ob-sql.el and
> considering between free and non-free engine may prefer non-free one. Of
> course, it is not very strong argument, but the boundaries are fuzzy in
> this area.
>

in the same way someone could choose to run emacs on MS Windows. This
doesn't mean emacs encourages people to use MS Windows. Rather it means
that people who are restricted to MS Windows can at least use a free
editor on that platform. In a similar manner, people who are restricted
to working with a non-free database can access it using free software. I
think this is particularly relevant given the growth in large databases
where users are unable to use a free database or run it locally simply
because of the size of the data and the resource requirements and
administrative complexity involved. 

>> For maintenance reasons and to add session support, I would suggest that
>> using sql.el instead of re-inventing this wheel would be a better
>> outcome. I've used sql.el for years and it works extremely well and I
>> don't htink it would be too hard to integrate into ob-sql.
>
> Sure. That's what we usually do - just use whatever REPL is available
> for a given ob-* language. Just a question of someone sending a patch.

Re: Supporting non-free SQL clients in ob-sql (was: [PATCH] ob-sql: Add support for Athena)

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> to be very clear, ob-sql is not adding any NEW interface to any external
>> program. It is just using the Emacs built-in SQL library (Elisp), which
>> has been part of Emacs for a long time (I was using it in late 90s to
>> work with Oracle RDMS). It is this library that provides the 'support'
>> for things like Oracle's RDMS. If you want more specific information,
>> ask on emacs-devel. Org mode is just using this built-in library.
>
> This is wrong.
> `org-babel-execute:sql' directly calls the CLI executable via `process-file'.
>
> That said, the discussion about sql.el is also relevant wrt non-free SQL
> client support. It will be helpful to clarify what is morally acceptable
> for all the scenarios, not just for Org's use-case.
>
>> The Oracle database is simply a relational database management system in
>> the same way as Postgres, MySQL, Ingris, MS-Sql server etc. All of these
>> have a CLI client and support connections via JDBC. The sql.el library
>> provides specialised comint based interfaces which run the CLI to
>> communicate with the RDMS. The closest of your 3 choices is b, as the
>> build-in Emacs sql library executes the RDMS CLI in a sub-process comint
>> buffer.
>
> Org does not use this comint interface yet. We may in future though, for
> sessions:
>
> (defun org-babel-prep-session:sql (_session _params)
>   "Raise an error because Sql sessions aren't implemented."
>   (error "SQL sessions not yet implemented"))

OK, thanks the for correction. I was confused because the set of
supported RDBMS is the same as those supported by sql.el and the
connection credentials uses the same variable name
i.e. sql-connection-alist.

However, I gues the point remains, sql.el and ob-sql.el support a number
of non-free RDMS and I think this is fine given that

1. There is no encouragement, implicit or explicit, to use a non-free
database
2. Provided support provides a way to interact with these non-free RDMS using
free software.
3. There is no requirement to install non-free software to use
ob-sql.el. The software is fully functional using a free RDMS like
postgres.

For maintenance reasons and to add session support, I would suggest that
using sql.el instead of re-inventing this wheel would be a better
outcome. I've used sql.el for years and it works extremely well and I
don't htink it would be too hard to integrate into ob-sql.

Re: Supporting non-free SQL clients in ob-sql (was: [PATCH] ob-sql: Add support for Athena)

[Tim Cross]

Richard Stallman  writes:

> [[[ To any NSA and FBI agents reading my email: please consider]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Would someone please tell me more concretely what kind of "support"
>   > > this is?
>
>   > You can find interactive support in `sql' library, functions such as:
>
>   > M-x sql-oracle 
>
>   > which supports proprietary Oracle Database:
>
> This raises two questions.
>
> 1. For this purpose, what kind of thing is "the Oracle Database"?
> a. A library to link with?
> b. A program to run in a subprocess?
> c. A server running SaaSS?
>

None of the above!

Richard,

to be very clear, ob-sql is not adding any NEW interface to any external
program. It is just using the Emacs built-in SQL library (Elisp), which
has been part of Emacs for a long time (I was using it in late 90s to
work with Oracle RDMS). It is this library that provides the 'support'
for things like Oracle's RDMS. If you want more specific information,
ask on emacs-devel. Org mode is just using this built-in library.

The Oracle database is simply a relational database management system in
the same way as Postgres, MySQL, Ingris, MS-Sql server etc. All of these
have a CLI client and support connections via JDBC. The sql.el library
provides specialised comint based interfaces which run the CLI to
communicate with the RDMS. The closest of your 3 choices is b, as the
build-in Emacs sql library executes the RDMS CLI in a sub-process comint
buffer.

Re: UTC or not UTC for timestamps in the past ([FEATURE REQUEST] Timezone support in org-mode)

[Tim Cross]

Max Nikulin  writes:

> On 22/01/2023 14:48, Tim Cross wrote:
>> Timestamp for a log
>> record I would probably want  or one of the
>> variants because the most common way I use those types of timestamp is
>> in diagnosing problems and comparing revords from various locations
>> where I don't care about the local time where the record was generated,
>> but with when it occurred in relation to other log recores.
>
> I agree that it is more convenient to have uniform offset in such case, 
> especially in the
> case of multiple files having their specific timezone. During trips I do not 
> expect so
> much changes of timezone to make comparison of timestamps significantly 
> harder.
>
>> I would argue that all depends on how you use the information. My org
>> files are consumed by me (reading) and by scripts, elisp and other
>> programs.
>
> For scripts given offset should not be an issue at all. And it is up to you 
> if you prefer
> to have UTC instead of local time offset in you files.
>
>> My preference is for a timestamp syntax which lets the user select the
>> format they want and not attempt to restrict it more than that.
>
> I do not mind to provide a user option for preferred storage format used for 
> clock entries
> and similar stuff.
>
>> Provided
>> you can specify timestamp with and without TZ information and you
>> support full and abbreviated time zone names as well as UTC, I don't
>> think it mattters - let the user choose what suits them best.
>
> Concerning time zone abbreviations, I would discourage them as much as 
> possible for
> storage format. They may be added to overlay though. Perhaps US residents 
> would be unhappy
> by such decision, but there are enough examples of the same abbreviation for 
> completely
> unrelated locations. Abbreviation may be useful in addition to timezone ID to 
> disambiguate
> local time close to a backward time jump.
>
> Tim, in your last messages I do not see statements causing my objections any 
> more. It
> seems we came to agreement: flexible enough storage format and configurable 
> display format
> for overlays. Have I forgot anything?

No, I think you have covered everything.

I think most of your remaining concerns can be adequately covered via
some updated documentation and with some luck, people will add some use
case workflows to worg showing how using different storage formats and
display overlays can address various scenarios.  

Re: Supporting non-free SQL clients in ob-sql (was: [PATCH] ob-sql: Add support for Athena)

[Tim Cross]

I just wanted to provide some additional information which RMS may find
informative.

Under the hood, ob-sql.el is using the built-in Emacs sql.el library. It
is this library which provides the 'support' for various SQL database
engines. That support has been part of the sql library since it was
added to Emacs in about 1998.

The 'support' is essentially specialised comint based interfaces tweaked
to work with the various SQL database engine command line clients such
as psql for Postgres and sqlplus for Oracle. This involves codes to use
the comint buffer to send commands/regions to the SQL client and read
back the results and run interactive 'repl' like sessions with the
client.

No additional software is installed by either ob-sql.el or sql.el - it
is all just elisp. Any additional software, such as the database client,
must be installed separately by the user (clients on remote machines can
also be used via ssh).

There are no jar files or any other additional bits of software
installed.

There is also some additional support in the form of font-locking which
includes support for different SQL variants i.e. in addition to ANSI
SQL, you can also specify Postgres SQL, Oracle SQL, MS-SQL etc to get
font-locking which supports some of the keyword differences between the
different SQL engines.

Personally, I don't believe any of this contravenes FSF
guidelines. Neither sql.el or ob-sql.el are encouraging use of any
specific database engine. What these libraries do is in fact allow
those who do want to use free software and avenue for interacting with
these systems using free software rather than on-free (similar to Emacs'
WIndows and Mac support).

The software as a service issue is far more difficult to assess when it
comes to databases as the most common architecture with databases is to
have the database on a separate server which is often not local and
where frequently, the user has little or no control over the
software. This has become even more common in recent years due to the
growth in big data and the problem that most individuals and small
groups cannot afford the costs or handle the administrative complexity
associated with hosting databases with large data sets and storage
requirements.

Ihor Radchenko  writes:

> Richard Stallman  writes:
>
>>   > > Because we already support Orcale, SAP Hana, MSSql and Vertico for 
>> example.
>>
>> Would someone please tell me more concretely what kind of "support"
>> this is? Which GNU package supports them, and how?  
>
> [Jean Luis mentioned M-x sql-* commands in another message. They may
> also be worth discussing, but are not what I meant in the original
> message]
>
> ob-sql is Org mode's feature (part of Emacs) allowing users to execute
> SQL code inline inside Org files:
>
> #+header: :engine sqlite
> #+header: :database ~/.local/share/qutebrowser/history.sqlite
>
> #+begin_src sql
>   SELECT URL FROM History ORDER BY "URL" DESC LIMIT 5 
> #+end_src
>
> #+RESULTS:
> | qute://help/commands.html |
> | qute://help/commands.html |
> | qute://help/  |
> | qute://help   |
> | qute://   |
>
> Note the :engine argument. It determines which CLI backend is being
> called by Org mode to query the database.
>
> ;; Engines supported:
> ;; - mysql
> ;; - dbi
> ;; - mssql
> ;; - sqsh
> ;; - postgresql (postgres)
> ;; - oracle
> ;; - vertica
> ;; - saphana
>
> Some of these engines are free software and we have no issue supporting
> them. Some are not.
>
>> ... Also, what kind
>> of things are those?  Are they nonfree programs, or are they SaaSS?
>> Some of them I recognize as nonfree programs, but some names I don't
>> recognize at all.
>
> At least, mssql is non-free with no source code available. Same for
> oracle, vertica, and saphana.
>
> I am not sure about SaaSS - even postgresql (free software) may be used
> as a service provider by running it on server the user does not control.
> Probably, I do not fully understand how SaaSS is defined (I did read
> https://www.gnu.org/philosophy/who-does-that-server-really-serve.html)
>
> Earlier in the thread, https://github.com/dbcli/athenacli was proposed
> as another CLI to support. Athenacli itself is distributed under BSD-3 -
> free license. However, the CLI is just an interface to Amazon Athena -
> server-side database service.
>
> From my limited understanding, the main purpose of Amazon Athena is
> querying "big data" databases for further analysis, which does not
> constitute SaaSS (AFAIU) - users may not own the "big data" and may not
> have enough resources on their own computers to run the SQL queries that
> utilize machine learning models.
>
>> There are situations where it is acceptable for a GNU package to
>> support its use together with certain nonfree programs.  Mainly when
>> the nonfree program so well known that this support mainly encourage
>> people who already use that nonfree program to start using the GNU
>> package with it, and is unlikely to do the 

Re: UTC or not UTC for timestamps in the past ([FEATURE REQUEST] Timezone support in org-mode)

[Tim Cross]

Max Nikulin  writes:

> On 22/01/2023 04:29, Tim Cross wrote:
>> Max Nikulin writes:
>>> - UTC is a recommendation for planning when participants are scattered over 
>>> multiple
>>>   timezones.
>>> - You admit that some timestamps in your files may be specified as time 
>>> zone identifier +
>>>   local time relative to this zone.
>>> - In both cases you may configure overlays to use local timezone or another 
>>> one whatever
>>>   you currently find convenient.
>>> - Time chooser offers several time zone options.
>> That seems to be in-line with what I was arguing, so yes, would agree.
>
> Great. At this stage my goal is to convince people that local time and UTC 
> for future
> timestamps are not enough for real life use cases. Arbitrary timezone may be 
> crucial to
> arrive at proper time despite it matters in rare cases. My concern is mostly 
> storage
> format, timestamp syntax in Org files.
>
> On 21/01/2023 06:38, Tim Cross wrote:
>>>> I would also argue UTC is good for 'traditional' timestamps which record
>>>> a specific point in time and for situations where you want accurate
>>>> calculations of time durations.
>
> Let's consider the following timestamp
>
> [2023-01-22 Sun 08:29@+1100]
>
> "@" is unimportant here, I take it from Ihor's examples. This timestamp is 
> from the
> "Date:" header of your message. It is not UTC, but in my opinion it is 
> equally precise
> (disregarding seconds) to a UTC timestamp.
>
> Would you prefer to have timestamps in your files in such form or as
>
> [2023-01-21 Sat 21:29Z] ([2023-01-21 Sat 21:29@UTC], [2023-01-21 Sat 
> 21:29@+00:00], etc)?
>
> Maybe [@1674336588] (seconds since epoch)?
>

It really depends on what the timestamp is for as to what my preference
would be.  For example, timestamp for an email message is likely best as
your initial example or date with remote senders TZ. Timestamp for a log
record I would probably want  or one of the
variants because the most common way I use those types of timestamp is
in diagnosing problems and comparing revords from various locations
where I don't care about the local time where the record was generated,
but with when it occurred in relation to other log recores.  

> I consider storage format as a part of Org UI, so I believe that [2023-01-22 
> Sun
> 08:29@+1100] with offset of the usual time zone is more convenient than 
> [2023-01-21 Sat
> 21:29@+]. Overlays may present your local time in both cases, but raw 
> value with usual
> offset is easier to comprehend.
>

I would argue that all depends on how you use the information. My org
files are consumed by me (reading) and by scripts, elisp and other
programs. 

> When a file is shared by a group of people spread across the globe, they are 
> free to use
> UTC or another fixed timezone or do not care at all concerning particular 
> offset, it just
> should present.
>

Guess I agree, but this is such a rare/small use case, I really don't
consider it terribly relevant. While I do share data relating to
projects I work on with others, I am the only one who uses org mode and
I suspect I'm the only one who uses Emacs. Sharing org mode files simply
isn't a use case I need to worry about and sharing timestamp data from
those files is rare as well - if I do need to, they will likely be
processed into some other more standard format anyway. 

> Postgres has a reason to insist on UTC since timestamps are stored in binary 
> format as
> microseconds since epoch. It is important for performance and there is no 
> room to specify
> offset. Org has to parse timestamps from strings anyway, so it is better to 
> use format
> more convenient for humans.
>

Again, depends on the use case and how you use the data. 

> A side note. In my opinion, *by default* timestamps should be created in 
> local time with
> no offset or timezone. There are should be some preferences to enable 
> absolute timestamps
> and a function to fix offsets or timezone identifiers for existing timestamp 
> when the user
> realizes that they become necessary (e.g. before a trip).
>
> So I do not see any advantages of UTC in comparison to time offset specific 
> to particular
> time zone, usually user's local one. My vote is [2023-01-22 Sun 08:29@+1100] 
> and not
> [2023-01-21 Sat 21:29@UTC] (of course, only in cases when identifiers like
> Australia/Sydney do not matter) with a configuration option with timezone 
> used fix offsets
> in stored timestamps.

My preference is for a timestamp syntax which lets the user select the
format they want and not attempt to restrict it more than that. Provided
you can specify timestamp with and wit

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 21/01/2023 06:38, Tim Cross wrote:
>> - Use UTC for meetings which are not face-to-face and which involve
>>people form different time zones.
>
> I agree with you that it should considered as first option by whose who are 
> planning an
> event. They still may prefer to choose timezone of particular location 
> because it is mixed
> meeting with attenders of both types: face to face and online, or just 
> because most of
> on-line participants are expected from particular area.
>
> However timezone may be out of your control if you receive invitation to join 
> on-line to
> some meeting. Attempt to convert it to UTC may lead to problems. So the 
> reason to choose
> UTC or specific timezone for particular timestamp is not really technical 
> one. It is just
> decision of some people.
>
> Do you have any objection to the following statements:
> - UTC is a recommendation for planning when participants are scattered over 
> multiple
>  timezones.
> - You admit that some timestamps in your files may be specified as time zone 
> identifier +
>  local time relative to this zone.
> - In both cases you may configure overlays to use local timezone or another 
> one whatever
>  you currently find convenient.
> - Time chooser offers several time zone options.
>

That seems to be in-line with what I was arguing, so yes, would agree.



>> I would also argue UTC is good for 'traditional' timestamps which record
>> a specific point in time and for situations where you want accurate
>> calculations of time durations.
>
> Mostly agree, so I prefer to postpone discussion of details till confirming 
> agreement in
> respect to future timestamps.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

"Thomas S. Dye"  writes:

> Aloha Max,
>
> Max Nikulin  writes:
>
>> On 20/01/2023 23:09, Thomas S. Dye wrote:
>>> Max Nikulin writes:
>>> Now, if Amsterdam's timezone arbitrarily changes its relation to UTC before 
>>> the
>>> conference takes place,
>>> then everyone who participates in the conference must notice this (or miss 
>>> the
>>> start of the conference).
>>
>> My point is that if timestamp is stored in UTC than it becomes incorrect in 
>> the
>> case of time offset change. If such timestamp is stored as local time + time
>> zone identifier then application presenting the timestamp to users will show
>> correct time as soon as zoneinfo data is updated.
>>
>>> A virtual conference is organized by someone in Amsterdam, who sets a start
>>> time corresponding to 9AM in Amsterdam at a date some years in the future 
>>> and
>>> invites participants from all over the world.  Now, if Amsterdam's timezone
>>> arbitrarily changes its relation to UTC before the conference takes place,
>>> then must everyone notice this?  Or, should Org, from the time the 
>>> arbitrary change is
>>> made public, simply adjust the conference time for all the
>>> participants in the Amsterdam timezone?
>>
>> Both variants are possible and a planning application should support them. 
>> It is
>> matter of negotiation between the committee and the participants. Timestamp
>> should be stored in UTC only in one case.
>
> Agreed.  So, IIUC, there are three cases:
>
> 1) Occurrence, where the timestamp includes UTC;
> 2) Event relative to user, where the timestamp does not include UTC or a time 
> zone; and 3)
> Event not relative to user, where the timestamp includes the relevant time 
> zone.
>

I would argue case 2 is really just a specialisation of case 3 i.e. it
has an implicit time zone which is the user's local time zone. 

> For completeness, Case 3 might benefit from a procedure to change the 
> relevant time zone.
> For instance, when the boss has scheduled time for you at 9:00 AM her time, 
> but doesn't
> know where she will be on that day.
>

If it is to be a fact-to-face meeting (event), implying it therefore has
a location, you would assume your local time zone. If not face-to-face
(occurrence), it needs a UTC offset in order to ensure same point in
time for all participants. The boss either needs to specify the time
zone they are referencing the 9am to or the user will need to make a
guess, which may or may not be correct.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 20/01/2023 15:17, Tim Cross wrote:
>> So far, nobody has shown any reason why using UTC to distinguish the
>> case where the times need to be adjusted and local tz when they don't
>> won't work a a  mechanism that can be used to allow org to handle things
>> better than it does now.
>
> Let's try to move in small steps. UTC as storage format.
>
> An issue with events scheduled as local time in some particular timezone (not 
> your current
> one) and stored as UTC timestamp when IANA tzdata is updated to use new 
> timezone offset:
>
> https://codeblog.jonskeet.uk/2019/03/27/storing-utc-is-not-a-silver-bullet/
> Storing UTC is not a silver bullet
>
> Do you think it should be ignored? I faced such issue.

I now think I can see where you have become confused.

I am NOT arguing that all timestamps should be stored in UTC.

I am only saying that when you have a meeting which is not face to face
and involves people from various time zones, the date+time for that
meeting should be stored in UTC. This is the only way org will be able
to ensure the meeting time (which would be converted into local time for
the user) stays relative to the other participants after a daylight
savings transition.

The use cases here is completely different from the example outlined in
that long article.  Firstly, much of the complications arising in that
example are due to tryhing to coordinate dates and times for multiple
parties. Here we are only focusing on 1 party. Secondly, I am only
proposing UTC for meetings where there is no physical location for the
meeting. In the case where there is a physical location, that is a
fact-to-face meeting and it should be recorded in the time zone of the
location of that meeting. In most cases, that will be the user's local
time zone, so for most face=to=face meetings, no explicit time zone is
necessary as the local time zone will be assumed.

So in summary, my argument is

- Use UTC for meetings which are not face-to-face and which involve
  people form different time zones.

- Use the time zone of the location of a meeting when the meeting is
  face-to-face and has a physical locaiton.

- Provide an interface in org to make it easier for users to select the
  correct format (UTC or lcoal/meeting tz). Exactly the best way to do
  this is yet to be determined. It could be just a general solution
  which allows you to select the TZ when you call the functions with C-u
  or it could be something more sophisticated and specific to booking
  meetings which asks questions (i.e. type of record meeting wiard).

- It is assumed that in most cases, timestamps will b displayed to the
  user in their local time zone regardless of how they are actualy
  recorded int he org file.

I suspect where you have become confused is because you read my first
post in the thread where I suggested using UTC for meetings would be
sufficient. However, I revised that based on responses and then proposed
only using UTC for meetings which are not face-to-face and where
participants are from different time zones. It appears you have missed
those posts. 

I would also argue UTC is good for 'traditional' timestamps which record
a specific point in time and for situations where you want accurate
calculations of time durations. In fact, I would argue that UTC is a
good choice for a majority of time stamps, but acknowledge there are
some situations, notably when scheduling an event with a physical
location, where it is better to use the time zone of the location. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 20/01/2023 15:11, Tim Cross wrote:
>> Max Nikulin writes:
>> 
>>> Tim, I am trying to say that any meeting either face to face or on-line may 
>>> be associated
>>> with arbitrary primary timezone.
>> and what you are saying is helpful how? In what way does what you are
>> sayhing help address my use case?
>
> Tim, are you trying to convince me that for Org it is enough to have 
> timestamps either as
> local time <2023-02-20 15:00> or as UTC something like <2023-02-20 05:00Z> 
> and ability to
> specify arbitrary timezone instead of UTC is redundant?
>

No. I have never stated anything like that. 

> I believe that in the case of support of optional arbitrary timezone in Org 
> files there is
> no point of distinction between you cases when all participants meet face to 
> face
> (<2023-02-20 15:00> or <2023-02-20 15:00@Australia/Sydney>) or it is online 
> meeting
> scheduled as <2023-02-20 09:00@Etc/UTC>.

That is all I've been saying! For meetings where everyone is in same
time zone, just use either  OR > and for meetings where there are participants from
different time zones, use . That simple.

All that remains is to figure out the best interface to make it easy for
the user to have the correct timestamp for the correct meeting type. 

>
>>> UI might offer you to choose time in your timezone and to select another 
>>> timezone for
>>> storage. For your convenience it still may be presented to you in your 
>>> local timezone even
>>> it is stored in UTC or some other one.
>> and I have said as much. So, how exactly is your contribution assisting
>> with the use case I've outlined?
>
> I had a hope to assure you that unifying the cases you are considering as 
> distinct should
> not make user experience worse.
>
> Local event and UTC is meaningful for UI to enter or adjust timestamp where 
> such cases
> should be easier to select than arbitrary timezone. For parsing, generating 
> agenda, or
> export a more abstract model can be used.

I really don't understand your continued reference to 'arbitrary
timezone' or how it is relevant. I also am not clear what you mean by
'unifying the cases'. If you mean handling the two different cases in
the same UI, that is exactly what I've been suggesting. If you mean
using the same time zone for both cases, I don't see how that could
possibly work and if you mean something else, I don't understand.

My position is very simple and very specific. Either use no TZ info or
local TZ spec for meetings where all participants are in the same TZ and
ust UTC where you have participants from different TZs. Note that htis
says nothing about how the timestamp is displayed (I would argue for
user's local time using an overlay or similar, like we do for links and
markup) and it says nothing about how the other participants record the
meeting (it is specific to the org user) and it says nothing about other
users for timestamps - only in reference to scheduled events.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Daryl Manning  writes:

> Perhaps a leading question (leading to outrage =p ), but does anybody even 
> use those anymore? 
>
> I don't believe I've used them at all in 5 years of using org-mode (and if I 
> did it was most likely because of
> some arcane older feature which required them). 
>
> Daryl. 
>
> On Fri, Jan 20, 2023 at 5:57 PM Ihor Radchenko  wrote:
>
>  Ihor Radchenko  writes:
>
>  > <2023-01-14 Sat 18:22@Asia/Singapore>  (SGD and similar abbreviations are 
> often ambiguous)
>  > <2023-01-14 Sat 18:22+0800>
>  > <2023-01-14 Sat 18:22+08>
>  > <2023-01-14 Sat 18:22@+0800>
>  > <2023-01-14 Sat 18:22@+08>
>
>  One thing we all missed in the discussion is diary sexps.
>
>  In particular, "last Sunday of month" <%%(diary-float t 0)> may depend
>  on the time zone because the number of days in month may vary.
>
>  How can we approach this? What could the format to specify the time zone
>  for diary timestamps?
>
>  -- 
>  Ihor Radchenko // yantar92,
>  Org mode contributor,
>  Learn more about Org mode at .
>  Support Org development at ,
>  or support my work at 

If your speaking about diary sexp support, I don't use them, but I have
seen a number of threads within the last year regarding people wanting
assistance with getting them right and I get the impression for some use
cases, particularly for repeating events, diary seexp specification is
the easiest way to define them.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

"Thomas S. Dye"  writes:

> Aloha Max,
>
> Max Nikulin  writes:
>
>> On 20/01/2023 03:09, Tim Cross wrote:
>>> To reiterate for the last time, there are 2 clear and different use
>>> cases for timestamps associated with meetings.
>>> 1. A meeting timestamp for a meeting where all the participants are in
>>> the same time zone.
>> ...> 2. A meeting timestamp for a meeting where all the participants are in
>>> different time zones
>>
>> Tim, every scheduled meeting event is associated with some particular time 
>> zone,
>> often implicitly. So it is single use case.
>>
>> It is up to participants to negotiate which timezone is the primary one for 
>> each
>> event. It is matter of people communication, not technical issue.
>>
>> First Monday 15:00 is (almost) equally precise for
>> - Australia/Canberra having DST
>> - Australia/Darwin where currently no DST
>> - +1000 (the highest chance of improper use unfortunately)
>> - UTC
>>
>> Aside from time transition intervals, it is possible to take any of this 
>> variant
>> and to present time local for other participant across the globe.
>>
>> Storage timezone may differ from the current user preference which time zone
>> should be used to display timestamp or to export it.
>>
>> The problem arises when several participants believe that their timezones are
>> primary ones and they do not sync their schedules through a shared file or a 
>> DB
>> entry. An application can not solve this problem, but it might try to help to
>> identify it. Some efforts are necessary from user side. Timestamp should 
>> contain
>> list of timezones of other participants and cached local time. In such case 
>> an
>> application may warn users that local time values become inconsistent due to 
>> DST
>> transitions or due to update of time zone database. Unsure to what degree it
>> should be implemented in Org.
>>
>> So
>> - It is participants fault if a meeting is not associated with particular
>>  timezone
>> - Having a fair time handling library, it does not matter what time zone is 
>> used
>>  to schedule the meeting.
>
> Or, one can recognize that the meeting is an occurrence that requires 
> absolute time and a
> timestamp with UTC.  Each participant will resolve UTC to the local time 
> where they happen
> to be when the meeting takes place.  The user might toggle between UTC and 
> the local time
> translation using overlays, like pretty entities. This avoids the problem of 
> negotiating a
> timezone caused by forcing an occurrence to be represented as an event 
> relative to a
> fictitious space/time.
>

Exactly.

I am really confused by the constant reference to negotiating between
participants or finding a shared/agreed time zone etc. That is all
totally irrelevant in this use case as outlined. If we were talking
about booking meetings or communicating information about booked
meetings between participants or a different bit of software which
manages sceduling of meetings like one of those many web meeting booking
systems, then that would be a different matter. However, that isn't what
we are talking about. We are talking about org mode. All I am talking
about here is being able to identify two different types of meetings -
ones which need to have times adjusted as a result of daylight savings
time transitions and ones which don't and what mechanism org could use
to distinguish the two. It is that simple. Your occurrence v event
terminology provides two different names which may help label the two
different use cases.

So far, nobody has shown any reason why using UTC to distinguish the
case where the times need to be adjusted and local tz when they don't
won't work a a  mechanism that can be used to allow org to handle things
better than it does now. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 20/01/2023 12:39, Tim Cross wrote:
>> No, I disagree with that statement. That is old thinking based when
>> meetings meant face to face meetings. Only meeting which have a specific
>> location can have a time zone and even then, it isn't really the
>> meetings time zone, but instead the time zone of the participants at the
>> meeting.
>
> Tim, I am trying to say that any meeting either face to face or on-line may 
> be associated
> with arbitrary primary timezone. Even when all participants are in Sydney 
> they may decide
> to fix time in Darwin. It is strange, but it is possible. UTC is just one of 
> time zones
> that may be convenient for on-line meeting despite no participant really use 
> it. Local
> timezone is usually preferred for purely face to face meetings. You are not 
> realizing that
> is decision since it is not verbalized. Consider timezone as something 
> unrelated to
> location but just a set of rules how time offset in respect to epoch evolves 
> in time.
>

and what you are saying is helpful how? In what way does what you are
sayhing help address my use case?




> UI might offer you to choose time in your timezone and to select another 
> timezone for
> storage. For your convenience it still may be presented to you in your local 
> timezone even
> it is stored in UTC or some other one.

and I have said as much. So, how exactly is your contribution assisting
with the use case I've outlined?

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 20/01/2023 03:09, Tim Cross wrote:
>> To reiterate for the last time, there are 2 clear and different use
>> cases for timestamps associated with meetings.
>> 1. A meeting timestamp for a meeting where all the participants are in
>> the same time zone.
> ...> 2. A meeting timestamp for a meeting where all the participants are in
>> different time zones
>
> Tim, every scheduled meeting event is associated with some particular time 
> zone, often
> implicitly. So it is single use case.
>

No, I disagree with that statement. That is old thinking based when
meetings meant face to face meetings. Only meeting which have a specific
location can have a time zone and even then, it isn't really the
meetings time zone, but instead the time zone of the participants at the
meeting.

Meetings without a specific location do not have time zones, implicit or
otherwise. If you have an on-line meeting with 5 people from 5 different
time zones, there is no time zone which takes precedence and cna be
thought of as the meeting time zone. You could decide to do that if you
wanted, but it doesn't achieve anything useful. 

> It is up to participants to negotiate which timezone is the primary one for 
> each event. It
> is matter of people communication, not technical issue.
>

The issue I'm talking about has nothing to do with the other
participants of the meeting. It is irrelevant to them when my time zone
changes due to daylight savings.

> First Monday 15:00 is (almost) equally precise for
> - Australia/Canberra having DST
> - Australia/Darwin where currently no DST
> - +1000 (the highest chance of improper use unfortunately)
> - UTC
>

That is a bad example and is wrong. Australia/Canberra and
Australia/Darwin are different timezones regardless. Darwin is +930 and
Canberra is +1000 (+1100 with DS). So Monday 15;00 in Darwin will be at
15:30 in Canberra outside DS time and 16:30 during DS time.

> Aside from time transition intervals, it is possible to take any of this 
> variant and to
> present time local for other participant across the globe.
>
> Storage timezone may differ from the current user preference which time zone 
> should be
> used to display timestamp or to export it.
>
> The problem arises when several participants believe that their timezones are 
> primary ones
> and they do not sync their schedules through a shared file or a DB entry. An 
> application
> can not solve this problem, but it might try to help to identify it. Some 
> efforts are
> necessary from user side. Timestamp should contain list of timezones of other 
> participants
> and cached local time. In such case an application may warn users that local 
> time values
> become inconsistent due to DST transitions or due to update of time zone 
> database. Unsure
> to what degree it should be implemented in Org.
>

and this is not the issue I'm trying to solve here. At no time did I
reference booking meetings or scheduling meetings with others. This has
nothing to do with eh use case I was outlining. 

> So
> - It is participants fault if a meeting is not associated with particular 
> timezone
> - Having a fair time handling library, it does not matter what time zone is 
> used to
>  schedule the meeting.

OK, I just give up.

I don't understand why my very simple point seems so hard for anyone to
grasp and why everyone seems so focused on the booking and scheduling of
meetings with outhers. This was never in the scope of the very simple
issue I want solved. 

All I want is for org to tell me when my meeting is. I don't care about
other people's time zones or when the meeting is for them, I don't care
who books the meeting and I don't care whether someone feels the meeting
has a time zone or not because it is all totally irrelevant for my use
case.

This is very very simple and doesn't need to be over thought.

I have two reoccurring meetings.

Meeting 1. All of the people for this meeting are in my time zone. When
daylight savings transitions occur, there is no impact. THis is how org
timestamps work now. Nothing is required.

Meeting 2. This meeting has 5 participants. They are all in different
time zones. When daylight savings time transitions occur in my timezone,
I need the time stamp to report an adjusted time based on the change
(either forward or back 1 hour). Currently, org cannot manage this and
my meeting time is out by one hour for 6 months of the year. 

How is this handled by org?

My suggested solution, which I feel is quite simple is to simply use a
timestamp with a UTC or Etc/UTC value for the time zone for meetings
with participants from different time zones and where the time of the
meeting must remain constant with respect to UTC. Assuming that once org
timestamps handling has been updated to display timestamps according to
the loc

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

"Thomas S. Dye"  writes:

> Aloha Tim,
>
> Tim Cross  writes:
>
>> "Thomas S. Dye"  writes:
>>
>>> Aloha Tim,
>>>
>>>> UTC is a time zone - just one where offset is +
>>>
>>> UTC is absolute time.  It lacks the spatial component that defines a time 
>>> zone.
>>>
>>
>> Really? I would have thought the prime meridian was the spacial
>> component for UTC? I thought the full long time zone name was Etc/UTC
>> and UTC as the abbreviation.
>>
>> Regardless, in all the libraries I've used, you can use Etc/UTC or UTC
>> in exactly the same way you would use something like Australia/Sydney or
>> AEST. So perhaps, from a pedantic standpoint, it is not a time zone, but
>> for all intent and purpose in this discussion, I feel that point is
>> irrelevant.
>
> Agreed.  It does seem irrelevant for time zone libraries.
>
> Nevertheless, from the Org perspective it might not be.  An occurrence, which 
> marks
> changes in the nature or relations of things at a time, requires absolute 
> time.  Meetings,
> which involve a change in relation among participants, are occurrences.  IMO, 
> this
> indicates Org should give occurrences a UTC timestamp, then translate that 
> for each of the
> participants using their local time zone.  The insane interval problems that 
> Ihor brought
> up are moot in absolute time.  A single timestamp serves a meeting regardless 
> of whether
> the participants are all in one time zone or spread around the globe.
>
> An occurrence contrasts with an event, which is tied to the user's 
> space/time.  Time here
> is relative to the user.  IMO, this means that Org should give events a 
> timestamp without
> reference to either absolute time or a particular time zone, like the one it 
> uses now.
>

Just checking if I understand. I think we are coming from the same
position and with the same conclusion.

In the situation where the meeting involves people from different time
zones, the time of the meeting as reported by org needs to be adjusted
after a daylight savings transition so that the time maintains the same
relative to UTC. i.e. meeting time reported in local time goes
forward/backward 1 hour depending on the daylight savings transition
(in/out). I guess this is what you call an occurrence? 

When all participants in a meeting are in the same time zone, you do not
want the time changed as the result of the daylight savings transition.
This is what you call an event?

So, using your terminology, what we now need is convenience functions
for setting an occurrence timestamp and an event timestamp. I'm not sure
if occurrence/event are the best terms, but I cannot think of better
ones. Just slightly concerned people will have trouble grasping the
difference and undersanding why some meetings are an occurrence while
others are an event. FOr the user, they are just meetings.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

"Thomas S. Dye"  writes:

> Aloha Tim,
>
>> UTC is a time zone - just one where offset is +
>
> UTC is absolute time.  It lacks the spatial component that defines a time 
> zone.
>

Really? I would have thought the prime meridian was the spacial
component for UTC? I thought the full long time zone name was Etc/UTC
and UTC as the abbreviation.

Regardless, in all the libraries I've used, you can use Etc/UTC or UTC
in exactly the same way you would use something like Australia/Sydney or
AEST. So perhaps, from a pedantic standpoint, it is not a time zone, but
for all intent and purpose in this discussion, I feel that point is
irrelevant.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Jean Louis  writes:

> * Tim Cross  [2023-01-19 10:48]:
>> You completely misunderstood the specific issue being discussed. You
>> clearly have not been following this specific point being discussed and
>> your long reply just confuses matters rather than helps.
>> 
>> This issue is in dealing with the meeting time when the local timezone
>> changes due to daylight savings time and the fact you have two different
>> requirements
>
> I do not use Org for time stamps. I am using PostgreSQL.

Then your input here is not terribly relevant IMO.


> My time
> stamps show correctly (hopefully) as I rely on the design of that
> software. I may be very wrong. Though as user I want simple thing, to
> record time, and that time has to be displayed in my time zone, and I
> want to have functions which will provide for example accounting
> statement in the time zone of the recipient in Washington, USA. As
> simple as that.
>

Completely irrelevant to the point being discussed. Yet again, you are
just pushing your beliefs and pet peeves without actually comprehending
what is being discussed. 

> There is no need for Org to deal with daylight savings, as UTC
> underlying functions are expected to deal with it. Time zone database,
> C library, I cannot know. But any error there shall go to system
> maker. Developing such complexity on Org level is not necessary.
>

Yet more indication you don't understand the issue. Nobody has suggested
that org does daylight savings calculation - in fact, the one constant
from all the issues raised in this thread is that all the time zone
calculations, conversion between time zones, calculation/conversion to
display format etc is handled by system libraries not org. 


> For Emacs it makes fun to have various calendars, but for
> international time, that has to be handled by system libraries.
>
>> 1. For meeting where all people are in the same timezone, a transition
>> in/out of daylight savings changes nothing. The meeting time stays the same
>> 
>> 2. For meetings wiht people from different time zones, when daylight
>> savings transition occurs, the timestamp needs to be changed.  Nothing
>> needs to happen for the people in other time zones - it isn't their
>> problem and their meeting time is not affected.
>
> Sure, but that is not calculation for higher level like Org, it is for
> lower level, like system library. Discussion shall be given to
> developers of GNU C library to solve calculations with daylight
> savings. If such functions do not exist, then they can be made.
>

You still missed the point. It isn't about the calculation - where that
happens is largely irrelevant and as has been stated numerous times, org
will use the built-in Emacs interface to system facilities for this.

The issue is far more fundamental. Display the agenda with correct
meeting times regardless of daylight savings transitions. As only
meeting with participants from different timezones are affected by such
transitions, we need a way to distinguish those timestamps from
timestamps for meetings with all participants in the same time zone.

>> Ihor['s suggested solution was to just use the TZ of the 'meeting', but
>> that is ambiguous. A meeting doesn't have a time zone and picking just
>> one of the recipients doens't help as now you just have the issues of
>> their daylight savings transitions etc.
>
> ☺️ A meeting can have time zone. My friend, that is exactly how we do
> meetings, I have even given examples from my life on meeting
> scheduling on this mailing list. 
>

Nobody said meetings cannot have time zones. Again, work on your
comprehension. It seems you skim read then jump to the wrong
conclusion.

A meeting does NOT have a time zone. Participants in the meeting have
time zones and it is possible that every participant in the meeting is
in a different time zone. The meeting itself has no time zone.



>
>> The 'solution' is to record this meeting in UTC tz. However, to make
>> this 'workable' for most people, the interface for managing timestamps
>> needs to make this easy.
>
> Then again you have to tell that it is "UTC", which means you are
> already specifying some time zone.

hence the bit where I said 

"However, to make this 'workable' for most people, the interface for
 managing timestamps needs to make this easy."

>
> You could tell that Org always thinks of UTC, but that again means UTC
> as mark, must be somewhere placed, all users must know that it is UTC,
> and again there is need for users to display time in their time zone.
>
> What do you achieve by that design? 
>

It achieves exactly the flexibility needed. As has been made clear many
many times in this thread, what we are talking about is the ability to
ad

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Jean Louis  writes:

> * Tim Cross  [2023-01-19 00:31]:
>> The problem is with meeting 2 and the assumption there is a definitive
>> timezone for the meeting.
>> 
>> Consider this scenario. I have a meeting with two other people. We are
>> all in different timezone. What is the timezone of the meeting?
>
> Org in this state can't handle such things.
>
> A person in any timezone shall be able to see that time in his local
> time zone if we speak of distant meetings, and in case of face to face
> meetings, that person shall have computer aid to show him the meeting
> time in any time zone that user is located, during travel and upon
> arrival to face to face meeting. 
>
> User is supposed to be assisted by computer. And not to assist to
> computer, or to get troubles from computer.
>
> - Time zone shall be more or less recognizable by city and country. 
>
> - User addresses in the address book shall be part of every computer system
>
> - It is natural and common sense to know addresses of people one wants
>   to meet
>
> - By using location of person one wants to meet, computer has got
>   enough information for representation of the time zone
>
> - By sharing appointment record to user in other time zone, that user
>   would see it in his time zone, or by choice in original time zone of
>   the meeting place
>
> A record of time, shall have two attributes, the UTC time and the time
> zone to be displayed. By using system time zone setting, Org file time
> zone settings, heading time zone settings or time stamp time zone
> setting, any export of Org shall contain (by user's option) the
> desired representation of time stamps.
>
> Function of sharing of meetings shall ask local user:
>
> - is user in different time zone? 
>
> And then by choice of the user's location, the time representation
> shall be prepared in such way that both parties understand each other.
>
> That is really not in the sphere of Org where there is not even a
> decent address book available.
>
> Just re-write the time by hand for your friend at other part of the
> world, write the timestamp in his time zone and your time zone, and
> problem solved.
>
> It is supposed to be text. It is not God.

You completely misunderstood the specific issue being discussed. You
clearly have not been following this specific point being discussed and
your long reply just confuses matters rather than helps.

This issue is in dealing with the meeting time when the local timezone
changes due to daylight savings time and the fact you have two different
requirements

1. For meeting where all people are in the same timezone, a transition
in/out of daylight savings changes nothing. The meeting time stays the same

2. For meetings wiht people from different time zones, when daylight
savings transition occurs, the timestamp needs to be changed.  Nothing
needs to happen for the people in other time zones - it isn't their
problem and their meeting time is not affected.

Ihor['s suggested solution was to just use the TZ of the 'meeting', but
that is ambiguous. A meeting doesn't have a time zone and picking just
one of the recipients doens't help as now you just have the issues of
their daylight savings transitions etc.

The 'solution' is to record this meeting in UTC tz. However, to make
this 'workable' for most people, the interface for managing timestamps
needs to make this easy.

For example, I would probablyh want an interface where by default, my
timestamps have no TZ data, but if I call the command to add a timestamp
with the universal argument, it will add a default tz and allow me to
easily change it to a different one.

My default 'no tz data' choice is best for me because I don't travel
much and am rarely in different time zones. Therefore, tz data not
needed and the smaller and easier to read/edit timestamps are
preferred. If on the other hand I was someone who travelled a lot, then
I would want the default to be to add full time zone information to
timestamps (though, I would probably want an overlay or similar to
display the timestamps in a more concise format converted to current
tz).

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Jean Louis  writes:
>
>> ...
>> Should be part of C library to observe those things.
>
> Sure. My previous proposals are all relying on `encode-time' which uses
> time.h from system libraries and utilizing TZDB that is taking care
> about all this insanity.
>
> We, however, might need to be careful about applying date increments. In
> `org-read-date' and `org-timestamp-change', which are implemented in
> Elisp without considering these complexities. (maybe also other
> functions; these are just the ones I can think about)
>
> What should we do when:
>
> 1. It is close to DST transition 2:59 -> 2:00 -> 2:01 -> ... -> 2:59 -> 3:00
>and the users asks to create a timestamp +1h from now
>or, worse, a timestamp +1h from now in a different time zone
>
> 2. A user asks +1w date shift and the time zone has a 1-day jump during DST?
>what about +7d? +1d?
>
> 3. What will be +1d 2:30am timestamp refer to when there DST transition
>as in (1)?

Yep, these are exactly the sorts of issues I was worried about wrt
adding time zone support. Challenge here is I'm not sure there is one
right answer. It will depend on individual use cases. I guess the best
we can do is select what seems to be the most 'sane' approach and
document th elimitations (and possibly how to work around them). The key
is likely to avoid user surprise. Limitations are often better accepted
when flagged up front and when 'discovered' later.

Initially, my thoughts on the 3 above are "I have no clue what the sane
default is" - all options seem to have equal pros and cons. Guess the
best we can do is go with the simplest solution and 'suck it and see". 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>>> Does it sound good enough?
>>
>> No, I'm afraid not. How does org distinguish between meeting 1 and
>> meeting 2? IN meeting one, when the timezone transitions in/out of
>> daylight savings, nothing needs to change, but in meeting 2, when this
>> occurs, the meeting needs to be re-sechduled so that it keeps the same
>> offset relative to UTC.
>> Some mechanism is needed so that the user can
>> identify timestamps like those fo rmeeting 1 from those for meeting
>> 2. My idea was to have meeting 1 type timestamps without TZ info and
>> meeting 2 require fully qualified TZ info. However, while this would
>> probably work, I don't like it because it isn't explicit and would be
>> prone to errors.
>
> I still don't understand.
>
> In Org, you will have a default time zone that will be used to build the
> agenda.
>
> In meeting 1, you set the time zone to your local zone
> In meeting 2, you set the time zone to the time zone where the meeting
> is scheduled.
>
> The, both the meetings will be first converted to the default time zone
> and appear in your agenda adjusted as required.

The problem is with meeting 2 and the assumption there is a definitive
timezone for the meeting.

Consider this scenario. I have a meeting with two other people. We are
all in different timezone. What is the timezone of the meeting?

Thinking more about it, in this situation, you probably just need to set the 
meeting time to UTC
and that would work. However, we would want some easy way to set this
when creating the timestamp (and that could be all that is needed - a
good enhancement to the interface to make it easy to set the timezone)
and good control over how values are displayed in the agenda and org
files (i.e. I imagine you might want a default where they are all shown
in your local time, but similar to working with links, the ability to
display the 'raw' version for editing and other purposes).

As yuou mentioned in another thread, it is likely many of these
scenarios can be adequately managed with good TZ support. It will be
critical that we also have a good UI for setting/adding TZ
information. Then, as you pointed out elsewhere, we will just need good
documentation/tutorials as some of these workflows are not terribly
intuitive and people find this stuff confusing. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>>> Could you please elaborate here?
>>
>> I have some meetings scheduled in my org files which show up in the
>> agenda.
>>
>> Meeting 1 is a reoccurring meeting which happens every 2 weeks. All of
>> the people in that meting are in the same timezone as I'm in. When we
>> transition into/out of daylight savings time, I don't want the timestamp
>> to change. THe meeting will remain at 3pm.
>
> Specifying the timezone should be good enough.
> Not specifying will put you at a risk if you travel (you default OS
> timezone will likely change).
>
>> Meeting 2. This is also a reoccuring meeting. However, this meeting is
>> with people from a number of idfferent time zones. When my timezone
>> moves into or out of daylight savings time, I need the meeting time to
>> be updated - moved forward/back 1 hour.
>
> Again, you can just specify the right timezone and let Org translate it
> to local one when calculating agenda.
>
>> Next week, I'm travelling to a different city for work and will be in a
>> different timezone. I need all my meetings to be adjusted except for
>> those I've already booked that are in the timezone I willl be in while
>> I'm away.
>
> If you don't specify the timezone for both old and new meetings, there
> will be no easy way to deal with this. What you may have to do is: (1)
> indicate explicit timezone for the meetings in the new place (there will
> probably be less of them compared to all other meetings); (2) tell Org
> to use your old time zone as default - it will make the previously
> scheduled meetings without timezone info use the right time zone.
>
>> Finally, I have a few timestamps I use to track some projects and
>> progress on various tasks as well as reports showing actual and
>> estimated effort comparisons as well as managing billing/invoicing. The
>> actual timestamp times are less important than the calculation of
>> durations etc. When durations do cross daylight savings transition
>> points, it is critical that additonal hours are not accidentally
>> added/removed from the duration calculation. Mistakes here could result
>> in me loosing revenue or over charging clients.
>
> For the past timestamps, you can either: (1) make Org put UTC offsets
> when recording clock data; (2) use the idea we discussed about multiple
> default time zones where you can specify different time zones at
> different periods of time (before after travel).
>
> Does it sound good enough?

No, I'm afraid not. How does org distinguish between meeting 1 and
meeting 2? IN meeting one, when the timezone transitions in/out of
daylight savings, nothing needs to change, but in meeting 2, when this
occurs, the meeting needs to be re-sechduled so that it keeps the same
offset relative to UTC. Some mechanism is needed so that the user can
identify timestamps like those fo rmeeting 1 from those for meeting
2. My idea was to have meeting 1 type timestamps without TZ info and
meeting 2 require fully qualified TZ info. However, while this would
probably work, I don't like it because it isn't explicit and would be
prone to errors.

Note that the above is a real scenario - I have to deal with this type
of problem frequently. The other types can, in general, be handled
through judicious use of TZ settings.

Re: Supporting non-free SQL clients in ob-sql (was: [PATCH] ob-sql: Add support for Athena)

[Tim Cross]

Ihor Radchenko  writes:

> Daniel Kraus  writes:
>
>> Tim Cross  writes:
>>
>>> I think you run a high risk of running into GNU policy issues wrt
>>> licensing and free software support given this is a cleint for an AWS
>>> only database.
>>
>> Is it because it's cloud only or because it's proprietary?
>
> Both. For cloud, see
> https://www.gnu.org/philosophy/who-does-that-server-really-serve.html
> (we can have cloud service support, but only those that are not unjust)
>
>> Because we already support Orcale, SAP Hana, MSSql and Vertico for example.
>
> We probably should not.
> Ideally, we need universal SQL cli support and then move all non-free
> clients to external packages.

Well, we don't really support those other databases in the same sense as
is bieng proposed here for this AWS db. Those other databases are only
supported in the sense that you can tell sql mode you are using one of
them and you can add the necessary configuration parameters to make a
connnection, but that is it. We don't, for example, provide the Oracle
client in order to use the oracle database. The user has to install the
Oracle client separately.

So, you could likely modify sql-mode to support the ability to pass
necessary parameters to a client which can connect to the AWS database,
but you cannot bundle the client (in this case, the python code you
referenced).

The fact this is also a cloud based database does make the situation
worse as the official FSF and GNU position is that cloud based services
represent a reduction in user freedom because they do not control those
services and cannot see the soruce code - essentially, use of those
services makes you dependent on and to some extent beholding to (in this
case) Amazon. With the other non-free databases, such as Oracle, you can
at least install the database on your own hardware.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> It also seems that the solution will need some mechanism (possibly on a
>> per time stamp basis) for the user to specify what should happen when
>> either the time zone has a daylight savings transition, when the
>> timezone rules change or when the user's 'default' time zone changes
>> because they have changed locations. 
>
> Could you please elaborate here?

I have some meetings scheduled in my org files which show up in the
agenda.

Meeting 1 is a reoccurring meeting which happens every 2 weeks. All of
the people in that meting are in the same timezone as I'm in. When we
transition into/out of daylight savings time, I don't want the timestamp
to change. THe meeting will remain at 3pm.

Meeting 2. This is also a reoccuring meeting. However, this meeting is
with people from a number of idfferent time zones. When my timezone
moves into or out of daylight savings time, I need the meeting time to
be updated - moved forward/back 1 hour.

Next week, I'm travelling to a different city for work and will be in a
different timezone. I need all my meetings to be adjusted except for
those I've already booked that are in the timezone I willl be in while
I'm away.

Finally, I have a few timestamps I use to track some projects and
progress on various tasks as well as reports showing actual and
estimated effort comparisons as well as managing billing/invoicing. The
actual timestamp times are less important than the calculation of
durations etc. When durations do cross daylight savings transition
points, it is critical that additonal hours are not accidentally
added/removed from the duration calculation. Mistakes here could result
in me loosing revenue or over charging clients.

So, for the first 2 I probably need to somehow flag/indicate that I do
or do not want the time adjusted as a result of a daylight savings
transition. For the 3rd group, I only want adjustments for timestamps
which are not in the 'current' (where I've travelled to) time zone. The
final one is really just about ensuring the transitions don't throw out
duration calculations accidentally.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Daryl Manning  writes:

> I'd argue that setting a specific datestamp  and time for DST would mean that 
> you expected to meet at that
> specific time and date as per DST. If the clocks changed you'd be out of luck 
> (that's where I'd argue you'd
> use a non-specified timezone for a meeting that re-occurs at 10:05 regardless 
> of say PDT or PST).
>
> But if this was an issue with a recurring meeting, then when the clocks 
> changed someone would be out an
> hour because they had specifically booked with DST in mind (note: this is 
> more useful than you think -
> being in non-DST countries and having scheduled meetings in DST based 
> countries, a lot of cal applications
> get this wrong when what I actually want is for that DST scheduled meeting to 
> now be reflected in my
> calendar on the fact they've switched to ST in their time zone - so shifted 
> an hour.). 
>
> But I feel this is something that would be for people who need to take 
> advantage of this. And, more often
> than not, is a recurring meeting issue when DST/ST changes occur. 
>

Yes, this is one of the areas where there are some subtle issues to work
through. With regards to just meetings, I see 3 common scenarios

1. Meeting wiht all participants in the same time zone. The time (lets
say 3pm) should not change when daylight savings comes in or goes
out. The meeting is always at 3pm even though that 3pm might be
different when considered against UTC time.

2. A meeting where some participants are in different time zones to the
org user. Here it isn't clear exactly what should happen when there is a
daylight savings transition in the org user's time zone. Should the org
user's meeting time change or should the participants in the other time
zones update their time for the meeting. On one hand, it makes sense
that the local org user change the meeting time for themselves either
forward/back an hour because it is their time zone which has
changed. However, if the meting has a majority of participants in the
same time zone as the org user, perhaps those in the other time zones
should adjust. Point is, it isn't obvious and there isn't a 'right
solution'. Both needs to be supported and probably need to have some way
to indicate which is the preferred behaviour. 

3. An org user who travels and is often in a different time zone from
their 'home' time zone. IN this scenario, they probably want their
meeting times to be adjusted to the local time zone they are in (unless
they are already recorded in that time zone). Note that this was a
specific example form a previous feature request for time zone support
in org time stamps.

This is just wiht respect to timestamps used for meetings. There are
other scenarios with other subtle issues. For example, someone already
mentioned a scenario where org is being used to record details about
experiments. In this situation, the amount of 'real' time passed between
two timestamps is possibly the most important factor. The fact daylight
savings time transitions have occured are likely irrelevant - just
important any calculations relating to duration are accurate and not
thrown out by one hour due to the wall clock moving forward/abck an
hour.

So far, it seems clear that the solution needs to be flexible and
support timestamps without time zone information, with fully qualified
time zone specification, with time zone abbreviated names and with time
zone offsets.

It also seems that the solution will need some mechanism (possibly on a
per time stamp basis) for the user to specify what should happen when
either the time zone has a daylight savings transition, when the
timezone rules change or when the user's 'default' time zone changes
because they have changed locations. 

Re: [PATCH] ob-sql: Add support for Athena

[Tim Cross]

Ihor Radchenko  writes:

> Daniel Kraus  writes:
>
>> I'm using this patch since a few month that adds support
>> for AWS Athena.
>> The only thing that's maybe against adding it is that
>> `athenacli` (https://github.com/dbcli/athenacli) is not an
>> official AWS tool but just a Python script.
>>
>> What's the opinion on this?
>
> Is this something commonly used?
>
> I see two main issues with the idea:
> 1. I do not like the idea of adding all the possible CLI tools over
>there in ad-hoc manner. It would be cleaner to provide a
>customization to add various cli tools in a defcustom/defvar without
>manually changing the functions.
>
> 2. I feel like it will be hard to maintain such unpopular clients. If
>(1) is addressed + good automatic tests are implemented, things may
>be acceptable for inclusion though.
>
> In summary, I am not against the idea of including a new sql cli, but we
> should better provide a centralized API to do so and make sure that we
> have test coverage, making sure that things are not broken in future,
> when the original committer is gone and nobody else is left familiar
> with specific obscure SQL client.

I think you run a high risk of running into GNU policy issues wrt
licensing and free software support given this is a cleint for an AWS
only database. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tom Gillespie  writes:
>
>
>> I will note that this doesn't address the issue of syntax for
>> historical and future dates. For historical dates those almost always
>> require significant additional metadata to compensate for things like
>> the julian/gregorian calendar switchover etc. for future dates we may
>> want to go ahead and specify something beyond -.
>
> This is somewhat orthogonal to time zones.
>
> I am not sure if julian/gregorian is handled by system time libraries.
> It should, no?
>
> As for years BC, <-0001-...> will be a breaking change. But I do not
> think that we need to really worry about this. Not unless we actually
> get feature request. What is the practical application?

Given that the stated approach is to leverage off OS facilities in this
area, it probably should also be noted that some OSs don't handle
historical dates, especially BC ones, at all well. For example, some OS
use a 32 bit number to represent the date+time and can really only
handle dates between approx 1900 and 2038 (or around there - cannot
remember specific range). So with respect to timestamps and time related
calculations, we are limited by the capabilities of the least capable
supported OS.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Daryl Manning  writes:

> I think timezone you're in should be declared globally, surely?  And then 
> defined in the timestamp?
>

Do you mean globally as in at the OS level or globally in org mode. If
the latter, I disagree. The OS has this information and there is no need
for org to repeat it (and possibly hav it wrong). I would go the other
direction and say that the TZ should only be defined in an org file i it
differs from the OS setting.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> Unfortunately, the common abbreviated forms like EST, AEST etc are
>> inconsistent here. Some places will have a standard and a daylight
>> savings type i.e. AEST and AEDT, while others will have just AEST. TO
>> make it worse, two locations with the same time zone offset my use
>> different versions i.e. Australia/Melbourn is AEST (+10/+11) and
>> Australia/Sydney is AEST (+10) and AEDT (+11) and Australia/Brisbane is
>> just AEST (+10). If everywhere which has daylight savings ensured they
>> used a different abbreviation for daylight saving and non-daylight
>> saving times, life would be easier, but they don't). Then of course
>> there are many countries which don't have a symbolic letter abbreviation
>> i.e. just have e.g -05 as their abbreviated TZ name.
>>
>> It is this and other related reasons why just relying on Emacs' internal
>> API for times and dates is not sufficient. The abbreviated times have
>> ambiguity and the full timezone specification is cumbersome and
>> ugly. Even worse is that issue won't be shown up as failures - you will
>> just silently get wrong results.  
>
> So, basically we may need a way to (1) identify ambiguous TZ
> specifications; (2) signal to user about potential issues.
>
> Note that these are also optional features we may implement any time
> once we have basic timezone support.
>
> For (2) we may use something similar to `world-clock' - display user
> timestamp at point for different time zones. Maybe via eldoc.


If the timestamp only contains the abbreviated name e.g. AEST, then
there is no automatic way to disambiguate - best we could
do is issue a warning. The abbreviated TZ name, unlike the full TZ name,
is really just a shorthand for the time offset from UTC at a specific
point in time. The full TZ name on the other hand states not only the
UTC offset, but also the TZ rules (and with things like the IANA TZ
database, the specific TZ rules in place at the point in time that
timestamp represents). This is essentially Max's main point - if
timestamps for the future have the full TZ name, then we can manage
things like changes in TZ database rules which occur after creation of
the timestamp or adjusting timestamps (for scheduled tasks) based on
changes in location etc). This would not be possible with abbreviated TZ
or just UTC offsets.

As an example, I'm in Australia/Brisbane TZ and I create a timestamp. If
it only has the AEST TZ info, when I travel to Australia/Melbourne org
would not be able to identify I'm in a TZ with an offset of +1100 rather
than +1000 because both use AEST (for NSW it likely would work as NSW
uses AEDT during daylight savings time). However, if the TZ was full
name i.e. Australian/Brisbane, then it does know and can adjust because
my local TZ has changed to Australia/Melbourne. 

So I guess the timestamp format and the code which manages them will
need the ability to use the full TZ name and not just the abbreviated
form (and I guess an option to allow the user to select). In fact, we
probably need a way to select between abbreviated/full dynamically as
well as you might use the different TZ types as a way to flag which
timestamps you want to adjust due to TZ changes (either in TZ db or in
user location) and those you don't want changed.

I also note a comment from an earlier thread regarding timestamps and
historical changes in tz info. I don't think this is an issue.  As far
as I know, past TZ rules/information rarely, if ever, change. It is only
with respect to future dates it is a problem and I doubt there is much
that can be done. For example Postgres recognises that a future
timestamp may become incorrect due to changes in TZ rules post creation
of the timestamp, but they specifically state they don't handle that
situation and that their model has an implicit assumption that the TZ
rules associated with a TZ don't change. In other words, from their
perspectrive, you would have to have a different process which monitors
TZ database information and when it changes, examine your db of
timestamps for ones which have been affected and adjust accordingly
(assuming it matters enough of course). I think org could eventually
reach a middle ground - if a future timestamp has the full tz name, then
changes can be applied, but if not they cannot and leave it to the user
to decide which is used. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 15/01/2023 03:30, Tim Cross wrote:
>> The UTC time stays the same, but the
>> meeting time for me changes twice per year (moving forward/backward an
>> hour).
>
> Meeting time remains the same expressed as local time (15:00), but alternates 
> between
> 04:00 and 05:00 UTC. So repeating schedule can not be stored as UTC, instead 
> UTC timestamp
> should be calculated from local time for each date. (Even libc can do it 
> while you work
> with single timezone.) It is OK to store in UTC already passed events, but 
> local time
> still may be more compact and user friendly.
>
> Actually I am trying to draw attention to a more tricky case when timestamp 
> stored as
> local time is even more important. Event time is bound to local time, and 
> timezone
> database changed due to new rules for this time zone: the government decided 
> to cancel or
> introduce DST transitions or to extend/shorten interval when daylight saving 
> time is
> active. For a timezone without DST time offset may be changed as well.
>
> While timezone database is stable, you can calculate UTC timestamps for any 
> moments
> expressed in local time. When new rules are published some future UTC 
> timestamps become
> invalid. If you know local time, you may update UTC timestamps. If you store 
> just UTC
> timestamp you have a trouble.
>
> Paul Eggert provided an example of updating time transition history, so what 
> is
> authoritative: local time or UTC, applies to the past as well. However I do 
> not agree with
> Paul completely. It is necessary to decide for each timestamp, what is the 
> primary data,
> time offset (so timezone identifier should be updated) or local time (so time 
> offset
> should be updated keeping timezone identifier).
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54764#30
> Re: bug#54764: encode-time: make DST and TIMEZONE fields of the list
> Thu, 14 Apr 2022 15:46:58 -0700
>> Again, that depends on the application. It's typically wrong to store an old 
>> timestamp
>> in a form like "1950-07-01 00:00 Europe/Lisbon", because there is no 
>> standard for what
>> "Europe/Lisbon" means. If you update your copy of TZDB, or interpret such a 
>> timestamp on
>> another computer, that can change the interpretation of such a timestamp. In 
>> this
>> particular case, a change in TZDB release 2021b altered the interpretation 
>> of this old
>> timestamp because we discovered that DST was observed in 1950 in Portugal.
>> If you want to keep the TZDB identifier for advice about how to interpret 
>> dates relative
>> to a timestamp, that's fine. But you should keep the UT offset in addition 
>> to the TZDB
>> identifier, if you want your app to be fully accurate and useful. For 
>> example, you
>> should store "1950-07-01 00:00:00 + Europe/Lisbon" for a timestamp 
>> generated by TZDB
>> release 2021a, so that when you interpret the timestamp in release 2021b 
>> you'll have an
>> idea of what you're dealing with.
>
> So keeping redundant information may be crucial to get warnings that some 
> timestamps need
> to be reviewed.

WRT future timestamps, we would probably have to take the same approach
as postgres i.e. the timezone rules in force at the time the timestamp
is created are assumed to be 'forever'. While this is not true, it is
really the only solution you can adopt and you just have ot accept that
there is the chance that the rule will change in the future and the
timestamp will then be incorrect.

Re: Thoughts on this ob language generator

[Tim Cross]

George Mauer  writes:

> I had a need the other day to execute some typescript in an org document. Now 
> I know that there's an
> ob-typescript package but that doesn't quite work the way I want and expects 
> typescript to be installed
> globally (which runs into a variety of versioning issues).
>
> There is a better option available with the `npx` program (installed 
> alongside `npm`) which can install a
> package along with its dependencies into a temporary sandbox and run its 
> binaries.
>
> I rewrote the typescript babel plugin to do this and then realized that there 
> was relatively little in it
> beyond variable and function names that was typescript-specific. The exact 
> same process can be used for
> anything that has an interpreter up on npm. Coffeescript, mermaidjs, all 
> sorts of things.
>
> So I made a macro. I'm interested what people here think: 
> https://github.com/togakangaroo/create-ob-npx

This looks interesting and could have some great potential. As you say,
tehre is a growing class of languages which could be supported using
this method. I'm interested in trying out the nbb package (Clojurescript
on node) using this method, but right now, no time.

Really just wanted to give feedbac as I noticed nobody else responded
and didn't want to give the impression there was no interest.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> to...@tuxteam.de writes:
>
>>> ... Having an
>>> ability to specify time zones manually will already cater needs for a
>>> number of users.
>>
>> Definitely. But the time stamp (with time zone) in itself doesn't
>> carry enough context to actually decide that. It's even not that
>> easy to wrap one's head around dates that "travel" (the easiest
>> example would be perhaps: "9:00 show up at work" -- when DST takes
>> effect, it's still 9:00 whatever the local time is).
>
> This is basically what we have now - conform to "current" system time
> zone. We are not going to remove this timestamp style. Just add an
> ability to explicitly specify the timestamps if needed.
>
>> When you have appointments with people in totally diverse time zones,
>> perhaps dates tend to be more fixed wrt UTC.
>
> AFAIK, people don't usually bother. As long as you can map from specific
> time zone (applying the currently active summer clock time changes) to
> something like seconds from epoch, you can always calculate back to you
> preferred time zone. Look at https://www.emacswiki.org/emacs/Usergroups.
> They say, for example, "Europe/Berlin", which may be either CET or CEST.
>
> In any case, selection of time zone for user timestamps is not something
> we need to worry about in Org code. Users are to decide. Org might
> assist, but I do not see anything meaningful we can do to help with DST.

I think I basically agree with the last statement. However, perhaps we
need to step back and ask ourselves what it is that people do want which
drives this feature request. I doubt it is simply the ability to add TZ
information to timestamps. I suspect the underlying motivation here is
to have org mode actually use this information in a meaningful way,
which essentially means all the complicated stuff I'm concerned about
and which you seem to imply we wouldn't manage anyway.

To put it another way, we need to clarify what people mean when they
request the feature of timestamp support in org-mode datestamps. What
does this actually mean? Is it as simple as just being able to specify
the timezone (seems relatively easy to implement, but doens't really add
much) or is the expectation that once you have time zone information, it
will be used to do things like adjust date+time in agenda based on
change in locale or change in daylight savings status etc.

Clarifying the end goal will likely focus the discussion a lot
more. My interpretation, which could well be too extreme, is that people
want more than just the ability to add TZ info to their timestamps. They
want their agenda to reflect correct meeting/schedule times based on
their current locale, which may have changed since the initial timestamp
was recorded. They want time duration calculations which are able to
handle DS transitions etc, they want their agenda/calendar to adjust in
a similar way to how their Google calendar will adjust based on DS
transitions. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

to...@tuxteam.de writes:

> [[PGP Signed Part:Undecided]]
> On Sat, Jan 14, 2023 at 03:05:22PM +, Ihor Radchenko wrote:
>>  writes:
>> 
>> > Now there's still enough work for the applications to do: presentation,
>> > parsing, disambiguation, if necessary asking the user for help. Someone
>> > mentioned PostgreSQL -- this is a nice example of what can be done beyond
>> > the (comparatively!) boring details of time zone management :-)
>> 
>> Do I understand correctly that PostgreSQL insists on using timestamps
>> with time zone info in favour or ordinary timestamps? Exactly to avoid
>> issues with the same timestamp pointing to a different real time from
>> epoch depending on the current OS time zone setting?
>
> It doesn't insist: it offers both data types, but the docs are very clear
> on what they prefer.
>

The internal representation of timestamps used by Postgres is ALWAYS
UTC, regardless of whether the field was defined as timestamp or
timestamp with timezone. The only real difference is that if the field
is defined as just timestamp, postgres assumes the data being entered is
in local time and will adjust using the local timezone offset. It will
ignore any tz information in the input. If the field is defined as
timestamp with time zone, it will use the offset defined by the timezone
in the input to determine the UTGC value. When displaying a timestamp,
postgres always uses the local time zone unless you request a different
timezone using the 'at time zone' construct. 

>> Thinking more about this, I can see how it can be important for
>> clocking, and similar auto-recorded information. Users may be surprised
>> to record clocking on some task yesterday just to find the clocking data
>> in future upon travelling from Singapore to San Fran.
>> 
>> So, when implementing time zones, we may need to take care about adding
>> the time zone info when auto-inserting timestamps.
>> 
>> In addition, we may provide some mechanism to set the time zone for:
>> 1. Individual files
>> 2. For all files, including possible time zone transitions over time.
>> 
>> What I mean by (2) is when the user travels from, say, Australia to USA,
>> it could be possible to say: Use Australia/Seattle up to certain time
>> and then use USA/Austin.
>> 
>> However, the above considerations are just nice-to-haves and should not
>> be a blocker to a more generic time zone support in Org. Having an
>> ability to specify time zones manually will already cater needs for a
>> number of users.
>
> Definitely. But the time stamp (with time zone) in itself doesn't
> carry enough context to actually decide that. It's even not that
> easy to wrap one's head around dates that "travel" (the easiest
> example would be perhaps: "9:00 show up at work" -- when DST takes
> effect, it's still 9:00 whatever the local time is). When you
> have appointments with people in totally diverse time zones, perhaps
> dates tend to be more fixed wrt UTC.
>

One thing to keep in mind is that the abbreviated time zone names are
really just abbreviations for time offsets. So AEST is the same as
+1000. However, full timezone names represent complete timezone rules
i.e. Australia/Sydney is not just an offset value, it can tell us (via
the tz database) when daylight savings transitions occur (both for now
and in the past). The abbreviated names lack this information and can
also be ambiguous because some countries use the same abbreviation for
inside and outside daylight savings periods.  

This means that if we were to store timestamps in org files and use the
abbreviated tz name, we cannot handle adjustments due to daylight
savings times because we don't actually know the 'real' timezone of the
timestamp - we only know the tz offset at the point when the timestamp
was created. If we really want a solution which is able to handle
mobility and movement between different timezones and adjust for changes
in offsets due to daylight savings etc, you need to record the true full
timezone name, not the abbreviation. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 14/01/2023 20:50, Tim Cross wrote:
>> I"m sorry, but I don't follow. The UTC time is the only time whihc is
>> not affected by daylight savings transitions, so is the only stable
>> metric. All the others are relative to that time but can change based on
>> things like changes in DST transition dates/times. When DST transitions
>> occur, time is lost/gained wrt the local time, but real time and UTC
>> time do not change. So when DST starts here at 2am on the llth it is 5pm
>> on the 10th UTC. At 2am locally, we jump to 3am, losing an hour, but the
>> UTC time is still 5pm. The meeting I had scheduled for 10am with friends
>> form the US was at 0:00 UTC. It is now at 11am local time, but still at
>> 0:00 UTC, however, I will likely fall asleep in it because instead of my
>> normal 7 hours sleep, I only got six despite going to be and getting up
>> at the same time.
>
> Future events may be affected by changes in timezones happened after 
> scheduling them. UTC
> timestamps becomes incorrect in such cases.
>
> Let's assume that a company from Sydney has a weekly meeting on Mondays at 
> 15:00 *local*
> time. Their standard time offset is +10:00
>
> TZ="Australia/Sydney" date --date '2023-07-15 15:00' '+%F %a %T %Z %z'
> 2023-07-15 Sat 15:00:00 AEST +1000
>
> They decided to invite a person from Singapore (no DST) to join the meeting 
> online next
> year on 2024-01-15 during the period of summer (daylight saving) time (+11:00 
> offset) in
> Australia:
>
> TZ="Australia/Sydney" date --date '2024-01-15 15:00' '+%F %a %T %Z %z'
> 2024-01-15 Mon 15:00:00 AEDT +1100
>
> The person in Singapore should schedule the event at
>
> LANG=C TZ=Asia/Singapore date --date 'TZ="Australia/Sydney" 2024-01-15 15:00' 
> '+%F %a %T
> %Z %z'
> 2024-01-15 Mon 12:00:00 +08 +0800
>
> the same moment as UTC timestamp
>
> date --utc --date 'TZ="Australia/Sydney" 2024-01-15 15:00' '+%F %a %T %Z %z'
> 2024-01-15 Mon 04:00:00 UTC +
>
> If Australia were decided to cancel daylight saving time transition then for 
> the Singapore
> partner the meeting would be shifted by 1 hour to
>
> LANG=C TZ=Asia/Singapore date --date '2024-01-15 15:00 +1000' '+%F %a %T %Z 
> %z'
> 2024-01-15 Mon 13:00:00 +08 +0800
>
> LANG=C date --utc --date '2024-01-15 15:00 +1000' '+%F %a %T %Z %z'
> 2024-01-15 Mon 05:00:00 UTC +
>
> If the guest from Singapore or some guy from Australia decided to store the 
> appointment in
> UTC, not as local time, they would expect beginning of the meeting at 14:00 
> (Sydney time)
> instead of 15:00.
>
> The primary data for this event is
>
> 2024-01-15 15:00 Australia/Sydney
>
> everything else is derived accordingly to current state of timezone database 
> and should be
> recalculated in the case of its change:
>
> - AEDT +1100
> - 2024-01-15 Mon 04:00:00 UTC +
> - 2024-01-15 Mon 12:00:00 +08 +0800 Asia/Singapore
>
> So future events bound to local time must be stored as local time and the 
> corresponding
> local timezone. UTC must be used for global events (e.g. an eclipse) or if 
> the negotiation
> is to fix event time in UTC. UTC is not a silver bullet for time 
> computations, it should
> be used consciously.

OK, I see what your arguing now. However, I disagree on the semantics
here. Really you have the exact same issue in both use cases. Only the
way it is handled is different. For example, I actually do have a
fortnightly meeting with people in the US and my meeting time changes
due to daylight savings transition. The UTC time stays the same, but the
meeting time for me changes twice per year (moving forward/backward an
hour). Likewise, it changes for most of the other people in the meeting
who are in various timzones except for one participant in Brisbane where
they don't have daylight savings time. His meeting time is constant all
year round.

With your semantics, the person in Singapore is the one who has to keep
changing the meeting time while for the Ausies in Sydney, it stays at
15:00 regardless. Worse yet, the Singapore participant has to chnage 4
times per year - when Singapore transitions and when Sydney does.

On the other hand, the downside with my approach is that when local
daylgiht savings transition occurs, all meeting times change, including
those with only local participants, which isn't great either. Again,
this is just some of the issues I'm concerned people are glossing over
when they say it isn't too hard to add TZ support and that we can
leverage off the built-in TZ facilities of Emacs and the OS. The
underlying semantics and model is far mor complex and getting a workable
and user friendly solution which maintains the utility of the existing
timestamps is complicated and hard. The fact you can only really rely on
full timezone names e.g. Australia/Sydney and not abbreviations like
AEST makes it even worse as we run the risk of loosing the compact
format. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 14/01/2023 20:08, Jean Louis wrote:
>> * Max Nikulin [2023-01-14 10:14]:
>>> Let's assume <2023-01-15 Sun 09:00 +1>
>>>
>>> It may be suitable for timestamps in the past, but future is more tricky.
>>> There is no problem if you are going to watch Lunar eclipse. However if your
>>> plan is to attend a local event there is a chance that you will arrive at
>>> wrong time. Sometimes offset of timezones is changed and it may happen
>>> between the moment when you added a scheduled time and the moment of the
>>> event.
>> Can't follow you.
>> with "+1" I would say it is time zone.
>> Basic point is that users shall learn to express themselves by using
>> time zone.
>
> "+1" is not a timezone, it is current offset shared by several timezones. You 
> can not
> assure that time offset at a particular location would not change due to new
> administrative rules.
>
> E.g. Europe/Berlin is a timezone, but, strictly speaking, is still
> underspecified. Sometimes timezones are split into smaller parts.

Yes, this is a problem. We really want a symbolic TZ
specification and we would need 'smarts' i the timestamp generation code
that is able to handle potential offset changes due to daylight savings
transition etc. Even then, the transition time can change between when
the timestamp is set for and when it actually occurs.

Unfortunately, the common abbreviated forms like EST, AEST etc are
inconsistent here. Some places will have a standard and a daylight
savings type i.e. AEST and AEDT, while others will have just AEST. TO
make it worse, two locations with the same time zone offset my use
different versions i.e. Australia/Melbourn is AEST (+10/+11) and
Australia/Sydney is AEST (+10) and AEDT (+11) and Australia/Brisbane is
just AEST (+10). If everywhere which has daylight savings ensured they
used a different abbreviation for daylight saving and non-daylight
saving times, life would be easier, but they don't). Then of course
there are many countries which don't have a symbolic letter abbreviation
i.e. just have e.g -05 as their abbreviated TZ name.

It is this and other related reasons why just relying on Emacs' internal
API for times and dates is not sufficient. The abbreviated times have
ambiguity and the full timezone specification is cumbersome and
ugly. Even worse is that issue won't be shown up as failures - you will
just silently get wrong results.  

If on the other hand all the timestamps were in UTC, you avoid lots of
these issues. However, you would then also require a 'view'
layer/overlay which would take the UTC time, apply whatever the local TZ
is and show that result as the timestamp. THis would avoid things being
out by an hour due to daylight savings transitions and would mean the
user would see timestamps relative to whatever their local time zone is,
regardless of changes in location which has also changed time zones.

The problem with this approach is that now you have lost the 'plain
text' nature of org files. When editing the value, the user would need
to remember the value stored in their org file is UTC, not local
time. Many exporters would also need to be updated to handle conversion
to local time as well.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> Daryl mentioned elsewhere in this thread that how hard this feature
>> would be depends largely on the available libraries for elisp with
>> respect to working with date/time values. Sadly, the available elisp
>> libraries are inadequate in this area. While many of the functions can
>> be called with an optional argument defining time zone information,
>> there is no library which can retrieve this information in a consistent
>> manner and which supports historical data lookup e.g. what were the DST
>> transition dates for NSW Australia in 1970 (trick questions, NSW didn't
>> adopt DST until 1971). The existing library functions are focused more
>> on simpler time calculations where time zone information is less
>> relevant i.e. running timers, timing command execution, comparing file
>> modification times etc. Sometimes, it is easy to forget that while Elisp
>> is powerful, it isn't really a general purpose programming language like
>> C or Rust or even CL and Scheme. It is primarily a programming language
>> focused  on a specific domain, the editor. While it has some great
>> libraries  it also has  anumber of areas where it lacks the sort of
>> sophistication we have grown to expect with more general purpose
>> languages.
>
> Emacs' own time zone support is relying on external libraries. AFAIU,
> historical context should be handled no worse than in the OS.
>
> More precisely, tzlookup from timefns.c relies on time.h and Windows
> equivalents. All the heavy lifting is done by core OS libraries.
>
>> A further complication is that accessing date and time related data is
>> very system dependent and there is no high level library which abstracts
>> this so that you  can easily get consistent results across all the
>> platforms Emacs runs on. Even the underlying data type can vary greatly
>> (i.e. some platforms use a 64 bit value while others only use a 32 bit one).
>
> This is not the problem we need to worry about. We can use whatever
> Emacs provides and rely on future improvements in Emacs where things are
> deficient. We should not aim for 100% accurate time zone support. Just
> good enough. It's not Org's job to implement the gory details of time
> zone support.

I guess we will have to disagree here. If org is going to claim to
support time zones, then 100% accurate is IMO essential. As it stands
now, we don't claim TZ support and time calculations are correct on the
basis they are all done relative to current locale. However, once you
add TZ information, if you don't apply it consistently and correctly,
that means the values can no longer be relied upon.

Or to put it in another way - currently, it is well understood where org
timestamps fall down. However, once you add TZ and provide the
expectation TZ data will be respected correctly, all bets are off.

Anyway, I've made my point and will leave it alone now. Ironically, time
will tell us who is right and who isn't.

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 14/01/2023 16:32, Tim Cross wrote:
>> If org was to add TZ capabilities to timestamps, the underlying format
>> would have to be UTC.
> ...
>> can change based on various criteria, including political whims
>> (e.g. Australia eastern DST transition date was changed in 2000 because
>> Sydney hosted the Olympics that year).
>
> Due to this particular reason storage format for significant fraction of 
> future timestamps
> (but not always) must not be in UTC
>
> 2024-01-14 Sun 21:14:58 ACS (+0930, Australia/Darwin)
>
> is not the same as
>
> 2024-01-14 Sun 11:45:58 UTC (+, Z)
>
> despite currently it is.
>

I"m sorry, but I don't follow. The UTC time is the only time whihc is
not affected by daylight savings transitions, so is the only stable
metric. All the others are relative to that time but can change based on
things like changes in DST transition dates/times. When DST transitions
occur, time is lost/gained wrt the local time, but real time and UTC
time do not change. So when DST starts here at 2am on the llth it is 5pm
on the 10th UTC. At 2am locally, we jump to 3am, losing an hour, but the
UTC time is still 5pm. The meeting I had scheduled for 10am with friends
form the US was at 0:00 UTC. It is now at 11am local time, but still at
0:00 UTC, however, I will likely fall asleep in it because instead of my
normal 7 hours sleep, I only got six despite going to be and getting up
at the same time.  

> Depending on use case the same particular fields of timestamp may be 
> authoritative or
> derived for user convenience from other data.
>
> UNIX timestamps in seconds in UTC timezone almost unavoidable will be used 
> underneath, but
> operations like "start of next day" require non-trivial computations to find 
> if time zone
> offset changes in between.

Yes, but if the underlying representation is UTC, then all you have to
do is apply local TZ data to get the local time. THis is how postgres
works. You just hav eto ensure the TZ data you get is the TZ data for
that date in that locale. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> I agree this would be a great feature to add. However, after having
>> looked at it in some detail, I realise that not only is it not a trivial
>> task, it is actually a very large and complex task and will require
>> extensive work which will almost certainly result in breakage with
>> regards to backwards compatibility.
>
> Not really. Our timestamp format, in fact, provides sufficient
> flexibility to add extra metadata to timestamps.
>
> In anticipation to add time zones in future, I have added the following
> to the Org timestamp spec (see
> https://orgmode.org/worg/org-syntax.html#Timestamps):
>
> DATE TIME REPEATER-OR-DELAY
>
> TIME (optional)
> An instance of the pattern H:MMREST where H represents a one to two digit 
> number (and can
> start with 0), and M represents a single digit. REST can contain anything but 
> \n or
> closing bracket.
>
> Note that REST imply that almost arbitrary suffix can be in TIME without
> braking the existing Org timestamp parsing code.
>
> REST, among other things may be
> https://en.wikipedia.org/wiki/ISO_8601#Time_offsets_from_UTC or a valid
> value of TZ POSIX variable. Exact details can be discussed.
>
> Note that TZ should be fully supported by `encode-time' (ZONE):
>
> (SECOND MINUTE HOUR DAY MONTH YEAR IGNORED DST ZONE)
>
> We do not need to worry about internal representation and conversions
> and simply rely on Emacs.
>
>> At the risk of over simplifying the matter, I would suggest the big
>> challenge here is that there are two somewhat competing (and
>> conflicting) use cases. On one hand, you want a high level abstraction
>> which makes working with dates and times easy and clear. TO some extent,
>> that is what we have now. On the other hand, we need something far more
>> complex which is able to handle multiple time zones. This means being
>> able to handle both base timezone offsets as well as offset adjustments
>> for things like daylight savings time. There is a lot of hidden
>> complexity here. You have to handle the fact that daylight saving
>> chang-over dates/times are dynamic i.e. not necessarily the same every
>> year. This adds the additional complexity of having to somehow track
>> historical information regarding such changes, which isn't as readily
>> accessible in a consistent manner on all platforms.
>
> We do not care about the details as long as Emacs can handle this. As
> long as the user supplies DST and ZONE somehow, we can rely on Emacs'
> support and system TZ implementation.
>
>> Then there is the other 'messy' stuff. For example, when calculating
>> time ranges and number of days, hours/ minutes between two dates you
>> need to remember to add/remove the hour if the range crosses over a
>> daylight savings period. Similarly you need to ensure you make the
>> correct adjustment when changing timezones (consider emacs on a laptop
>> for someone who travels a lot between different time zones). Not only do
>> you need to take into account the different timezone offset, you also
>> need to look at the date and then adjust according to the timezone
>> offset for the current timezone at the time of the timestamp rather than
>> just considering the current time offset.  
>
> Again, we don't need to worry about this. Once we use `encode-time',
> operations on time should just work. This is what we do anyway in most
> of Org code.
>
>> I expect what is needed is an 'internal' UTC based representation which
>> is used for all calculations and an 'interface' layer, which converts
>> the UTC representation into the local display representation for
>> consumption by humans.
>
> This is what we already to via `encode-time' and `decode-time'.
> Check out `org-time-string-to-time'.
>
>> However, the real challenge here is that this is a very large piece of
>> work and it needs someone who is prepared to take it on. I suspect until
>> someone who needs to scratch this itch sufficiently comes along, this is
>> a feature which will be unlikely to make it to the top of the todo
>> list. There are simply far too many existing feature improvements and
>> additions people will prefer to work on before taking on this
>> one. Things are made more difficult because it isn't the sort of task
>> which can be achieved with small increments over time. This is more
>> likely to be something which needs to be developed in a feature branch,
>> which once it reaches a sufficient level of completeness can be used and 
>> verified working for a wide variety of environments before then working
>> out how to fold it ba

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Max Nikulin  writes:

> On 14/01/2023 02:06, Jean Louis wrote:
>> This is good for review as related to PostgreSQL database:
>
> I agree that PostgreSQL is an example of good implementation of time-related 
> calculations.
>
>> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_timestamp_.28without_time_zone.29
>
> I do not find this section enlightening in the context of this discussion. It 
> is more
> related to storage formats of DB than to higher level aspects suitable for 
> applications.
>
>> It would mean that timestamps become time zone aware. Something like this:
>> <2023-01-15 Sun +1>
>
> Welcome to another set of issues.
>
> First of all, you have just date without specific time point. It may mean 
> particular
> calendar day irrelevant to time zone. ICQ had a bug when they decide to store 
> birth dates
> as timestamps. Significant fraction of users received automatic greetings on 
> wrong day, a
> kind of off by one error. So dates must stored as dates omitting time and 
> zone.
>
> Let's assume <2023-01-15 Sun 09:00 +1>
>
> It may be suitable for timestamps in the past, but future is more tricky. 
> There is no
> problem if you are going to watch Lunar eclipse. However if your plan is to 
> attend a local
> event there is a chance that you will arrive at wrong time. Sometimes offset 
> of timezones
> is changed and it may happen between the moment when you added a scheduled 
> time and the
> moment of the event.
>
> Notice that Org timestamps already associated with a timezone, the one is set 
> in libc
> (system timezone, or the one set for particular process). So daylight saving 
> time and
> administrative time transitions should be taken into account. So Org 
> timestamps may be
> ambiguous (mostly) 1 hour per year around backward time transition, e.g. for 
> timezones
> having DST. This case explicitly specifying time offset helps to avoid 
> uncertainty.

If org was to add TZ capabilities to timestamps, the underlying format
would have to be UTC. My previous post where I suggested there was two
'layers', the underlying storage layer (used for calculations like
duration or comparison etc) and a presentation layer (what the human
sees, which is often in whatever their local TZ is). While overly
simplified, this is basically the approach used by postgres. If org mode
was to adopt TZ support for timestamps, I am quite certain that the only
sane way to do this will be to store the timestamps as UTC. Anything
else will just cause massive complications for anyone who moves between
different timezones as well as complicating calculations which cross
over DST transition points. 

I think most people are aware of the challenges of dealing with daylight
savings and the fact that the transition date+time into and out of DST
can change based on various criteria, including political whims
(e.g. Australia eastern DST transition date was changed in 2000 because
Sydney hosted the Olympics that year).  This means there is also an
historical context when calculating and formatting timestamps - it isn't
as easy as looking up to see if a location has DST and when the
transition date/time is - you have to determine that as of the date of
the timestamp.

THen we have the further complication of calendar changes and different
calendars. For example, many people don't realise that Russia didn't
adopt the Gregorian calendar until 1918. For example, the reference to
Napoleon earlier is more complicated when considered from a Russian
locale.

Daryl mentioned elsewhere in this thread that how hard this feature
would be depends largely on the available libraries for elisp with
respect to working with date/time values. Sadly, the available elisp
libraries are inadequate in this area. While many of the functions can
be called with an optional argument defining time zone information,
there is no library which can retrieve this information in a consistent
manner and which supports historical data lookup e.g. what were the DST
transition dates for NSW Australia in 1970 (trick questions, NSW didn't
adopt DST until 1971). The existing library functions are focused more
on simpler time calculations where time zone information is less
relevant i.e. running timers, timing command execution, comparing file
modification times etc. Sometimes, it is easy to forget that while Elisp
is powerful, it isn't really a general purpose programming language like
C or Rust or even CL and Scheme. It is primarily a programming language
focused  on a specific domain, the editor. While it has some great
libraries  it also has  anumber of areas where it lacks the sort of
sophistication we have grown to expect with more general purpose
languages.

A further complication is that accessing date and time related data is
very system dependent and there is no high level library which abstracts
this so that you  can easily get consistent results across all the
platforms Emacs runs on. Even the underlying data type can vary greatly
(i.e. 

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Tim Cross]

Daryl Manning  writes:

> Following on from thread at https://www.reddit.com/r/orgmode/comments/zrppqw/
>
> [First off, I just wanted to say thank you to everyone that works on 
> org-mode. It is a wonder.]
>
> While I realize a few kicks at this can may have been taken, I wanted to 
> (re-)propose Timezone support in org-mode. The
> world is much less local these days and we're all more remote and 
> coordinating globally these days.
>
> *Background*
>
> 1. org-time-stamp-formats TZ currently only affects display and exports
> 2. org-agenda itself is not TZ aware
> 3. Several discussions on this have taken place over time
> 4. Concerns raise included breaking backwards compatibility
>
> *Proposal*
>
> 1. org-mode sets an optional variable (org-timezone-aware t) which enables TZ
> 2. org-agenda needs a way to determine which timezone it is in
> 3. Once enabled, any timestamp not exhibiting a TZ in it is considered "local 
> time" wherever that is (I do not think UTC
> would work for this)
> 4. org-agenda can calc local based on TZ differences
>
> I understand this is by no means trivial and quite gnarly with DST and such 
> to figure out but I do believe libs exists to
> deal with that heavy lifting. Currently, it does feel like a hole in org-mode 
> as a 21st century organizer (disclaimer:
> digital nomading so might feel it more keenly). Also, just interested in 
> making org-mode a more awesome tool for
> everybody. 
>
> I'd love an understanding of the alluded to reservations raised in the reddit 
> thread and in the mailing list threads
> mentioned that the format change might break things (I was unsure if that was 
> referring to say, how time ranges were
> handled, or how say date ranges got dealt with (for example, say a flight 
> from Singapore to Vancouver which takes off in
> one time zone and lands in another - something that is often in my cal.). 
>

I agree this would be a great feature to add. However, after having
looked at it in some detail, I realise that not only is it not a trivial
task, it is actually a very large and complex task and will require
extensive work which will almost certainly result in breakage with
regards to backwards compatibility.

At the risk of over simplifying the matter, I would suggest the big
challenge here is that there are two somewhat competing (and
conflicting) use cases. On one hand, you want a high level abstraction
which makes working with dates and times easy and clear. TO some extent,
that is what we have now. On the other hand, we need something far more
complex which is able to handle multiple time zones. This means being
able to handle both base timezone offsets as well as offset adjustments
for things like daylight savings time. There is a lot of hidden
complexity here. You have to handle the fact that daylight saving
chang-over dates/times are dynamic i.e. not necessarily the same every
year. This adds the additional complexity of having to somehow track
historical information regarding such changes, which isn't as readily
accessible in a consistent manner on all platforms.

Then there is the other 'messy' stuff. For example, when calculating
time ranges and number of days, hours/ minutes between two dates you
need to remember to add/remove the hour if the range crosses over a
daylight savings period. Similarly you need to ensure you make the
correct adjustment when changing timezones (consider emacs on a laptop
for someone who travels a lot between different time zones). Not only do
you need to take into account the different timezone offset, you also
need to look at the date and then adjust according to the timezone
offset for the current timezone at the time of the timestamp rather than
just considering the current time offset.  

I expect what is needed is an 'internal' UTC based representation which
is used for all calculations and an 'interface' layer, which converts
the UTC representation into the local display representation for
consumption by humans. The problem with this is that it is likely to
break the core feature of org files i.e. that they are in plain text. I
guess we could possibly do somehting clever with overlays such that UTC
date/times are rendered/presented in local time format. However, we
would then also require some type of interface for users to enter
dates/times (to ensure they are converted to the correct UTC internal
format).

However, the real challenge here is that this is a very large piece of
work and it needs someone who is prepared to take it on. I suspect until
someone who needs to scratch this itch sufficiently comes along, this is
a feature which will be unlikely to make it to the top of the todo
list. There are simply far too many existing feature improvements and
additions people will prefer to work on before taking on this
one. Things are made more difficult because it isn't the sort of task
which can be achieved with small increments over time. This is more
likely to be something which needs to be 

Re: OS advice

[Tim Cross]

Ypo  writes:

> Hi
>
> Orgmode is sometimes desperately slow on my PC: 
>
> Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100 Mhz
>
> (RAM)4,00 GB
>
> I am running Windows 10, everything I use works OK, but Orgmode. 
>
> Do you think that if I install a Linux OS, Orgmode would run fast? Any OS 
> suggestion?
>

Sadly, the answer is likely "that depends". There are just too many
unknown variables to provide a definitive answer. However, what I can
tell you is

- I have frequently taken hardware which users have found old and slow
  when running Windows and given it a new life running Linux. Linux can
  certainly perform better with less resources given some caveats. 

- Unlike Windows, Linux comes with a wide variety of destkop
  environments and window managers. Some are resource hungry and others
  are extremely light-weight. Selecting the right window manager will be
  crucial. For older and slower machines with only a small amount of
  memory, I would consider window managers like XFCE or maybe MATE. 

- From the specs you provide, my guess is that memory is your main
  bottle neck. This would further suggest that if you were to switch to
  Linux, avoid memory hungry desktop environments like Gnome or
  KDE. AGain, XFCE is small and fast and very reliable. It lacks the
  visual candy of other environments, but given your specs, something
  needs to be given up and visual candy seems a good starting
  point. However, this change will likely require some adjustment on
  your part. While there is little you cannot do on a Linux system, the
  level of integration and automation 'out of the box' is likely to be
  less. You will certainly be able to create an environment which is
  just as efficient and convenient as Windows, but it will likely take
  additional effort and willingness to adapt on your part. 

- Emacs and org mode can also be memory hungry. It is possible (likely
  in fact) that you could get much better performance, even under
  windows, by modifying how you use org mode. Things I would recommend
  include

- Keep your org files as small as possible. Use multiple files
  rather than one big file.
- Don't load any Emacs packages you don't actually use. Don't
  load/install any org packages you don't actually use/need.


A common error I see people make now that we have convenient emacs/elisp
packages is to install lots of packages. When I've been helping people
with Emacs performance, the first thing we do is go through all the
things they have installed/configured. Frequently, there are lots of
things installed which they never use.

What I sometimes recommend is that they comment out as much of their
Emacs and org configuration as possible and then use the system for a
few days. During this time, only enable something once you find you need
it. It is often surprising to them how much stuff they had configured or
installed which they really never used. The other benefit is that
smaller and simpler setups are less likely to have undesired side
effects or interactions with other packages, leading to fewer problems
and increased stability.

At the end of the day, a system with only 4Gb of memory is on the tight
side for a modern setup. I would argue the minimum size these days is
more like 8Gb and a 'good' setup is at least 12Gb. I personally have a
minimum of 16Gb and prefer 32Gb, but I also use a lot of VMs and other
container techniques to manage multiple stable and unrelated development
environments. On the other hand, my wife and children use small systems
running Linux XFCE with only 4Gb and find them quite adequate for what
they do (mainly email, surfing the web, basic office documents with
libre office etc). These systems are things like asus notebooks, small
form factor, slower CPU and 4Gb memory. They find them quite adequate
and appreciate the small form factor, but they also don't spend 8 hours
a day on them!

Re: [TASK] Enhance Worg HTML styling

[Tim Cross]

Leo Butler  writes:

> On Fri, Jan 06 2023, Tim Cross  wrote:
>
>> alain.coch...@unistra.fr writes:
>>
>>> Tim Cross writes on Thu  5 Jan 2023 09:43:
>>>
>>>  > As a simple example, try increasing the font size and see what
>>>  > happens to the menus. Keep in mind that some users require a very
>>>  > large font (for example, I use a 26 or 28 pt font.
>>>
>>> OK, I understand.  (Even with default font size, I hate that the table
>>> of contents hides some parts of the text -- I had forgotten about
>>> that.)
>>>
>>>  > [...] There have also been numerous other issues (many have already
>>>  > been addressed) such as broken links, links to data which doesn't
>>>  > exist or has wrong MIME type [...]  Perhaps even more unfortunate
>>>  > is that sometimes, you can come across some good information in the
>>>  > worg site, but finding that same information again at a latter date
>>>  > is often extremely challenging.
>>>
>>> But that's not "Org styling", right?  Or am I confused about what
>>> "styling" means?
>>>
>>> At any rate, I agree with your other examples, understand why the
>>> styling (as I understand it) should be improved, and am no longer
>>> worried about changes.  Thank you very much for your time.
>>
>> It isn't just about the styling - that is just one aspect and the one I
>> planned to start with. However, styling and presentation will also be
>> part of providing better links/navigation, which I think is also
>> necessary to make exploration and discovery easier and more consistent.
>> .
>
> Can you give us an example/model of a site that does these things
> better? I agree with your assessment, but feel that some kind of target
> is needed.
>
> TIA,
> Leo

As Bastien said, small increments is likely the way to go.

So, for an initial target, how about just having set of CSS styles which
support different screen sizes and different font sizes - one where,
unlike the current one, the page content is not obscured by the menus
and navigation buttons.

For an added bonus, a design which provides consistency and convenience
which helps with both general browsing and locating specific information
within the site would be good. 

As for an example site, I don't have anything specific in mind and have
ideas taken from various sites. Examples are of little benefit IMO - it
really just comes down to someone having the time to define a new style,
apply it to a development site (likely also on sourcehut) and then get
feedback. We don't need to over think this or get too bogged down in
design - just go for functional and see what falls out the other end.

One question I'm not sure about is whether it would be better to craft
CSS from scratch or better to adopt a CSS framework (which is
appropriately licensed) and use that. I tend to feel the latter would be
better, but others may have different opinions.

Re: [TASK] Enhance Worg HTML styling (was: [BUG] worg-setup.org is outdated)

[Tim Cross]

alain.coch...@unistra.fr writes:

> Tim Cross writes on Thu  5 Jan 2023 09:43:
>
>  > As a simple example, try increasing the font size and see what
>  > happens to the menus. Keep in mind that some users require a very
>  > large font (for example, I use a 26 or 28 pt font.
>
> OK, I understand.  (Even with default font size, I hate that the table
> of contents hides some parts of the text -- I had forgotten about
> that.)
>
>  > [...] There have also been numerous other issues (many have already
>  > been addressed) such as broken links, links to data which doesn't
>  > exist or has wrong MIME type [...]  Perhaps even more unfortunate
>  > is that sometimes, you can come across some good information in the
>  > worg site, but finding that same information again at a latter date
>  > is often extremely challenging.
>
> But that's not "Org styling", right?  Or am I confused about what
> "styling" means?
>
> At any rate, I agree with your other examples, understand why the
> styling (as I understand it) should be improved, and am no longer
> worried about changes.  Thank you very much for your time.

It isn't just about the styling - that is just one aspect and the one I
planned to start with. However, styling and presentation will also be
part of providing better links/navigation, which I think is also
necessary to make exploration and discovery easier and more consistent.
.

Re: [TASK] Enhance Worg HTML styling (was: [BUG] worg-setup.org is outdated)

[Tim Cross]

alain.coch...@unistra.fr writes:

> Bastien Guerry writes on Wed  4 Jan 2023 11:21:
>
>  > Strong +1 on working on Worg's styling.
>  > 
>  > The task may be daunting, but we can also tackle it incrementally.
>  > 
>  > >From memory, orgmode.org/worg is visited by ~30k persons each month,
>  > that 1000 persons per day.  A patch enhancing the .css will make 1000
>  > persons happiers each day.
>
> So far it was an obscure discussion for, but as a visitor of
> orgmode.org/worg I now feel concerned.  What's wrong with Worg's
> styling?  A specific example might be enlightening.  (Enhancement for
> some can be deterioration for others.)
>

As a simple example, try increasing the font size and see what happens
to the menus. Keep in mind that some users require a very large font
(for example, I use a 26 or 28 pt font.

The current site is not good from an accessibility perspective, renders
inconsistently with different browsers, does not have consistent
keyboard navigation, arguably has inconsistent styling in some areas
etc.

If you are able to use the defaults (default font and size, default
fg/bg colors, normal 'desktop' screen, things seem ok. However, once you
need different fonts, different size text, different fg/bg colors or are
using a mobile device or assistive technologies, like a screen reader,
things rapidly degrade. There have also been numerous other issues (many
have already been addressed) such as broken links, links to data which
doesn't exist or has wrong MIME type, issues associated with file name
case etc.  There is also a fair amount of inconsistency in how pages are
presented - some seem to have good navigation support while others do
not, some pages seem to fit into an overall 'site map' while others seem
to be out in their own island etc. As a result, the ability to
effectively browse the site and follow 'threads' of information is often
challenging. Perhaps even more unfortunate is that sometimes, you can
come across some good information in the worg site, but finding that
same information again at a latter date is often extremely challenging. 

Re: Intention of verbatim text?

[Tim Cross]

 writes:

> My current solution is to convert ~code~ to code and
> =verbatim= to verbatim.
>
> In that case the user can decide himself how to render them. In my
> default CSS I would render the ~code~ in monospace with a light gray
> background (different from the whole page background) and =verbatim=
> with monospace only but without extra background color.
>

I think I would agree in an approach which leverages the power of CSS is
the best approach. My own personal preference would be to have as much
of the rendering/formatting of various elements managed by CSS classes
as possible as this would make it much easier for users to change how
things are rendered by modifying the CSS classes. Frameworks like Bulma
(https://bulma.io) show how powerful just using CSS can be. Much easier
to modify a CSS class than change a tag.

Re: Intention of verbatim text?

[Tim Cross]

 writes:

> Hi,
>
> in org you can have inline verbatim and code text elements like this.
>
> Example with =verbatim= and ~code~.
>
> I would like to understand what "verbatim" really means. What is the
> semantic behind it? What content should go in there?
>
>
> I'm aware of the separation of content and its presentation.
> I'm also aware of the different renderings in my Emacs. Booth are
> monotype but with different colors.
>
> The org html export to create both with  tag. So in HTML output
> there is no difference between verbatim and code anymore.
>
> I also read a lot about the HTML tags code, pre, kbd and samp.
>
> I wonder that maybe I totally misunderstand the intention of
> "verbatim"?
>
> The background of my question is that I have my own
> org-to-html-converter [1] and try to decide how to treat =verbatim=.
> Which HTML tag should I use.
>
> Thanks
> Christian
>
> [1] -- 

IMO (and it is just my opinion, not that of the org developers), the
main use of verbatim (i.e. =text=) is to escape any further
interpolation as org markup. It basically says, when exporting, export
'as is' with no further processing.

Consider a situation where you want to write A_B, but don't want it to
be interpreted as A with a subscript B. There are a number of ways to
handle this. One is to set the #+OPTION to ^:nil and turn off the
behaviour for the whole file or you can use =A_B= to just turn it off
for that use (though this does have other possibly unintended
side-effects, such as a font change, but I'm really just trying to
illustrate the point here).

I would have to look more closely at the output of the HTML export to
verify your assertion that both verbatim and code are rendered the
same. With respect to 'phrases', I guess there is no real difference in
outcome. However, the standard HTML code tag does not preserve line
breaks. Traditionally, for blocks of code, the code tag would also be
wrapped in the pre tag. However, things have evolved and now it is much
more common to see just the code tag with an additional CSS class which
is used to manage the preservation of line breaks and whitespace. From a
purely tecnical 'correctness' standpoint, I would argue that =text=
should be rendered as text and not text. When
exporting data, we don't have any semantic markers/information for
=textg=, so wrapping it in a semantic tag like  is IMO incorrect
as we are imposing a semantic interpretation without justification. .

Re: [BUG] worg-setup.org is outdated

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> First step is to get a working local copy so that I have something to
>> work with. AFter that and a bit of exploring, I should have a better
>> understanding and idea how to go forward. 
>
> Hi Tim,
>
> May I know if you got a chance to continue working on this?
> Let us know if you need any help from us.

Hi Ihor,

sorry, no real progress.

I did get a working local copy and started looking into what needed to
be done to improve things. Unfortunately, the more I understood, the
more it became obvious that simple tweaking was unlikely to consistently
improve the situation. Each little improvement I made just caused or
exposed other issues and it quickly spiralled down.

Unfortunately, other commitments then took over and I've been too busy
to focus on org stuff for some months now. Even participation in the ML
has been challenging.

A significant re-design of the worg styling is required in order to get
a presentation which both looks good and which works with respect to
accessibility requirements. I don't believe the current styles are
workable. Someone with greater CSS fu than me might do better, but from
what I could tell, the basic underlying premise for the existing styles
is flawed. I suspect it would be possible to 'fix' things, but it would
be a major style re-working. 

I would still like to get back to this, but right now, don't know where
things are likely to be in the short term. There are some job related
things which will be eitehr completed or stepped up to the next stage by
mid Feb and I may have a clearer picture by then. For now though, I have
limited resources to dedicate to org. 

Re: [SECURITY] Arbitrary code evaluation security in Org (was: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable)

[Tim Cross]

Ihor Radchenko  writes:

> Ihor Radchenko  writes:
>
>> P.S. Considering intense discussion around the topic, what about
>> reverting my commit from the release? We can then re-consider the whole
>> design and apply something more elaborate later.
>
> I now reverted the discussed commit.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=345402148
>
> We need to come up with a uniform security system where all the code
> evaluation is (1) easy to control via variables and user queries; (2)
> not too annoying for users; (3) provides the necessary context for users
> to decide if a code is safe to run.
>
> Before we continue the discussion, I will try to summarize the current
> state of Org manual and code wrt code evaluation of the code coming from
> Org documents (including downloaded Org files).
>
> In the manual we now have
> 17.13 Code Evaluation and Security Issues
>
> This section talks about
>
> 1. Source block bodies and `org-confirm-babel-evaluate'
> 2. shell/elisp links and `org-link-shell-confirm-function' +
>`org-link-elisp-confirm-function'.
>
>Notably, `org-link-elisp-skip-confirm-regexp' is not mentioned in the
>manual.
>
> 3. Table cells, with no way to query or disable execution.
>
> In reality, we have many more places in the code where arbitrary text
> from Org files can be evaluated.
>
> I have marked some places in the above commit with FIXME.
>
> From my audit of Org sources the following scenarios may lead to
> arbitrary code execution:
>
> 1. Code block bodies
> 2. Elisp sexps in header args. Notably, not only `org-babel-read' is
>responsible for evaluating header args. :results and :exports are
>evaluated independently.
> 3. Table cells
> 4. "eval" macros during expansion
> 5. Diary sexps
>
> To the best of my knowledge, this list should be complete. Though I
> would appreciate someone double-checking.
>
> 
> According to the above:
>
> - `org-confirm-babel-evaluate' allows either using
>   `org-babel-confirm-evaluate' prompt (when t), not asking at all (when
>   nil), or using arbitrary prompt function.
> - `org-link-shell-confirm-function' + `org-link-elisp-confirm-function'
>   are similar, except defaulting to `yes-or-no-p'.
> - `org-link-elisp-skip-confirm-regexp' extends indiscriminate function
>   queries to also skip certain (single) regexp.
>
> The situation is not ideal.
>
> We have similar, but more detailed system for downloading remote files:
>
> - `org-safe-remote-resources' allows users to define certain files/urls
>   that are known to be safe
> - `org-resource-download-policy' provides choice between "always query",
>   "query unless safe", "never download", and "always download"
>
> Also, `org--confirm-resource-safe', unlike `org-babel-confirm-evaluate'
> allows manipulating `org-safe-remote-resources' interactively by marking
> current URL or URL domain safe for future.
>
> 
> What we can do
>
> 1. Introduce a new customization `org-confirm-evaluate-safe-regexps'
>listing regexps that are considered safe or cons cells
>(src-body/header-arg/table/macro/diary . regexp)
>
> 2. Introduce a new customization `org-confirm-evaluate' that can be set
>to t/nil/prompt/safe/[function]/[alist]:
>- t/safe :: to display default prompt unless the code block in buffer is
>marked safe
>- prompt :: always ask (the prompt will then not display options to
>add current sexp to safe list)
>- [function] :: custom function that returns t/safe/prompt/[alist]
>- [alist] :: (src-body/header-arg/table/macro/diary/t .
>  t/safe/prompt/function)
>  (t . ) stands for default value.
>
> 3. The default prompt will mimic `org--confirm-resource-safe',
>additionally accepting Y/N to accept/decline all the evaluation in
>current command.
>
> This system will be used across Org for code evaluation. Old variables
> will be supported, but obsoleted.
>
> WDYT?

Hi Ihor,

this looks promising. However, there is a lot to be considered here and
it requires some thought before any meaningful feedback can be provided.

The big challenge here will be in getting sufficient fine grained
control that all use cases can be catered for while also providing a
high level interface suitable for the 'general' case which is both
reasonably easy to understand at the user level and flexible enough for
a default configuration.

For many users who don't share org files, who work on a single user
desktop and who implicitly trust the files they are working with, it is
unlikely they want to be bothered by any of this. It should 'just
work'. I suspect this is the majority. Others will have some sharing of
documents - perhaps in a work group, a class or some form of team
activity. The trust here is likely fairly high but perhaps not
absolute. There is probably a need for some choice on execution on a per
file basis. Finally, there are those org 

Re: [FR] Present list of errors in separate buffer when running org-export, similar to what org-lint does (was: [Syntax discussion] Should we treat src blocks without LANG as paragraphs?)

[Tim Cross]

Ihor Radchenko  writes:

> As Max proposed, it may be a good idea to extend the concept of org-lint
> to export.
>
> We may unify the LaTeX errors, some errors/warnings potentially signalled
> by export backend, and maybe some warnings from org-lint during the
> export process. These errors can ideally be displayed just like in
> org-lint, with ability to jump to the problematic LoCs in
> Org/intermediate file.
>
> This is a medium task and one will need to:
> 1. Generalize org-lint interactive interface (list of errors) to work
>with other code. Extract it into a separate library.
> 2. Accumulate export issues into a list to be displayed. For example,
>broken links can be displayed
> 3. Make functions like `org-latex-compile' accumulate LaTeX errors as
>well. (Optionally) associate LaTeX buffer lines with Org source.
> 4. Display the list of errors/warnings at the end of successful or
>failed export.
>
> Each step if fairly simple, except optional part in 3.
> Patches welcome!
>

Anything which provides additional information to assist the user in
identifying issues in their document is a plus in my view. 

Re: org-persist files in /tmp

[Tim Cross]

Max Nikulin  writes:

> On 22/12/2022 19:34, Ruijie Yu wrote:
>> One possible approach to this is to have all org-persist related
>> temporary directories into an overall "$TMPDIR/org-persist" directory.
>
> Predictable name in a "world" writable directory generally is not a good 
> idea. Multiple
> users may try to run Org on the same machine. There are some kernel 
> parameters to prevent
> certain type of attacks, however I am unsure concerning their default values 
> in various
> Linux distributions and what will happen if one user creates a symlink to 
> somewhere the
> under home directory of another one. So unfortunately a directory reusable by 
> different
> emacs sessions should be avoided.
>
> Ihor, I do not like that after your latest changes temporary directory became 
> world
> readable.
>
> Another point is that creating temporary files and directories must be an 
> atomic
> operation. In between of removing and recreating it an attacker might manage 
> to create a
> file with the same name.

Could some of the issues people are concerned about regarding use of
/tmp be avoided if instead the temporary files were put into ~/.cache?
To me, that would seem to be the appropriate location for such files. It
would mean that org would need to 'manage' or clean out old files, but
that shouldn't be a big issue.

Re: org-persist files in /tmp

[Tim Cross]

Ihor Radchenko  writes:

> "Fraga, Eric"  writes:
>
>> for some reason, I am now getting many (tens) directories of the form
>> org-persist-NN in /tmp.  These seem to include an index file and a
>> cache type sub-directory structure.  Why are these there and does
>> anything clean them up?
>>
>> I have nothing related to org-persist in my configuration that I can
>> see.
>
> If you run something like make test or emacs -Q + org, it is expected.
> These are throwaway directories used by org-persist for emacs -Q.
>
> Do we need to care about cleaning up /tmp?

Probably not - at least not on most modern Linux systems as these tend
to have a systemd task which will clean up the temp directories on
reboot. You can usually tweak the settings for systemd-tempfiles if you
want to modify when and how often temporary files are cleaned up.

Re: Recommended way to work on main without upgrading Org?

[Tim Cross]

Karthik Chikmagalur  writes:

> Hi,
>
> I'm trying to work on the main branch of Org, with the intent of creating a 
> patch.
> However, I need to continue using Org 9.5 for everyday work in a separate 
> Emacs session as
> I can't have things breaking.  Is there a recommended way to run two 
> simultaneous
> instances of Emacs using two different Org versions?
>
> I use Straight to install Org 9.5.  I've cloned the latest main branch from 
> Savannah to
> work on, but I'm not able to test my changes cleanly since there are two org 
> versions in
> the mix -- technically three including the built-in version that's been 
> shadowed.
>
> Karthik

Have a look at the chemacs2 package
https://github.com/plexus/chemacs2.git

With this package, you can define multiple Emacs profiles, making it
easy to start Emacs with different configurations or package
versions. For example, I have a number of different profiles, including
'default', 'experiment' and 'vanilla'.

My default profile is my normal day-to-day emacs configuration.
The 'experiment' profile is where I am trying out different packages or
different configurations. When there has been a major new release of a
package I rely on (such as org), I will usually upgrade in the
'experiment' profile and try it out to verify it has no major issues
before I install it in my default profile.

The 'vanilla' profile is a very basic and as close to 'out of the box'
configuration as I can do. I use it when I've discovered a bug and want
to both verify it isn't due to any of my configuration and provide
a minimal reproducible case which works with Emacs default
configuration.

All I have to do to use the non-default profile is start Emacs with the
command argument --with-profile= where  is the profile name
I want to run.

Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> Based on the information in section 17.13, how do I configure my Emacs
>> so that
>>
>> 1. All the code in the files I wrote just runs and doesn't bother me with
>> annoying execute questions.
>>
>> 2. Code in files from colleagues is shown to me and I'm asked if it
>> should be executed. Once I say yes, I don't want to be bothered about it
>> again i.e. next time I open that file, I want org mode to know I trust
>> it.
>>
>> 3. Absolutely no code in files which are not from the two preceding
>> sources is to be executed unless I explicitly approve it. 
>
> Not yet, but I hope that we can integrate the approach we use in
> `org-safe-remote-resources' and `org--confirm-resource-safe'.
>
>> It feels like what we currently have is a selection of disconnect knobs
>> which the user can tweak, but with no over-arching mechanism to help
>> manage these settings for various scenarios.
>
> Agree. I hope that we can slowly work towards connecting these knobs.
>
>> Finally, are we 100% certain that these different code evaluation
>> circumstances are the only ones? Can we be certain there isn't areas
>> where options or variables are set which couldn't be used to evaluate
>> code (similar to be previously identified execution of code in block
>> headers which occurred before the checks on code evaluation?)?  
>
> No, we can't. But it is true for any software that allows third-party
> code, not just for Org or even Emacs.
>

Yes, you are right there are no guarantees. However, I do feel that if
we are going to add this security layer, we also need to perform some
form of audit and documentation of the specific points where org
constructs will/can trigger evaluation of included code. As the bug with
rich text demonstrated, there will likely be corner cases we
miss. However, we should at least try to identify as many as possible
and document them (and include automated tests when possible).

One of the downsides regarding improved security controls is that it
raises the level of expectations. We need to try and ensure we meet
those expectations. What we really need are some good ethical hackers to
help us identify potential 'hot spots'. The ability to do this
effectively does involve a particular type of mind set and skills not
necessarily common amongst most users. 

> And we cannot use the more universal sandbox approach either. Not in
> Emacs.
>
>> It also feels like the approach we have taken here is almost a throwback
>> to a past era where he had a lot more trust. What we seem to have is
>> protection against accidental execution of code rather than protection
>> against code with malicious intent which has been crafted to be
>> difficult to spot and deliberately takes advantages of small 'holes' or
>> over-sight in the controls supplied.
>
> I do not think that we can do anything about crafted code in Emacs other
> than displaying that code and letting the user decide.
>

Agreed. What we need next is the ability to track those decisions so
that the user isn't nagged about the same things every time and the
ability to set different defaults based on some preference (such as
source/origin, location, etc. 

> And I haven't seen any better solution, except anti-virus scanners that
> are constantly fighting new malicious code.
>
> At least, we do show the code. It is already better than "yes/no"
> dialogue you get when running random executable on Windows.

Agree. The question is whether we actually do provide that y-n and
display in sufficient situations. 

Given the additional information you provided, I'm somewhat less
concerned.

These 'knobs' definitely do need some form of additional abstraction
which will more easily allow end users to set a basic policy regarding
the treatment of different org files or org files from different
sources. I suspect, in the long-term, we are likely to need some type of
file 'cookie' i.e. a local-variable setting which will either set the
policy for that file or set the origin/source/trust level etc so that whatever
security policy the user has defined can be applied. IN some ways,
similar to the 'cookie' placed into your custom variables when you say
you trust some new theme.

Re: [BUG] Org-9.6 declares compatibility with Emacs-25.1

[Tim Cross]

Timothy  writes:

> Hi Max,
>
>> Notice emacs-25.1 in the elpa package description:
>
> Yea, I’m pretty sure this is just an oversight. This should be bumped to 26.
>

Yep, that would be my assumption as well. Support is for the two
previous releases i.e. 27.x and 26.x.

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Tim Cross]

Tom Gillespie  writes:

> Treating src blocks missing a lang as paragraphs is
> incorrect because according to the syntax spec they
> are syntactically still blocks (greater or lesser depending
> on your inclinations).
>
> I think the general principle we want to follow here is
> that a block (or any entity in general) should not lose
> its type because some part of its syntax is malformed
> (I have made similar arguments about property drawers).
>

I think you might be right here.

The big characteristic of source blocks and example blocks which make
them different from paragraphs is the line breaks and indentation. If
they are treated as paragraphs, won't this information be lost?

Re: Failing to load, showing this 'Symbol's function definition is void: defvar-1'

[Tim Cross]

Sharon Kimble  writes:

> I unfortunately upgraded this morning to emacs-30.0.50, and since then I 
> can't get into my usual emacs of 29.0.50. 
>
> When I'm loading emacs-29.0.50 from /usr/local/bin/ it is consistently 
> failing to load
> saying "Symbol's function definition is void: defvar-1".
>
> My init.el is this -
> 
> ;;; init.el --- sharon's config -*- eval: (read-only-mode 1) -*-
> ;; Make sure that Git version of Org mode is being loaded instead of the 
> built-in version.
> (add-to-list 'load-path (expand-file-name 
> "/home/boudiccas/.emacs.d/elpa/org-9.5.5"))
> ;;;(add-to-list 'load-path (expand-file-name 
> "/home/boudiccas/.emacs.d/elpa/org-9.6"))
> (add-to-list 'load-path (expand-file-name 
> "/home/boudiccas/git/org-contrib/lisp"))
>
> (require 'package)
> (setq package-enable-at-startup nil)
> (package-initialize)
>
>
> (require 'ob-tangle)
> (org-babel-load-file "/home/boudiccas/.emacs.d/config22-2.org")
> 
>
> It seems to be baulking at the last 2 lines, can somebody help please?
>
> Thanks
>Sharon.

Emacs 30.0.50 is the bleeding edge of the development tree. It will be
unstable by definition. The error you are getting looks like an internal
Emacs error unrelated to org mode.

I would wait a day or so and pull new sources and re-build to see if the
issue is fixed. If not, log a bug report.

If a working Emacs is critical to your activities, I would revert back
to 28.2 or maybe 29, which is in pre-release state, so a little more
stable than the bleeding edge dev code. Note that there is an Emacs 29
branch, so you can checkout emacs-29 to get the most recent release
candidate for Emacs 29.

In general, org mode won't attempt to fix issues introduced in the HEAD
soruces of Emacs as these sources tend to be in flux and issues will
often be resolved by other non-org specific changes. Issues at this
level tend to be a 'wait and see'. 

Re: Bug: Function org-heading-components is not resilient [9.4.3 (9.4.3-elpa @ /home/data1/protected/.emacs.d/elpa/org-20201216/)]

[Tim Cross]

Jean Louis  writes:

> * Ihor Radchenko  [2022-12-17 12:59]:
>> The error looks like you attempted to run `org-heading-components' in
>> non-Org buffer. `org-heading-components' behaviour in non-Org buffers is
>> undefined.
>
> OK I can change it for my personal use, however, consider that
> function `org-heading-components' is useful to parse headings of Org,
> as Org markup too often comes in different other modes, for example,
> when I write e-mails, in mail-mode on this mailing list we have too
> often Org markup. 
>

There are specialised modes available for this sort of thing. See
https://orgmode.org/worg/org-tutorials/org-outside-org.html for some ideas.

> Also consider that many Org function work outside of the Org mode, I
> find it not consistent by design. for example, create
>
> * Heading
>
> and run
>
> M-x org-id-get-create

I think this is an unrealistic expectation. We have sufficient
challenges ensuring org functions work within org buffers without adding
the additional burden of expectation they would work outside these
buffers where there is no guarantee of syntax or formatting constraints.

If you find some org functions work outside of org buffers, that is just
happenstance. There is no inconsistency here. Many other modes have
functions which will also work to varying degrees outside the specific
mode for which it was written. That does not mean you should use them
outside the mode they were designed for. If you do use them, it is at
your own risk.

An expectation that a function will work outside the mode it was
designed for is a user error of expectation not a mode error. 

This error in understanding is likely due to the lack of real name space
support in Emacs. If we had real name spaces, org functions would not be
visible outside of org modes. Unfortunately, Emacs doesn't have such a
concept, so it is down to users respecting the conventions. One of those
conventions is not to use mode specific functions outside the mode they
were designed for. 

Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> I do wonder if it would be a good idea to try and document when org will
>> evaluate code in org files. This would include not just babel block
>> evaluation, but also elisp evaluation in table formulas, block header
>> arguments, file option arguments and possibly other subtle cases. This
>> may enable us to see if we have the granularity of controls correct or
>> identify inconsistencies and omissions. This information might then be
>> useful in defining a security model which could then identify what
>> controls are actually necessary and how to implement them to provide a
>> more straight-forward configuration for end users. It could also provide
>> valuable input into what additional tests may be necessary to ensure
>> things are working as expected. 
>
> 17.13 Code Evaluation and Security Issues

That page is a good start. However, I think it highlights why not having
a clear model limits the utility of the options being provided.

Consider this scenario -

I am a reasonably experienced Emacs user, but wouldn't describe myself
as proficient with elisp. I can do some basic stuff and I have an
init.el file which I basically understand, although there are some
borrowed bits which to be honest, I don't fully understand.

I use org mode a lot. I really like literate programming and find the
whole babel stuff really cool. I also make extensive use of tables and
the spreadsheet capability to track statistics on my projects and use
gnuplot, ditta and plantuml to generate plot, diagrams etc from that
data. Obviously, I trust all the code in these files as I wrote it!

On occasion, I get org files from colleagues. While I do basically trust
my colleagues, I don't want to just blindly allow execution of the code
in these files. I would like to check what the code does before agreeing
to allow it to run.

On very rare occasions, I get an org file from an untrusted source. In
this situation, I want iron clad assurances none of the code from this
file is executed.

Based on the information in section 17.13, how do I configure my Emacs
so that

1. All the code in the files I wrote just runs and doesn't bother me with
annoying execute questions.

2. Code in files from colleagues is shown to me and I'm asked if it
should be executed. Once I say yes, I don't want to be bothered about it
again i.e. next time I open that file, I want org mode to know I trust
it.

3. Absolutely no code in files which are not from the two preceding
sources is to be executed unless I explicitly approve it. 

How does this fictitious user achieve this configuration? Is it a
reasonable expectation that most users will be able to write the elisp
necessary to achieve this model?

It feels like what we currently have is a selection of disconnect knobs
which the user can tweak, but with no over-arching mechanism to help
manage these settings for various scenarios.

Finally, are we 100% certain that these different code evaluation
circumstances are the only ones? Can we be certain there isn't areas
where options or variables are set which couldn't be used to evaluate
code (similar to be previously identified execution of code in block
headers which occurred before the checks on code evaluation?)?  

It also feels like the approach we have taken here is almost a throwback
to a past era where he had a lot more trust. What we seem to have is
protection against accidental execution of code rather than protection
against code with malicious intent which has been crafted to be
difficult to spot and deliberately takes advantages of small 'holes' or
over-sight in the controls supplied.

Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

[Tim Cross]

Max Nikulin  writes:

> On 15/12/2022 19:25, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>> I would consider reverting the commit causing user prompt for every
>>> variable.
>> I disagree. If anything, we can set the default value of
>> `org-confirm-babel-evaluate-cell' to nil and apply this patch.
>> Then, we can get the old behaviour back yet allowing concerned users to
>> have more security.
>
> I am leaving it up to you. Form my point of view it will be dead code that 
> increases
> complexity with no practical outcome. Unfortunately setting
> `org-confirm-babel-evaluate-cell' to anything other than nil results in 
> annoyance rather
> than security.
>
> Perhaps advising `org-babel-execute-src-block' with `y-or-n-p' is a better 
> treatment for
> my paranoia.
>
>> This patch does not only affect src blocks. It affects all the users of
>> `org-babel-read'.
>
> Mostly it is called with INHIBIT-LISP-EVAL set to t.
>
>>> https://list.orgmode.org/Y1uFDWOjZb85lk+3@protected.localdomain
>>> Re: [BUG][Security] begin_src :var evaluated before the prompt to
>>> confirm execution
>> How is it related to the current discussion?
>> The purpose of the security feature discussed here is not for web
>> browsers or anything like that.
>
> I am not going to add malicious source blocks to my private org files. For 
> some code it is
> better to have a prompt, but generally the issue is not excessively 
> important. I tend to
> inspect org files fetched from net in some other application at first 
> (browser, less, or
> vim).
>

While I would argue that is a good practice, it isn't always practicable
for all users. For example, Emacs together with Emacspeak is my primary
accessible interface. While I could use a browser, it would be severely
inconvenient and frustrating.

I have to admit I also have some concerns here. These may or may not be
well founded. My biggest concern is that we don't seem to have a clear
security model. It feels like we are responding to security threats in a
piecemeal and reactive manner and without any clear model. As a result,
there is a risk we will end up with a complex solution where we don't
fully understand how all the separate pieces interact. This could result
in a complex configuration with a low level of confidence, two of
the worst attributes to have when it comes to security.




> Accidental evaluation of code is a real danger for those who insist on 
> opening links to
> org file directly in emacs or even propose to use org as a kind of browser. I 
> raised the
> security issue in response to passionate messages demanding direct ways to 
> work with org
> files from the web. I decided to remind the context with hope that it would 
> help to
> reevaluate severity of the issue.
>

My concern here is that given the power of link configuration, source
blocks, local variables, .dir-local, evaluated block header code etc, it
is extremely difficult to actually know when code will be executed.

One thing I do feel is a risk is that if we don't get the right balance,
the questions about code evaluation may fall to the level of annoying
noise which people get rid of by simply enabling all code evaluation
without question. Obviously, this would be a very bad security decision,
but as we know from years of experience, users will frequently
prioritise convenience over security. If they are unnecessarily 'nagged'
regarding code evaluation, I suspect this is what will happen (noting
that unnecessary is very subjective). 

> I do not have a better proposal, but I think I see movement in a wrong 
> direction.

I'm not sure if I have a better proposal either. I'm not even sure if
this is solely an org issue. It could be that Emacs itself needs a
clearer or better understood and explicit security model. This seems
particularly relevant given the growth in support for downloading,
installing and running code from packages distributed by repositories
with no assessment or vetting of code. I find it quite inconsistent that
when I download and install a new theme, I have to explicitly mark it as
'safe', but I can download a package from melpa, elpa etc with no
additional checks or without having to agree to allow access to whatever
service or data it wants. 

I do wonder if it would be a good idea to try and document when org will
evaluate code in org files. This would include not just babel block
evaluation, but also elisp evaluation in table formulas, block header
arguments, file option arguments and possibly other subtle cases. This
may enable us to see if we have the granularity of controls correct or
identify inconsistencies and omissions. This information might then be
useful in defining a security model which could then identify what
controls are actually necessary and how to implement them to provide a
more straight-forward configuration for end users. It could also provide
valuable input into what additional tests may be necessary to ensure
things are working as expected. 

Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

[Tim Cross]

Max Nikulin  writes:

> On 15/12/2022 16:31, Ihor Radchenko wrote:
>> The actual parser does allow empty lang in src blocks, setting :lang
>> element property to nil. Should we stop doing this and treat such src
>> blocks as paragraphs? Or should we allow empty lang and instead adapt
>> the exporters to treat empty lang correctly?
>
> Source blocks without language may be treated as #+begin_example blocks. I 
> believe, a
> warning should be issued in such case.
>
> I do not see a reason why caption is not allowed for examples.

Yes, I think I agree. Semantically, a src block without a language is
really just an example block - it cannot be executed or evaluated and is
essentially reduced to a example block.

I think having a warning is also a good idea as it will alert users when
they have inadvertently forgotten to add the lang.

I don't see any reason not to allow captions for examples either.

Re: org-insert-structure-template

[Tim Cross]

Anthony Carrico  writes:

> I'm trying to remember what the old keybinding was before it got switched to 
> 'C-c C-,'...

IIRC there wasn't one.


Previously, a completely different system was used for adding these
templates and it was bound to  <  (or was it >, I cannot remember).

The problem was that the old 'template' system was not terribly
flexible/powerful. For example, you could not mark a region, call tte
template function and have it wrap the template around the marked
region. 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> I guess we are limited by what the packages we rely on support. For
>> example, if geiser doesn't support Emacs 26 but org is supposed to,
>> there isn't much we can do. We cannot afford to fork geiser and modify
>> it to add the support and even if we provided a patch to add the support
>> to the geiser project, they may not merge it.
>>
>> Best we can do is best effort and reflect this limitation in the manual
>> where we stipulate what our backwards compatibility policy is. 
>
> Ok. Here are the tentative patches for Org manual and WORG maintenance
> page.
>

No objections from me. Just based on reading the patches, they appear to
clarify our support policy and therefore should be helpful. 

Re: Multiple versions of Org in load-path problem

[Tim Cross]

David Masterson  writes:

> Tim Cross  writes:
>
>> David Masterson  writes:
>>
>>> "Michel Schinz"  writes:
>>>
>>>> Just for the record, I also ran into problems when installing Org 9.6
>>>> using Emacs' package system on top of an older version that came with
>>>> Emacs. If I tried to install it as usual (M-x list-packages, then
>>>> install the package from there), I had errors during compilation related
>>>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>>>> fatal error in an unrelated package.
>>>>
>>>> I managed to solve that problem by:
>>>> 1. uninstalling Org 9.6 and exiting Emacs,
>>>> 2. starting Emacs with -q,
>>>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>>>> 4. restarting Emacs.
>>>
>>> Interesting!  I tried this (essentially) and it worked for my case.  In
>>> my case, I had a built-in Org-9.3 and I was trying to use list-packages
>>> to install Org-9.6. I checked that using -q still added Org-9.3 to
>>> the
>>> load-path, but, since Org wasn't loaded, the install via list-packages
>>> worked.
>>>
>>> The question is what's the proper way of doing this without '-q'?
>
> [...]
>
>> I don't think there is any safe way to install an updated version of
>> org-mode other than
>>
>> 1. Use the -q approach outlined above
>
> Thinking about it, this only works if Org is in elpa as melpa (etc.) are
> not added to package-archives.  You'd have to do some handwritten elisp
> out of *scratch* to setup package-archives if Org-9.6 was still coming
> out of melpa. That's why this can only be labeled as a hack and not a
> solution.
>

Well, yes, if your going to use this technique to load a package which
is not in the default package archives you would need to add that
archive first. People who use this technique often just have a
'update.el' file which they load/evaluate when starting Emacs with -q. 


>> 2. Craft your init.el file such that org functionality is only loaded
>> when explicitly requested and always update as the first action after
>> starting emacs.
>
> In this case, something happened in package-install when trying to
> install Org-9.6 with a built-in Org-9.3.  During the compilation check
> (.el -> .elc) many files failed because the new 'org-assert-version'
> macro was not defined.  Sort of like, after package-install started
> working on Org-9.6, org-macs.el (where org-assert-version should be) got
> loaded *before* the new load-path had been set causing it to load the
> old one from 9.3.  Thereafter, everything went awry. 
>

You must have some custom code in your init.el or are using something
like use-package as package.el doesn't try to install or upgrade
packages during init by default.

Where people can come undone is when they are using use-package and set
the :ensure t option. In this case, use-package will not know about the
bundled version and will attempt to install org from ELPA. If use
package runs after org has already been loaded (possibly because some
other package has been loaded which depends on/requires org mode and has
loaded the bundled version) then things will break because you will end
up with a mixed version install. This is why I always ensure org is the very
first use-package in my init.el and it comes before any other code which
loads or initialises anything.  

>> The first approach is actually the easiest. The second is hard to get
>> right and very fragile because packages like use-package and more
>> specifically, other packages with leverage off org functionality, make it
>> impossible to reliably know exactly when org is loaded.
>
> Using ':after" in use-package is supposed to help that, but I'm not sure
> it is reliable.  Packages are often incomplete about what other packages
> it depends on.
>

You cannot rely on :after. The problem is, other packages may require
org functionality and will load org when they are loaded. This can be
very subtle as there are a lot of packages which only make very small
use of org mode, but even that requires that org mode is loaded. 

>> An approach used by many 'canned' distributions is to postpone package
>> updates. You have a function you run to check for updates which
>> generates a list of packages to update and writes that list to a
>> file. Each time emacs is started, it looks for this update list and if
>> it finds it, it installs packages updates at the very beginning of the
>> init process (before any of your other init.el code or custom
>> blocks). The process also looks for org in the list of pac

Re: Multiple versions of Org in load-path problem

[Tim Cross]

David Masterson  writes:

> Adding this to bug #59882
>
> "Michel Schinz"  writes:
>
>> Just for the record, I also ran into problems when installing Org 9.6
>> using Emacs' package system on top of an older version that came with
>> Emacs. If I tried to install it as usual (M-x list-packages, then
>> install the package from there), I had errors during compilation related
>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>> fatal error in an unrelated package.
>>
>> I managed to solve that problem by:
>> 1. uninstalling Org 9.6 and exiting Emacs,
>> 2. starting Emacs with -q,
>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>> 4. restarting Emacs.
>
> Interesting!  I tried this (essentially) and it worked for my case.  In
> my case, I had a built-in Org-9.3 and I was trying to use list-packages
> to install Org-9.6. I checked that using -q still added Org-9.3 to the
> load-path, but, since Org wasn't loaded, the install via list-packages
> worked.
>
> The question is what's the proper way of doing this without '-q'?
>
>> I'm not sure this is related to your problem, or whether that helps (but
>> I hope it does)...
>
> I think it does.
>
> Side note:
>
> In my testing, I found a strange case where, in *scratch*, I get:
>
> (message "%s" org-version)
> ;; Error undefined
> ;; Do 'C-h v org-version'
> (message "%s" org-version)
> 9.3
>
> So, 'describe-variable' on org-version causes Org to be loaded?!?  Why
> do I have a feeling this is related to this bug?

I don't think there is any safe way to install an updated version of
org-mode other than

1. Use the -q approach outlined above

2. Craft your init.el file such that org functionality is only loaded
when explicitly requested and always update as the first action after
starting emacs.

The first approach is actually the easiest. The second is hard to get
right and very fragile because packages like use-package and more
specifically, other packages with leverage off org functionality, make it
impossible to reliably know exactly when org is loaded.

An approach used by many 'canned' distributions is to postpone package
updates. You have a function you run to check for updates which
generates a list of packages to update and writes that list to a
file. Each time emacs is started, it looks for this update list and if
it finds it, it installs packages updates at the very beginning of the
init process (before any of your other init.el code or custom
blocks). The process also looks for org in the list of packages to
update and if it is found, updates it first. 

I don't think there is a safe way to load org mode after the init
process i.e. after booting emacs by M-x package-update.

I've had good success using straight.el. I had to be careful regarding
how I structured my init.el file (ensuring any straight stuff happens
first and the first use package stanza is for org. The main reason
straight works well for me is that my work flow is to do a M-x
straight-pull-all when I want to update my packages. This does a git
pull for all the sources, but does not do any build/install. This occurs
when I next start Emacs and because I have all the straight stuff at the
start and because org mode is the first straight-use-package, the update
and install happens before any other org functionality is loaded,
avoiding mixed version issues.

Re: Release 9.6

[Tim Cross]

Max Nikulin  writes:

> On 29/11/2022 13:58, Bastien wrote:
>> Last but not least: thanks to Ihor his
>> truly amazing work and for being the de facto maintainer.
>
> I think, Ihor's role in this release is crucial. He spent a lot of time 
> fixing bugs and
> reviewing patches, not to mention the org-fold framework to overcome 
> performance
> limitation of overlays (the latter has been recently addressed in Emacs 
> though).

I agree. The amount of time and effort Ihor has put into org mode is
both crucial and hugely appreciated.

One question I do have is with respect to the new folding code, the
changes in Emacs to improve overlay performance and the correct way
forward.

On one hand, it was an immense amount of work for Ihor to implement a
better performing solution and something I'm sure those with large org
files will appreciate. However, on the other hand, I guess it also puts
a greater burden from a maintenance perspective on org maintainers as
org is now using its own unique approach to hiding/showing content
(folding).

Has anyone done any comparisons between the new overlay implementation
in Emacs 29 and the new folding approach in org 9.6? Is there still
sufficient performance benefit with org's approach over using Emacs
overlays or do we need to seriously consider changing the default back
to native Emacs overlays?

Re: User

[Tim Cross]

Paul Schlesinger  writes:

> Have been using org mode for more than 10 years and package manager since 
> emacs 26.  When I moved to emacs
> 28, the included org package was 9.4x and a nag message to install org from 
> the elpa repository appeared everytime
> an org file is processed.  I run emacs on windows and list-packages does not 
> show and "org" package to install
> although there are many "org-xxx" packages.  Suggestions appreciated!
>
> Paul H. Schlesinger MD, PhD
> Washington University School of Medicine
> Don't let your models of reality become confused with reality itself.

What is the setting for your package-archive variable? This is the
variable which tells Emacs what package archives to use. The default is

'(("gnu" . "https://elpa.gnu.org/packages/;)
 ("nongnu" . "https://elpa.nongnu.org/nongnu/;))

You need to ensure the above two entries are in the list for
package-archive. Note that you may also have other repositories, such as
melpa, but you should have the two above repositories as a minimum.

It is also possible you have something in your .emacs or init.el file
which is preventing package.el from performing correctly. There has been
some evolution/refinement in package.el since it was first released and
I've seen users have problems because of old configuration settings in
their init file which are no longer compatible.

These days, with modern Emacs versions, you don't need to have any
package.el configuration in your init.el to get basic operations to
work. Therefore, if your still encountering problems, I would start by
commenting out any configuration relating to package.el. The only
configuration code many people now have for package.el is code to add
additional ELPA repositories, like MELPA, to the package-archive list. 

Finally, note that the one big change which did happen wrt org is that
org is no longer distributed via a separate private org ELPA
repository. The org package is now distributed via GNU ELPA (first entry
in above setting) and the contrib package in NONGNU ELPA (second entry
above). If your configuration is old enough, you may have code which is
'pinning' Emacs/package.el to only use the old orgmode.org
repositories. Therefore, highly recommend you review your init.el.

Finally, as it has been a while since you updated, don't forget to go
through the NEWS file. There is likely to be changes that may be a
surprise or require adjustments to your work flow and being aware of
them can save a lot of time.

good luck.

Re: [MAINTENANCE] Org orphanage?

[Tim Cross]

Ihor Radchenko  writes:

> Dear all,
>
> This email is inspired by recent request by org-vcard maintainer to
> transfer maintenance to someone else:
> https://github.com/flexibeast/org-vcard
>
> As a part of Org project (https://sr.ht/~bzg/org/), we currently have
> https://git.sr.ht/~bzg/org-contrib/:
>
>org-contrib: Contributed packages to Org in search for new maintainers.
>
> There, we do a very minimal maintenance and encourage people to take
> over the containing packages.
>
> Should we extend the org-contrib's current idea to other Org-related
> packages that are seeking a maintainer? Similar to
> https://github.com/emacsorphanage
>
> We may temporarily use Org mailing list as a place to report bugs.
>
> - This will give a better chance for interested people to contribute,
>   and hopefully take over the work on abandoned packages.
>
> - We can also guarantee some (very) minimal maintenance.
>
> - And it will promote Org mailing list as a place to discuss Org-related
>   staff.
>
> WDYT?

I think it is an excellent idea. A rename to something like
org-orphanage would also be good as it makes the status of these
packages much clearer and may encourage adoption of those packages
people find useful.

Re: [RFC] Re: Headings and Headlines

[Tim Cross]

Ihor Radchenko  writes:

> Bastien  writes:
>
>> Ihor Radchenko  writes:
>>
>>> I know for sure
>>> that changing `headline' element to `heading' element type will break
>>> important packages like org-roam. And there is no good way to work
>>> around this. We cannot make symbol aliases in Elisp in scenarios like
>>> (memq (org-element-type ...) '(headline inlinetask)).
>>
>> We cannot make symbol aliases in Elisp but maybe we can support both
>> symbols for a transitory period during which we warn third-part devs
>> about replacing the deprecated 'headline symbol?
>
> The best idea I can come up with is the following:
>
> 1. We replace headline -> heading where it is safe
> 2. We introduce a new constant: org-element-heading-type, defaulting to
>'headline
> 3. We use the new constant instead of 'headline element type symbol
> 4. We announce loudly that 'headline will be deprecated in favour of the
>new constant
> 5. Few years later, we change the org-element-heading-type value to
>'heading
>
>>> I came to the conclusion that it will, in fact, be easier to change all
>>> things to use "headline" -- all the instances of "heading" in Org code
>>> are in function names, variable names, and docstrings. All can be
>>> changed using obsolete aliases.
>>
>> Given Vikas and Tim feedback, I would rather move forward by changing
>> "headline" to "heading" *where it does not break anything* then see if
>> the proposed scenario above is workable.
>>
>> In this case, I believe it's better to be partially correct (heading
>> where possible) than to be consistently wrong (headline everywhere) :)
>>
>> WDYT?
>
> I tried, but it will be confusing when we talk about Org elements.
> Phrases like "Headline element" now make sense as they correspond to the
> element type. Changing to "Heading element" while keeping the actual
> element as (headline ...) sounds extremely confusing.
>
> That said, we may do what I proposed above and then use
> "`org-element-heading-type' element". Somewhat cumbersome, but at least
> less confusing.

I think we are needlessly complicating this. We are talking about the
use of a term in an internal code base. While I would agree heading is
more correct, I don't think it is such a big issue to use headline if
that make the transition to a consistent usage easier. When it comes to
code, I think consistency trumps correctness.

If agreement is not possible, my second vote would be for the status
quo. Leave it as it is and focus on more important issues that have a
real impact on users. 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

>> It might be worthwhile defining what is meant by 3rd party packages.
>>
>> For example, ob-scheme relying on geiser as a 3rd party package is one
>> thing. Org roam is another type of 3rd party package. I think they need
>> different approaches. The first is about our (org) interface to them and
>> the latter is about their (org roam) interface with org.
>
> Sorry for not being clear.
> I was referring to third-party packages Org optionally depends on.
> Like geiser or citeproc or engrave-faces.
>
>> For the former (e.g. geiser type), I don't think backwards compatibility
>> is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
>> fairly trivial to update to the current version. We just need to support
>> the current version.
>
> Yes, but what if, say, the newest geiser version no longer supports the
> Emacs version Org still supports?
>

I guess we are limited by what the packages we rely on support. For
example, if geiser doesn't support Emacs 26 but org is supposed to,
there isn't much we can do. We cannot afford to fork geiser and modify
it to add the support and even if we provided a patch to add the support
to the geiser project, they may not merge it.

Best we can do is best effort and reflect this limitation in the manual
where we stipulate what our backwards compatibility policy is. 

>> For the latter (e.g. org-roam), backwards compatibility is much more
>> important. Org needs to provide as stable a public API for such packages
>> as possible. It may even be worthwhile separating the API into a
>> public/external API and an internal/private API. The former would be as
>> stable as possible and when changes are made, all efforts to provide
>> backwards compatibility are made. The latter is also stable, but more
>> subject to change. It is the API intended for internal org use. If
>> external packages use it, it is at their own risk. We would still try to
>> keep it stable, but with less of a guarantee and we may not provide
>> backwards compatibility if doing so was going to cause too much
>> complexity or maintenance burden etc.
>
> That's what we already do.
> I mean, https://bzg.fr/en/the-software-maintainers-pledge/ :)
>
> Public/private is the usual org-* vs. org--*.
> We never break org-*. Even if we do, we first deprecate things.
>

Yes, I know. I included it for completeness. However, I'm not convinced
worg is an effective communication channel for this information. We may
need to either add it to the manual or reference it from the
manual. (I'm reminded of the Douglas Adams quote from the Hitch Hikers
Guide [1]).

>> The big thing here is IMO avoiding surprise. We probably should add a
>> section to the 'Hacking' section of the manual which outlines our
>> position wrt backwards compatibility and API (public/private) and change
>> etc. I think most people expect change (even if they might not like
>> it). What they don't like is unexpected change. As long as we are
>> conservative and communicate change when necessary, we will likely be
>> OK.
>
> I hope that we never had unexpected changes. Isn't it for granted? I
> felt like it was always the case in Org and Emacs forever.
>
> It feels so obvious that I cannot even figure out how we could clarify
> it in the manual.

The one thing we can expect is unexpected change!

What is expected by one can be unexpected by others and what may seem
obvious to you or me may not be as obvious to others. We should be
extremely careful about assumptions regarding what can be taken for
granted. Org mode has a very broad user base. We have users with varying
familiarity with Emacs, software development practices and basic
software lifecycle management. For example, I know that symbol names
with '--' in them tend to indicate private API elements. However, that
is because I've been using Emacs for a long time. Someone new or someone
who is not a programmer may not be familiar with this convention.

There has been multiple occasions where I've seen posts on this list
from users complaining about unexpected and breaking changes. Such posts
usually end up in a thread about all the things org developers do to try
and mitigate such things. This would indicate it isn't necessarily taken
for granted by all. Often, those complaining were unaware of changes
being documented in the NEWS file or failed to check it before upgrading
or were unaware of backwards compatibility policy or didn't understand
the extent to which org relies on other external packages/services etc.

Then there is the situation where a change has unforeseen
impact. Consider the change a while ago with electric-indent.

I agree that in general, the org development process is pretty good in
its approach to handling change and maintenance of compatibility. Where
we are perhaps a little weak is in ensuring we communicate this
effectively. 

Perhaps we don't need to do anything extra. However, I feel it would be
beneficial to be very explicit regarding 

Re: [MAINTENANCE] Do we have any backwards-compatibility policy for third-party packages?

[Tim Cross]

Ihor Radchenko  writes:

> Hi,
>
> Org promises to support the last three Emacs releases.
> However, it is less clear what is our policy wrt third-party packages.
>
> We do need third-party packages, for example, in babel backends.
> Sometimes, we have to make changes to the ob-*.el files in order to
> accommodate changes in the required third-party packages. Like in recent
> changes to ob-scheme where `run-geiser' function is now obsolete
> upstream.
>
> Should we try to support obsolete functions/variables in third-party
> packages? Should we try to work around breaking changes and support both
> before/after package versions?
>
> The answer is not obvious as older Emacs versions might not be supported
> by some third-party packages. Then, logically, we have to support older
> package versions compatible with the oldest Emacs version we support.
> But it might be hard to keep track of such scenarios.
>

It might be worthwhile defining what is meant by 3rd party packages.

For example, ob-scheme relying on geiser as a 3rd party package is one
thing. Org roam is another type of 3rd party package. I think they need
different approaches. The first is about our (org) interface to them and
the latter is about their (org roam) interface with org.

For the former (e.g. geiser type), I don't think backwards compatibility
is as important. Thanks to package.el and GNU ELPA/NONGNU ELPA, it is
fairly trivial to update to the current version. We just need to support
the current version.

For the latter (e.g. org-roam), backwards compatibility is much more
important. Org needs to provide as stable a public API for such packages
as possible. It may even be worthwhile separating the API into a
public/external API and an internal/private API. The former would be as
stable as possible and when changes are made, all efforts to provide
backwards compatibility are made. The latter is also stable, but more
subject to change. It is the API intended for internal org use. If
external packages use it, it is at their own risk. We would still try to
keep it stable, but with less of a guarantee and we may not provide
backwards compatibility if doing so was going to cause too much
complexity or maintenance burden etc.

The big thing here is IMO avoiding surprise. We probably should add a
section to the 'Hacking' section of the manual which outlines our
position wrt backwards compatibility and API (public/private) and change
etc. I think most people expect change (even if they might not like
it). What they don't like is unexpected change. As long as we are
conservative and communicate change when necessary, we will likely be
OK.

Re: [RFC] Re: Headings and Headlines

[Tim Cross]

Ihor Radchenko  writes:

> André A. Gomes  writes:
>
>> The project's documentation refers to headings and headlines as
>> synonyms.  Relying on a single definition would be beneficial.  If I had
>> to choose between the two, I'd go with heading.
>
> I've been looking into changing all the instances of "headline" to
> "heading" and I ran into a serious issue: We use `headline' _symbol_ in
> multiple places in the code.
>
> Most importantly, org-element.el uses element type `headline' to parse
> headings. We cannot easily change this symbol for backwards
> compatibility reasons.
>
> I'm afraid that a complete switch to use "heading" everywhere consistently
> is not possible without backwards-incompatible change. I know for sure
> that changing `headline' element to `heading' element type will break
> important packages like org-roam. And there is no good way to work
> around this. We cannot make symbol aliases in Elisp in scenarios like
> (memq (org-element-type ...) '(headline inlinetask)).
>
> I came to the conclusion that it will, in fact, be easier to change all
> things to use "headline" -- all the instances of "heading" in Org code
> are in function names, variable names, and docstrings. All can be
> changed using obsolete aliases.
>
> On the other hand, overwhelming feedback in this thread is the
> opposite -- change "headline" to "heading".
>
> Maybe others have better ideas how to deal with `headline' symbol issue?

I think consistency is the highest priority. Most people preferred
heading, but that was based on the assumption adopting one or the other
was of roughly equal complexity.

Given we cannot change headline to heading without introducing backwards
and external compatibility issues, I would favour just changing to
headline and documenting what deadline defines/means.

Re: Bug: html-postamble string does not allow space [9.4.4 (release_9.4.4 @ /usr/local/share/emacs/27.2/lisp/org/)]

[Tim Cross]

Ihor Radchenko  writes:

>>
>> This is all an aside to the actual bug, so please don't miss that
>> (unless it has been fixed) i.e. a org-html-postamble string with a space
>> in it does not work.
>
> I am confused here.
>
> The original bug talked specifically about situation like
>
> #+options: html-postamble:"test with spaces"
>
> Are you saying that you cannot have spaces in org-html-postamble
> variable? If yes, could you please provide a reproducer?
>

To be honest, it was so long ago when I looked into verifying this
issue, I no longer recall the precise details. My memory was that just
having a space in the footer triggered the issue - it didn't have to be
only when the value was set via #+options, but I could be wrong. If you
cannot reproduce the bug just using spaces set, for example, with a
setting in org-publish-project-alist, then I'd say the issue is
resolved.

My main concern here was that it wasn't clear whether the underlying
issue had been addressed and while the doc improvements are great, I
didn't want the actual triggering issue to get lost amongst all the
rest.

Tim

Re: org-assert-version considered harmful

[Tim Cross]

"Cook, Malcolm"  writes:

> Hello,
>
> I found this recent thread researching why my strategy for staying abreast 
> with org head had started failing with invalid-function
> "org-assert-version"
>
> My strategy had been to build org initially with
>
> ` cd ~/.emacs.d && git clone 
> https://git.savannah.gnu.org/git/emacs/org-mode.git &&   cd org-mode && make 
> autoloads && make
> ` 
> and ensure this clone of org was picked up in my "~/.emacs.d/org-mode/lisp by 
> including the following in my .init.el very early
> (right after bootstrapping the package system and use-package):
>
> (use-package 
>
>   :pin manual 
>
>   :load-path "~/.emacs.d/org-mode/lisp"
>
> )
>
> Then, when I occasionally wished to update org, I would
>
> `cd ~/.emacs.d/org-mode && git pull && make autoloads && make`
>
> Recently I started getting errors invalid-function "org-assert-version".
>
> Upon cursory reading of this thread I guessed that I could fix them by adding 
> a `make clean` to my update mantra.
>
> It worked.
>
> Am I advised to do otherwise?  Is there a best/better practice?
>

I think it is good practice to always do make clean for any code you
build from sources yourself. There is a reason most Makefiles have a
'clean' target and when it comes to building software, starting from a
known clean state is critical. This can make the build slower, but for
small packages like org mode, the difference is insignificant. Always
safer to do make clean before make. Alternative approaches really only
necessary in larger and more complex source trees where there can be
significant time differences between full and incremental builds
(i.e. Emacs source tree has 4 different 'clean' targets; clean,
extraclean, distclean and bootstrap).

Re: Org babel API (was: [FR] Display stderr contents after executing shell blocks (even when stdout :results output is requested) (was: Org 9.6-pre and Bash sessions))

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> For me, this is another symptom of our need to define a clearer model
>> for babel processes so that we can get some consistency across the
>> board. Such a model would likely also make it easier for people to
>> develop new babel back ends. We may even need 2 models, one for
>> interactive/repl based back ends and one for non-interactive/scripted
>> back ends. 
>
> We do have two models already.
> We have some small bits documented in
> https://orgmode.org/worg/org-contrib/babel/languages/index.html
>
> Generally, the defaults are implemented in ob-core.el.
> Session API is in ob-comint.el. Shell command API is in ob-eval.el.
>
>> The 2 big questions I see are "What should be defaults be?" and "How do
>> we handle the options without adding lots of new switches or suffering
>> an option permutation blow out?".
>
> I am sorry, but I do not fully understand what you are talking about.
> Which defaults are you referring to?

In general, all the defaults for babel blocks, but morfe specifically
here, deffault handling for stderr.

Re: Bug: html-postamble string does not allow space [9.4.4 (release_9.4.4 @ /usr/local/share/emacs/27.2/lisp/org/)]

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> What probably needs clarifying is
>>
>> 1. mention the string option in the manual
>
> It is documented in the manual: 13.9.4 HTML preamble and postamble
>

You snipped out the relevant paragraph I copied from the manual. If you
look at it, you will see that it does not mention string as an option
for org-html-postamble. Furthermore, the previous paragraph, which talks
about org-html-preamble, which does mention string, states that if the
string matches the name of a function, it will be called 9and expected
to return a string). This is not mentioned at all in the doc string for
org-html-postamble.

Therefore, I still think there is inconsistency between what the doc
string of the variable states and what the manual states. There is no
mention of function matching against strings in the cod string of the
variable. There is no mention of string as an option in the paragraph
referring to org-html-postamble in the manual. 


>> 2. If spaces are not allowed in the string, clearly document
>> that. Currently, the doc string just says that if set to a string, use
>> that string as the postamble, which I think implies spaces are OK. 
>
> They are allowed. It is just #+OPTIONS keyword that does not allow
> strings as values. I have no idea why.

This is all an aside to the actual bug, so please don't miss that
(unless it has been fixed) i.e. a org-html-postamble string with a space
in it does not work.

Re: Bug: html-postamble string does not allow space [9.4.4 (release_9.4.4 @ /usr/local/share/emacs/27.2/lisp/org/)]

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> Confirm.
>>
>> I am able to reproduce this issue with 
>
> I might be missing something, but does
> `org-export--parse-option-keyword' even support spaces inside values?
>
>> I also feel the manual page could be improved as it doesn't actually
>> mention setting the html-postamble to a string (that is only mentioned
>> in the variable docstring).
>
> The manual does even mention the "html-postamble" option. Just the
> variable. I am confused. Was it removed at some point? (I do not see any
> commits doing so)

Sorry for late response. Only just noticed this amongst all the items in
my org mailbox.

It was a while ago. I do recall I was able to reproduce the issue, so
there was a bug here. Either it is a code bug or a documentation bug.

My error was in omitting the leading org- for the variable name. Correct
name is org-html-postamble. Apologies. 

With regards to the documentation, the section in the manual is

   The default value for ‘org-html-postamble’ is ‘auto’, which makes the
HTML exporter build a postamble from looking up author’s name, email
address, creator’s name, and date.  Set ‘org-html-postamble’ to ‘t’ to
insert the postamble in the format specified in the
‘org-html-postamble-format’ variable.  The HTML exporter does not insert
a postamble if ‘org-html-postamble’ is set to ‘nil’.

Note no mention of string, although the previous paragraph talking about
org-html-preamble does mention string.

and the doc string for the variable is

Non-nil means insert a postamble in HTML export.

When set to ‘auto’, check against the
‘org-export-with-author/email/creator/date’ variables to set the
content of the postamble.  When set to a string, use this string
as the postamble.  When t, insert a string as defined by the
formatting string in ‘org-html-postamble-format’.

When set to a function, apply this function and insert the
returned string.  The function takes the property list of export
options as its only argument.

Setting :html-postamble in publishing projects will take
precedence over this variable.

What probably needs clarifying is

1. mention the string option in the manual

2. If spaces are not allowed in the string, clearly document
that. Currently, the doc string just says that if set to a string, use
that string as the postamble, which I think implies spaces are OK. 

Re: [FR] Display stderr contents after executing shell blocks (even when stdout :results output is requested) (was: Org 9.6-pre and Bash sessions)

[Tim Cross]

Ihor Radchenko  writes:

> Rudolf Adamkovič  writes:
>
>> Ihor Radchenko  writes:
>>
>>> I do not think that it make sense to display that buffer when the code
>>> finishes successfully. I can see this kind of behaviour
>>> breaking/spamming automated scripts or export---code working in the
>>> past may throw error output into unsuspecting users.
>>
>> But the exit code has nothing to do with the standard error.
>>
>> Unix programs can, and often do, halt with non-zero exit codes while
>> producing error output containing important information, such as
>> deprecation warnings.  Further, many programs use error output as the
>> alternative "anything but the result" stream.
>>
>> Preserving user data, instead of trashing it, data does not count as
>> "spamming ... unsuspected users".  On the contrary!
>>
>> For example, I use a program for work that uploads data to a certain
>> 3rd-party server.  It exits with a zero code but also shows extremely
>> important notices on error output.  As an "unsuspecting user", if I used
>> Babel to run the program, I would end up in a trouble.
>>
>> So, we should never implicitly trash user-generated data, let alone
>> based on a "completely made up" belief that a non-zero exit code somehow
>> implies "no important error output".  It does not.
>>
>> (I speak only about Unix-like systems here.  Perhaps on other operating
>> systems, things work differently.)
>
> Dear All,
>
> As explained in the above quote, it may be reasonable to display stderr
> in the shell (and possibly other) src blocks upon execution.
>
> + Stderr may contain important information even if the code block
>   succeeds
> - Displaying stderr will raise *Error* buffer, which may or may not be
>   expected or desired.
>
> What do you think?

I think this is perhaps the tip of a more complex iceberg. Not sure if
we can address bits individually. Suspect we are better off
clarifying the basic babel model and then look at specific language
exceptions/limits and adjusting the model or documenting where back ends
need to diverge from the basic model.

Part of the challenge here is that the relevance/importance of stderr is
somewhat dependent on the language. Same holds true for handling of
error/return codes and to 'results' output.

For me, this is another symptom of our need to define a clearer model
for babel processes so that we can get some consistency across the
board. Such a model would likely also make it easier for people to
develop new babel back ends. We may even need 2 models, one for
interactive/repl based back ends and one for non-interactive/scripted
back ends. 

The 2 big questions I see are "What should be defaults be?" and "How do
we handle the options without adding lots of new switches or suffering
an option permutation blow out?".

I do agree with the other posts regarding the importance of stderr.

Re: Auto detect ob-clojure backend (was: [PATCH] Fix ob-clojure handling source block variable's value is a org-mode table or list)

[Tim Cross]

Daniel Kraus  writes:

> Ihor Radchenko  writes:
>
>> Daniel Kraus  writes:
>>
>>> +(defcustom org-babel-clojure-backend (cond
>>> +  ((executable-find "bb") 'babashka)
>>> +  ((executable-find "nbb") 'nbb)
>>> +  ((featurep 'cider) 'cider)
>>> +  ((featurep 'inf-clojure) 
>>> 'inf-clojure)
>>> +  ((featurep 'slime) 'slime)
>>> + (t nil))
>>
>> What if users have, say, cider installed and also babashka executable?
>> Will it be expected to use babashka?
>
> Yes. The only thing that makes me slightly hesitant is that e.g.
> someone doesn't have `bb` installed. Executes clojure source blocks
> which are then evaluated in, let's say cider.
> Then they install `bb` and the next time they start Emacs, the same
> source block on re-evaluation would be executed with babashka.
>
> I think this is still the best out of the box experience as it
> "just works" for most users without having to customise something
> and if they want it fixed, they can pin it to a certain backend.
>
> What's your opinion?
>
> Cheers,
>   Daniel

Just chime in with my 2 cents.

I think bb is a much better solution from a babel perspective and would
love to see it as the default, even when you have both bb and cider
installed.

I stopped using clojure in org because it was way too fragile -
depending heavily on cider features which often changed. However, now we
have babashka and nbb, I'm thinking about using org again with clojure.

I recall looking at the babel code for clojure some time back to see if
it could be made simpler and more reliable. However, there wasn't much
that could be improved on given the design of cider and its focus on
interactive clojure development. I then thought using something like the
Clojure CLI tools might be the way to go. Now I feel that babashka for
clojure and nbb for clojurescript might be the right answer. 

Re: Error opening an .org file

[Tim Cross]

Juan Manuel Macías  writes:

> Renato Pontefice writes:
>
>> I’ve edited and commented all the lines of it. And The error
>> persist. So (was you told)maybe the error is not of init.el
>> But I’m unable to run emacs from termnal.
>> How can I do?
>> I’m in Mac osx 12.6.1 where do I can find emacs?
>
> Renato, I'm not a Mac user, but I imagine it will be like any other
> Unix: you have to launch a terminal from your desktop (i guess there
> will be in macos an app called "terminal" or something like that), and
> once inside the terminal, just type:
>
> emacs --debug-init
>
> and press enter. Emacs (GUI) will open and you should pay attention to
> the error messages that Emacs will show you. That will give you a clue
> as to where the error is in your Emacs startup.
>

It has been a while since I used macOS. However, how you start emacs in
a terminal depends on how emacs was installed and which version of
emacs.

Note that the emacs which macOS includes is VERY old (I think it is
Emacs 21). This is really too old to be useful these days. You need to
install a current version of Emacs. My recommendation would be to use
homebrew to do this. However, I fear, based on the questions ask, the
OPs familiarity of the OS is likely to make installing homebrew and then
emacs a bit challenging to do via email. Certainly would be off topic
for the org mode list. Possibly better help would be available via the
emacs help list. I would try to help further, but I don't have a working
macOS system at present, so cannot refresh/verify the steps to get
sufficient clarity to be helpful. 

Re: bug#58774: 29.0.50; [WISH]: Let us make EWW browse WWW Org files correctly

[Tim Cross]

"Dr. Arne Babenhauserheide"  writes:

> [[PGP Signed Part:Undecided]]
>
> Ihor Radchenko  writes:
>
>> If necessary, we can introduce a special variable in Org mode that will
>> disable all the potential third-party code evaluation, even if user has
>> customized Org to execute code without prompt.
>
> If that would be part of org-mode, this would be close to a
> safe-org-mode.
>
> An important part in what I wrote about safe-org-mode is that it has to
> ensure that what is shown cannot trick the user into thinking something
> else would get run.
>
> A way to reduce risk would be to introduce a domain-allow-list (or
> prefix-allow-list) in eww for filetypes that could be unsafe, so you
> could for example add "orgmode.org" to your allowlist and for those
> domains org-files would auto-open in org-mode.
>
> Such security risks have a tendency of getting weaponized down the road
> when they really hurt. Like when people didn’t care about npm
> dependencies and had them suddenly deleting their files. And opening in
> the currently used Emacs may give a malicious file access to remote
> files opened via tramp, even if you (by virtue of being careful) require
> a password for the connection to sensitive servers. That way, running
> something in Emacs can be even more dangerous than running it in the
> shell.
>

and people constantly use M-x package-install to install packages
from GNU ELPA, nonGNU ELPA and MELPA, often with this misguided belief
that these packages are being vetted by the security fairies. 

As was seen after the openssl security failures, just because lots of
people use something and just because lots of people may work on and
look at the code, it is no guarantee the code is secure or has no
malicious content. Our systems have become far too complex for such ad
hoc processes providing any assurance. Likewise, as has been shown with
NPM and various browser 'extension stores', packages which were once
trustworthy can easily become owned/developed by new parties with less
ability or are less trustworthy. 

While adding the sorts of controls you outline is not a bad idea, I
think it is far more important to train people to accept that their
system simply is not secure. You should start from the position that
Emacs is not secure. Why? Because it is a large, complex and powerful
piece of software which has no formal security analysis or testing and
is usually augmented with numerous packages of unknown quality from
largely unknown sources. Essentially, Emacs already suffers from all the
same issues identified for systems like node and the NPM ecosystem. 

The only think which is really providing protection for us Emacs users
is that the rewards for compromising Emacs are too low for the effort
required. Similar to why you don't see many viruses on macOS - it isn't
that it is significantly more secure than Windows (these days), but
rather the pool of potential 'targets' and scale of rewards are higher
when you focus on the Windows environment. It is all about return on investment.

Security is a huge challenge for open source. The effort and resources
required to constantly analyse and test projects for security issues is
too high for most medium to large projects. The fact many open source
projects also rely on other open source projects for various libraries
etc also means that in addition to worrying about the security of the
code in a project, the project also has to worry about 'supply chain'
security i.e. ensure the external project dependencies are also secure
and securely managed.

So what do we do? In the famous words of Douglas Adams "Don't Panic!

Rather than worry if some package or change will make Emacs less secure,
assume it already is insecure and then examine how you use it and
consider both the likelihood of being compromised and the impact when
that occurs. This will differ depending on who you are and what you
do. For example, if your a researcher working in a field where your
research has high commercial value or a journalist working in countries
with a poor human rights history and government parties may want to
compromise your sources etc, both the likelihood and consequences could
be high and you may need to take additional measures or modify how you
use emacs (e.g. only use packages you have reviewed and tested and only
update after formal review and testing of updated version, don't use
Emacs for email or web browsing, only run emacs in an isolated locked
down container etc). On the other hand, if your just a keen hobbyist,
the likelihood and consequences of a security breach are both likely low
and you may decide adding additional packages is an acceptable risk.
Even if you decide your risks are low, you may still decide to not use
Emacs for some purposes. For example, you might decide not to use Emacs
for password management or not use Emacs packages which require you to
keep sensitive data (toekns, passwords, API keys etc) using insecure
mechanisms etc. Keep in mind 

Re: bug#58774: 29.0.50; [WISH]: Let us make EWW browse WWW Org files correctly

[Tim Cross]

Stefan Kangas  writes:

> Ihor Radchenko  writes:
>
>> The "problem" with shell links you are describing is a question of
>> setting variables and is also disabled by default.
>>
>> eww-mode, when loading Org page, could simply set
>> org-link-shell-confirm-function to its default value.
>
> Note that with the suggested feature, any link you follow risks being
> loaded in Org mode, before the user even has a chance to inspect the
> file.  Which Org features, currently existing or introduced in the
> future, would EWW have to add workarounds for?
>
> It is very hard to foresee which parts of Org will be problematic and
> have to be disabled.  See the security vulnerability in enriched-mode
> that prompted the release of Emacs 25.3, for example.
>
> Adding this opens a can of worms that will expose unsuspecting users to
> a whole class of new problems.  And the only benefit is to save some
> users from having to type "M-x org-mode RET", or adding call to a
> suitable hook.
>
> All in all, this seems like a bad trade-off.  So I don't think we should
> add such a feature.

This concern seems to be based on FUD rather than any real or identified
risk.

The risk here is no different to risks associated with opening any org
document from a foreign source e.g. in an ELPA package. Note that org
mode's default configuration wrt code execution is to ask the user for
permission to execute. If the user can run M-x org-mode on eww buffer
containing a text file which is an org file, the same risks apply and
any vulnerability would need to be addressed anyway.

This is also very different to the issues encountered with enrich text
some years back. The problem then was with elisp code embedded in text
properties. It was a bug in how enriched text processed the data. 

However, I think we are probably looking at this problem from the wrong
level. It isn't really about how to get eww to render org files in
org-mode. This issue is really about being able to customize what
function is called to 'render' the data retrieved based on the
content-type header of the content.

Currently, it is fairly straight-forward to define a custom handler
based on the URL, but not based on content-type. Being able to easily
associate a function to handle downloaded content based on the
content-type would be useful. Users could then easily add whatever
functionality they wanted based on what the server told them about the
content type. 

Re: [BUG] Undocumented convention for org-time-stamp-custom-formats to be "<...>" (was: time-stamp in DONE tag is not really displayed)

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>>> I propose to do the following:
>>> 1. org-time-stamp-formats and org-time-stamp-custom-formats will be
>>>treated as is, unless they contain "<" and ">" and the first and the
>>>last char.
>>> 2. If the formats do contain <...>, strip the "<" and ">".
>>> 3. Document (2) in the docstrings.
>>>
>>> Any objections?
>>
>> Little unsure/confused regarding what is being proposed here.
>>
>> - If we are removing <...>, does that mean we just retain [..] for
>>   inactive timestamps and all other timestamps are 'active' by default?
>
> org-time-stamp-formats is a constant. Users are not supposed to change
> it. So, the proposed stripping of "<" and ">" has a pure purpose of not
> breaking third-party code that might be using its current default value.
> It is more about some internal code assumptions.
>
> org-time-stamp-custom-formats currently has the following docstring:
>
> Custom formats for time stamps.  See format-time-string for the syntax.
> 
> These are overlaid over the default ISO format if the variable
> org-display-custom-times is set.  Time like %H:%M should be at the
> end of the second format.  The custom formats are also honored by export
> commands, if custom time display is turned on at the time of export.
>
> There is nothing in the docstring indicating that "<" and ">" are
> required or used in any way. The current Org code assumptions only
> create confusion among the users.
>
> I propose to strip "<" and ">" just to avoid breakage if users happened
> to set org-time-stamp-custom-formats to the value with "<...>" that is
> required to make this defcustom working in older Org versions.
>
> If users abused the undocumented behavior and used "[...]", we do not
> need to worry as it is out of what was promised.
>
> At the end, old user settings of org-time-stamp-custom-formats should
> remain working within docstring promises.
> Old user code using org-time-stamp-formats constant should also remain
> working.
> But users will not need to provide brackets in
> org-time-stamp-custom-formats after the proposed change.
>
> The above is the most safe way to retain backwards compatibility while
> removing the existing confusion with the docstring.
>
>> - How will this change impact code which distinguishes active/inactive
>>   timestamps based on presence/absence of <..> and [..]?
>
> org-time-stamp-formats value will not change.
> org-time-stamp-custom-formats value had no impact on buffer text---just
> the overlayed timestamp display. No code should be affected.
>
>> - What impact will this have on existing org files?
>
> None.
>
>> - Will this cause issues in parsing when you may have dates/times which
>>   are not supposed to be timestamps, but will look the same as
>>   timestamps? How will you distinguish them?
>
> No buffer text will be affected.
>
>> Personally, I like the clear distinction between what is a timestamp and
>> what isn't and what is an active timestamp and what is an inactive
>> timestamp.
>
> There is nothing about active/inactive timestamps in the discussed
> variables. The usage of "<" and ">" is historical and mostly ignored by
> Org code. First and last characters are simply stripped, and the required
> brackets are being used when inserting the actual timestamps.
>
> The proposed change simply aims to remove the undocumented requirement
> about brackets.

OK, thanks for clarifying. Based on this, I don't see any issues with
your proposed change.

Re: How ro delete DONE attemps

[Tim Cross]

Nick Dokos  writes:

> Renato Pontefice  writes:
>
>> Hi,
>> I’m wondering how can I delete, on my .org file, the line that have:
>> - an old Timestamp (i.e. if I set a thing to be done today (<2022-10-24 Mon 
>> 17:26>)
>> - a TODO item (always with a past date) <2022-10-24 Mon 17:26>
>>
>> That to have a more clean .org file.
>>
>> Is it possible?
>> How can I obtain it?
>>
>
> It's just text: you can use ordinary Emacs commands to modify the
> file, e.g. C-k will kill a line, or you can use C-d to delete
> characters or M-d to delete words.

The other thing you can do is use org's built-in archiving support. With
this, you can have 'old' items which are no longer relevant moved to a
separate archive file, creating a cleaner org file and keeping an
archive record of past items.

Re: GPL violation related to Org by priprietary app (was: Best android app)

[Tim Cross]

Ihor Radchenko  writes:

> Jean Louis  writes:
>
>>> If it is true, could you please provide links to legal basis on from
>>> GPL's and Swedish law's points of view?
>>
>> In my opinion the term "Org Mode" is collective trademark:
>>
>> Trademark FAQs | USPTO:
>> https://www.uspto.gov/learning-and-resources/trademark-faqs#type-browse-faqs_1934
>> valid only for US jurisdiction.
>>
>> Trademarks need not be registered, especially when it is clear who was
>> first using it, there is no doubt that Org Mode is term used to
>> promote software and software is commercial subject.
>>
>> Those who started using first "Org Mode", like author or whoever is
>> assigned to it, are free to tell to the website, to stop using it, or
>> demand part of their profits in Sweden.
>
> If GPL is not violated I see no reason for current Org maintainers to
> bother. I'd better focus on improving Org rather than trying to engage
> into legal complexities.

Even if it was violated, this is not something the maintainers are
empowered to act on anyway. Org mode is part of Emacs and the FSF owns
the copyright. If there is any GPL violation, the FSF has a whole legal
team which deal with such matters. It certainly isn't something
maintainers or arm chair lawyers are able to address.

Also, based on my limited legal experience and past dealings with
trademarks, copyright and licenses, I don't think there has been either
a GPL license violation or a trade mark violation.  However, if someone
believes differently, they should refer the matter to the FSF legal
office.

Re: [BUG] Undocumented convention for org-time-stamp-custom-formats to be "<...>" (was: time-stamp in DONE tag is not really displayed)

[Tim Cross]

Ihor Radchenko  writes:

> Uwe Brauer  writes:
>
>> My time-stamps are of the form <2022-10-23 Sun>
>> I have an entry like this 
>>
>> - State "DONE"   from "WAIT"   [2022-10-23 21:06] \\
>>
>>
>>
>> However it is displayed when I use org-toggle-time-stamp-overlays  as
>>  [23.10.%]
>>
>> Neither the year not the time is displayed, why!
>>  
>> I have set 
>> org-time-stamp-custom-formats 
>> to 
>> (" %d.%m.%Y " . " %d.%m.%Y")
>>
>> I am puzzled, any ideas?
>
> Confirmed.
>
> This is because Org expects the first and the last characters in
> org-time-stamp-custom-formats to be opening/closing brackets.
> (undocumented)
>
> Why?
> Because org-time-stamp-formats does so.
>
> Why does org-time-stamp-formats does so?
> No idea.
> This code dates back to initial Org commits.
>
> I think it would make sense to change it.
> However, if we change special treatment of the first/last characters in
> org-time-stamp-custom-formats, it will also make sense to change
> org-time-stamp-formats constant.
>
> For backwards compatibility, we will need to keep special treatment to
> strip brackets around the formats, if present.
>
> I propose to do the following:
> 1. org-time-stamp-formats and org-time-stamp-custom-formats will be
>treated as is, unless they contain "<" and ">" and the first and the
>last char.
> 2. If the formats do contain <...>, strip the "<" and ">".
> 3. Document (2) in the docstrings.
>
> Any objections?

Little unsure/confused regarding what is being proposed here.

- If we are removing <...>, does that mean we just retain [..] for
  inactive timestamps and all other timestamps are 'active' by default?

- How will this change impact code which distinguishes active/inactive
  timestamps based on presence/absence of <..> and [..]?

- What impact will this have on existing org files?

- Will this cause issues in parsing when you may have dates/times which
  are not supposed to be timestamps, but will look the same as
  timestamps? How will you distinguish them?

Personally, I like the clear distinction between what is a timestamp and
what isn't and what is an active timestamp and what is an inactive
timestamp.

Re: Some links in online manual do not work

[Tim Cross]

Tim Landscheidt  writes:

> Hi,
>
> at https://orgmode.org/manual/HTML-Export.html, the links
> for the first five (5) and the last two (2) subsections
> work, the links for:
>
> - "Headlines in HTML export"
>   (https://orgmode.org/manual/Headlines-in-HTML-export.html)
> - "Links in HTML export"
>   (https://orgmode.org/manual/Links-in-HTML-export.html)
> - "Tables in HTML export"
>   (https://orgmode.org/manual/Tables-in-HTML-export.html)
> - "Images in HTML export"
>   (https://orgmode.org/manual/Images-in-HTML-export.html)
> - "Math formatting in HTML export"
>   (https://orgmode.org/manual/Math-formatting-in-HTML-export.html)
> - "Text areas in HTML export"
>   (https://orgmode.org/manual/Text-areas-in-HTML-export.html)
>
> however all return "301 Moved Permanently", pointing back to
> "HTML-Export.html".
>
> The fact that all failing links are named
> "Something-in-HTML-export.html" might suggest an issue with
> the webserver configuration.
>

This looks like the nginx case issue again.

I've looked at this and there does not seem to be any 'clean' way to fix
this which also doesn't have significant processing overhead or a
maintenance burden.

I wonder if it would be worthwhile adding an option to HTML export which
would force all link targets and exported filenames to lower case,
thereby avoiding issues on platforms and with web servers which have
different positions wrt case sensitivity?

Re: [HELP] Help implementing org-lint/warnings buffer during export (was: Org HTML export accessibility (was: org exported pdf files))

[Tim Cross]

Ihor Radchenko  writes:

> Tim Cross  writes:
>
>> From an org-mode perspective, the key things which need to be maintained
>> (and which perhaps we could make even easier or possibly have
>> 'defaults') is the ability to add the alt attribute to any non-text
>> element. For example, images, videos, sound files etc. All such content
>> should always have some text describing the 'element'. However, it is
>> also important to be able to not have an alt tag in some situations (for
>> example, when using images as 'spacers' for formatting etc, we don't
>> want an alt tag. Things to be aware of are things like using single
>> characters or symbols (e.g. < and > for previous/next) or using unicode
>> and other symbols whose meaning/purpose may seem very obvious when
>> viewed, but is less so when 'spoken'. 
>>
>> From an authoring perspective, it probably would be good if org mode was
>> able to alert the user to content which lacks an alt attribute but which
>> probably should have one e.g. an image with no caption, a link to a
>> video/audio file etc.
>
> This sounds like a good idea.
>
> Org currently attempts to be slightly helpful by indicating, for
> example, LaTeX compilation warnings. However, it is just done by writing
> a simple message in the echo area.
>
> What would be more useful is the kind of buffer displayed by org-lint,
> but instead used during export:
> - If there are any export issues (LaTeX errors/warnings) they can appear
>   in the buffer
> - If there are any stylistic issues (like lack of alt attributes during
>   html export), they can also appear
>
> Ideally, we should be able to jump directly to the line containing
> error.
>
> org-lint code could be used as a base, but otherwise we need to
> implement something generic way to check style/export warnings on
> per-backend basis.
>
> This is probably something we need to do before we dive into the
> accessibility specifics. AFAIU from the Tim's reply, many of the
> accessibility guidelines may need to be decided by the document author.
> Tim, let me know if I am wrong and some of the accessible tagging must
> be done unconditionally.
>
> I am marking this as a help request.
>
> Let me know what you think.

I had a very similar idea wrt lint like functionality to alert user to
possible 'stylistic' and/or other problems in exported content. Adding
accessibility to that would then be the next step.

I'm very much against enforcing standards. Experience has taught me that
these sorts of thing change more frequently then you might expect. Also,
like the old sayings go "every rule has an exception" "you need to know
the rules before you can break them", etc. Far better to provide the
tools which can assist the author, but avoid enforcing some particular
view/opinion.

We would probably want to make the linting rules dynamic - allowing easy
add/removal/update of rules. People could then possibly use it to also
check for local policy/standard requirements by adding their own. 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko  writes:

> Agree. Let's not go too far yet and focus on extending special blocks
> and the inline special block element I propose. These topics (especially
> for markup element with less edge cases) pop up on the Org ML from time
> to time and worth looking into regardless whether Org is going to be
> used as technical documentation format.

+1

I'm confident your across the concerns I raised and if I understand your
proposal correctly, adding support for the markup entities RMS
requested should be possible without any significant adverse impact on
existing usability. Greater clarity regarding the underlying drive for
changing from texinfo for Emacs would be good, but that is essentially
and emacs devel question and as you point out, ability to support the
additional elements identified by RMS would potentially address other
issues, such as spaces, quoting and nesting within existing markup.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko  writes:

> Dear List,
>
> I am forwarding an official email from RMS changing subject line to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for manuals, we
> may first need to implement the discussed special blocks export and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically 
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> From: Richard Stallman 
> Subject: Re: Org mode and Emacs
> To: Tim Cross 
> Cc: emacs-de...@gnu.org
> Date: Mon, 26 Sep 2022 08:10:03 -0400 (4 days, 8 hours, 26 minutes ago)
> Flags: seen, list
> Maildir: /gmail/10 - Tech/software/org
>
> [[[ To any NSA and FBI agents reading my email: please consider]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a new
> standard source format for GNU manuals, _provided_ Org format has been
> extended with the capability to express all the constructions and
> distinctions that Texinfo can express, generate all the output formats
> Texinfo can generate, and use TeX to make beautiful printed output.
>
> Texinfo can generate these output formats: Info files, HTML, ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I am sure people can extend Org software to handle these semantic
> distinctions and generate these output formats.  Since it has been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format will look
> like.  That part is the crucial part; once it has been specified,
> people can work independently to implement various parts of handling
> that format.
>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> --

I realise this will likely come across as another post from "Debbie
Downer", but I feel it is important to add a warning here and not get
too carried away with the excitement of seeing org mode accepted to the
point it becomes the official documentation format for Emacs. There are
some potential pitfalls here which need to be considered and which could
impact on how we satisfy the remaining 'blocker' to org mode taking on
this important role. A few questions we might want to consider

What impact will adding all the additional formatting/markup primitives
have to the user experience?

One of the big benefits org has is simplicity in markup. This is one of
the driving themes in the 'markdown' movement. Will adding a lot of
additional syntax and markup tags add to cognitive load and complexity
and losing some of what makes org mode great to use. This could be one
of those situations where less is more.

Will adding a lot of additional markup entities have any impact on the
development of new and maintenance of existing export back ends? i

With all the additional entities, I suspect the demand for nesting of
entities will also increase. This has been an area org has struggled
with in the past. I suspect the big issue is that allowing nesting of
markup entities and maintaining simple syntax is very difficult.

Is there a risk of one aspect of org mode dominating all others and
potentially transforming it from a very flexible and general solution to
a technical documentation focus?

Texifno has a very specific focus. It aims to be an advanced formatting
system for writing software technical documents. As such, it is very
suitable for Emacs documentation. Org mode on the other hand is not a
documentation framework. While it does a fine job in this area, it is
not its prime focus. Org has many other unrelated roles, such as task
management, time tracking, simple spreadsheet and data
management/manipulation, literate programming and live document
generation, data capture etc etc. Will adopting org mode as the default
documentation format for Emacs run 

Apology [was: Re: Org HTML export accessibility (was: org exported pdf files))

[Tim Cross]

Dear all,

I think I owe everyone an apology. I have allowed frustration from
another area of life colour my response here and as a result, my tone
and assessment was too negative.

While it is correct that we cannot use org mode to generate accessible
PDFs and that does mean in environments where policy mandates accessible
content (which is PDF), we cannot use org mode.

The ability to generate accessible HTML is on the other hand quite
possible. Unlike PDF, the burden for doing this rests primarily with the
author and not the data processing framework. There are probably some
things we can do to improve or encourage accessible authoring, such as
alerting authors to content which looks like it should have an alt
tag. I will give some thought to that and when I get a chance, will see
how well various html based back ends deal with accessibility checking
tools.

For those interested and because it might help with understanding in
this area, I thought I'd outline the actual cause of my frustration. The
following isn't directly related to org-mode, but may be informative for
some. However, it is a little long, so feel free to just delete and move
on if your so inclined.

There is a little irony here as well. I've been using org mode since it
was first released. I even recall email discussions with Carsten when he
was first looking at how to improve outlining. It is org mode which
allowed me to generate really good quality documents and track all the
data and tasks I had to manage in my various job roles. People often
commented that they found it interesting that some of the best looking
documents produced in our area were from the person who is legally
blind! The irony being I cannot easily access the PDF output I created
and I became part of the problem by generating inaccessible documentation!

One very long standing frustration I have had in my career has been to
do with access to training materials. Most training organisations are
extremely reluctant to provide electronic copies of their learning
materials. I have lost count of the number of non-disclosure forms I
have been forced to sign in order to get electronic documents from a
training organisation (even though they are legally required to provide
their materials in an accessible format). Even when I have managed to
sign the necessary paperwork and get the documents, they have often been
in the form of DRM protected PDFs with an expiration date. While those
without any disability can retain the learning materials for future
reference, it is not a luxury afforded to anyone dependent on assistive
technology. Worse yet, most DRM protected formats also require the use
of non-free platforms, such as Windows or MacOS (I did often get some
perverse satisfaction from cracking the DRM protection, which in most
cases, is fairly easy to do). However, there is an ironic component here
as well. Usually, the DRM protected PDFs are actually very accessible
once you jump through all the necessary hoops. They are typically well
tagged and easy to navigate. On the other hand, the non-DRM PDFs are
rarely accessible despite correctly formatted PDFs actually being one of
the most accessible formats available. Often, once forced to provide
electronic copies of their learning materials, training organisations
will provide image PDFs, generated from a scanned version of their
materials. Image PDFs are 100% inaccessible - they are just pictures, so
you cannot even extract the text using tools like pdftotext[1] Even when
not image PDFs, they often lack the necessary tagging etc (though, this
situation has improved in recent years as many tools now default to
accessible output rather than requiring it to be enabled).

Even once you jump through all the necessary hoops, your not out of the
woods yet. My current frustration has been with obtaining the important
bit of paper which says your trained and certified. After completing the
course I looked at what I needed to do to sit the certification
exam. The exam is one which has to be done at a large certification
examination centre and it is done electronically. It is actually run by
a very large US based training organisation, who I will not name.

It runs out that I cannot do the training at this time. I have to give
them  a minimum of 12 months notice to sit the certification exam
because due to my 'special' needs, the whole examination centre has to
be booked out just for me! To make it worse, the assistive technology I
have to use is a program called JAWS, which only runs on windows and
which I am totally unfamiliar with. My suggestion to just have a
sighted person assist me by reading the questions and entering the
answers has been rejected as well as all other suggestions and
appeals. It is highly likely I will just forgo certification. While it
would have been handy, it isn't essential.

I outline all of this not for sympathy but to try and promote
understanding of the challenges faced by many who need access to