Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Sterling Hooten]

Thanks for the quick feedback!

> On 2023-01-27, at 08:09, Ihor Radchenko  wrote:
> 
> Following ISO and other standards is indeed a reasonable idea. However,
> the standards are not necessarily designed for human consumption.
> In contrast, Org mode is designed to be read by humans as well, even
> without Emacs - just as plain text.
> 
> Design for human consumption is one of the reasons we do provide the
> redundant information like week day (I personally did find it extremely
> useful on multiple occasions) and do use spaces, deviating from ISO. The
> above ISO example is barely readable by humans. Another example from
> wiki page of ISO 8601 is even worse: R5/2008-03-01T13:00:00Z/P1Y2M10DT2H30M

Certainly agree that the ISO format can be difficult for humans to read, both 
from 
the lack of spaces and terse syntax.

This isn’t (much) of a problem from a display format perspective because we can 
parse 
the encoded format and present the user with a human readable version. So the 
readability 
issue is more about the encoded format. But unlike the display format, which 
could follow 
whatever grammar or locale preference of the user, the encoded format must be 
unambiguously parseable. If it’s possible to make the ISO format more human 
readable 
while still preserving parseability this could be viable.

I’m less arguing against the option for encoding things in a variation of the 
ISO standard, 
but urging that Org support using an encoding of the ISO format in its raw 
state.

> And we need to deviate from ISO 8601 anyway. At least, because it does
> not define time zones, only absolute UTC offsets. So, the ability to
> conform with the existing formats remains questionable.

This is correct for the 2019 version of the ISO 8601.

From my understanding the newest ISO draft is incorporating an existing syntax 
used 
in java.time.ZonedDateTime [JAVAZDT] to allow for time zones. So we could still 
aim for 
compliance with published standards.

The Internet Extended Date/Time Format  (IXDTF) is a forthcoming standard which 
defines an extension syntax for timestamps as specified in [RFC3339]  which 
itself is 
compatible with the [JAVAZDT] syntax.

The IXDTF is of particular interest in this situation because the format 
provides a 
general way to attach any additional information to a timestamp. The authors 
have done 
a great job of lucidly explaining some of the nuances of timestamps.

https://datatracker.ietf.org/doc/draft-ietf-sedate-datetime-extended/

> I will need more time to look through the provided links and think about
> the ideas.

Look forward to your comments!

Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Sterling Hooten]

Hi all,

Collaborating around the subject of "time" is difficult; there are
subtleties abound in implementation, the perspectives people come from,
and the language used in discussions. I'm going to provide a glossary to
establish common terminology, use these terms to analyze our current
state, offer a roadmap for solving the problem in stages, suggest a
format for timestamps, urge compatibility with "exotic" use cases, and
finally call for outside help with implementing a timezone aware agenda
system.

Summary and references are at the end.

This is an initial glossary compiled from various standards and sources;
it's incomplete, probably incorrect, and open to critique, but is useful
in articulating a possible road map forward.

• Time

  Time (concept)
What clocks measure (Einstein)
  Time axis
Mathematical representation of the succession in time according
to the space-time model of instantaneous events along a unique
axis (ISO).

  Instant (object)
A single point on time axis (ISO).
  Moment in time
See: instant.
  Mark
A set of symbols related to the object, or carrying some
symbolic meaning
  Time scale
System of ordered marks which can be attributed to instants on
the time axis , one instant being chosen as the origin. e.g.,
GMT, UTC, TAI.
  Basis time
See: time scale.
  Time (mark)
The designation of an instant on a selected time scale, used in
the sense of time of day.
  Time interval (object)
part of the time axis limited by two instants and, unless
otherwise stated, the limiting instants themselves a part of
time limited by two instants or moments in time (ISO). The
elapsed time between two events (NIST).
  Duration (object)
as a quantity characterizing a time interval. These can be
written in different formats.
  UTC
Time scale with the same rate as International Atomic Time
(TAI), but differing from TAI only by an integral number of
seconds.
  Offset
Constant duration difference between times of two time scales
(ISO). i.e., a quantity to combine with a time scale to produce
a wall time. e.g., Nepal uses a +5:45 offset from the UTC time
scale.
  Time shift
See: offset.
• Calendar and civil time
  Wall time
what shows on the clock on the wall at a location. Like "local
system time" but needn't reference a computer to do the
calculation.
  Standard time
Time scale derived from UTC, by a time shift established in a
given location by the competent authority (ISO).
  Local system time
Local system time is determined by applying the system's time
zone offset and year offset values to UTC. The Time of day
system value displays the local system time. Local system time
and system time are used interchangeably.
  Time Zone
A place/region. Can map between wall time and a time scale with
a table and an offset. A set of rules for determining the local
observed time (wall time) as it relates to incremental time (as
used in most computing systems) for a particular geographical
region. e.g., Brasília time presently has an offset of −03:00
from the UTC time.
  Calendar event
A calendar object that is commonly used to represent things that
mark time or use time. Examples include meetings, appointments,
anniversaries, start times, arrival times, closing times.

• Implementation These concern how we actually decide to record,
  reference, or manipulate time.
  Representation
Expression indicating a time point, time interval or recurring
time interval. e.g., [2023-02-02 Thu 12:58 +1w], "this next
suday at 2pm EST", 3600 seconds from Unix epoch
  Format
A description of the abstract form used for a representation.
e.g., [-MM-DD] 'Explain in prose relative to this moment in
time using locale and include your timezone'
  Encoding
The method of storing a representation of time e.g., datestruct
in memory, Org timestamp in body of heading, value of a
"created" key in a database
  Format syntax
Rules that allow for parsing a encoding unambiguously into some
time scale.
  Timestamp (mark)
An encoded representation in a selected format. e.g., 24/01/2023
or 2023-01-24
  Delimiting syntax
Rules that allow for detection and extraction of an encoding.
Necessary for encodings embedded in prose. e.g., '[]' for org
timestamps.

  Displayed time
The formatting of a representation exposed to a user.
  Calculating
Manipulating a set of time points, time intervals, or recurring
time intervals. e.g., determining instant from an offset,
comparing two representations in some lattice.
  Incremental time
A d

Re: [FR] Please add writing to existing heading in org-bibtex

[Sterling Hooten]

Thanks for the instructions, this is my first patch.

I think calling  `nonew’ invalidates the argument `no-indent’ as 
`org-bibtex-put’ eventually 
calls `org-entry-put’ which uses `org-indent-line’. I’m not sure what’s the 
best way to handle that.

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index d4e9b4368..8eab4cae2 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -513,6 +513,10 @@ use =attachment:= style links instead of the standard 
=file:= link type.
 *** New function ~org-get-title~ to get =#+TITLE:= property from buffers
 
 A function to collect the document title from the org-mode buffer.
+*** ~org-bibtex-write~ can now write to heading at point with optional 
interactive argument
+
+Previously, a new heading was created. Now with argument =nonew= the
+bibtex data will be added to properties of heading at point.
 
 *** ~org-fold-show-entry~ does not fold drawers by default anymore
 
diff --git a/lisp/ol-bibtex.el b/lisp/ol-bibtex.el
index 313b1cde8..91aa4c36e 100644
--- a/lisp/ol-bibtex.el
+++ b/lisp/ol-bibtex.el
@@ -718,29 +718,33 @@ Return the number of saved entries."
   (interactive "fFile: ")
   (org-bibtex-read-buffer (find-file-noselect file 'nowarn 'rawfile)))
 
-(defun org-bibtex-write (&optional noindent)
+(defun org-bibtex-write (&optional nonew noindent)
   "Insert a heading built from the first element of `org-bibtex-entries'.
+
+With prefix argument NONEW modify properties of heading at point.
 When optional argument NOINDENT is non-nil, do not indent the properties
 drawer."
-  (interactive)
+  (interactive "P")
   (unless org-bibtex-entries
 (error "No entries in `org-bibtex-entries'"))
   (let* ((entry (pop org-bibtex-entries))
 (org-special-properties nil) ; avoids errors with `org-entry-put'
 (val (lambda (field) (cdr (assoc field entry
-(togtag (lambda (tag) (org-toggle-tag tag 'on
-(org-insert-heading)
-(insert (funcall org-bibtex-headline-format-function entry))
-(insert "\n:PROPERTIES:\n")
-(org-bibtex-put "TITLE" (funcall val :title) 'insert)
+(togtag (lambda (tag) (org-toggle-tag tag 'on)))
+ (insert-raw (not nonew)))
+(unless nonew
+  (org-insert-heading)
+  (insert (funcall org-bibtex-headline-format-function entry))
+  (insert "\n:PROPERTIES:\n"))
+(org-bibtex-put "TITLE" (funcall val :title) insert-raw)
 (org-bibtex-put org-bibtex-type-property-name
(downcase (funcall val :type))
-'insert)
+insert-raw)
 (dolist (pair entry)
   (pcase (car pair)
(:titlenil)
(:type nil)
-   (:key  (org-bibtex-put org-bibtex-key-property (cdr pair) 'insert))
+   (:key  (org-bibtex-put org-bibtex-key-property (cdr pair) 
insert-raw))
(:keywords (if org-bibtex-tags-are-keywords
   (dolist (kw (split-string (cdr pair) ", *"))
 (funcall
@@ -748,9 +752,9 @@ drawer."
  (replace-regexp-in-string
   "[^[:alnum:]_@#%]" ""
   (replace-regexp-in-string "[ \t]+" "_" kw
-(org-bibtex-put (car pair) (cdr pair) 'insert)))
-   (_ (org-bibtex-put (car pair) (cdr pair) 'insert
-(insert ":END:\n")
+(org-bibtex-put (car pair) (cdr pair) insert-raw)))
+   (_ (org-bibtex-put (car pair) (cdr pair) insert-raw
+(unless nonew (insert ":END:\n"))
 (mapc togtag org-bibtex-tags)
 (unless noindent
   (org-indent-region
@@ -771,7 +775,7 @@ drawer."
   (interactive "fFile: ")
   (let ((pos (point)))
 (dotimes (_ (org-bibtex-read-file file))
-  (save-excursion (org-bibtex-write 'noindent))
+  (save-excursion (org-bibtex-write nil 'noindent))
   (re-search-forward org-property-end-re)
   (insert "\n"))
 (org-indent-region pos (point


> On 2022-12-27, at 13:34, Ihor Radchenko  wrote:
> 
> Sterling Hooten  writes:
> 
>> The default behavior of org-bibtex-write is to insert a new
>> heading with the bibliographic data in the properties. But an
>> alternative workflow would just update the properties of the heading at
>> point, rather than creating a new one. The below patch is a simple
>> implementation I’ve been using for a month. Would it be possible to
>> integrate this upstream?
> 
> Yes. It will make sense.
> 
>> -(defun org-bibtex-write ()
>> -  "Insert a heading built from the first element of `org-bibtex-entries'."
>> +(defun org-bibtex-write (&optional no-new)
>> +  "Insert a heading built from the first element of `org-bibtex-

[FR] Please add writing to existing heading in org-bibtex

[Sterling Hooten]

The default behavior of org-bibtex-write is to insert a new
heading with the bibliographic data in the properties. But an
alternative workflow would just update the properties of the heading at
point, rather than creating a new one. The below patch is a simple
implementation I’ve been using for a month. Would it be possible to
integrate this upstream?

Thanks,
Sterling

diff --git a/lisp/ol-bibtex.el b/lisp/ol-bibtex.el
index 81b99167b..38198eae4 100644
--- a/lisp/ol-bibtex.el
+++ b/lisp/ol-bibtex.el
@@ -703,8 +703,9 @@ Return the number of saved entries."
   (interactive "fFile: ")
   (org-bibtex-read-buffer (find-file-noselect file 'nowarn 'rawfile)))
 
-(defun org-bibtex-write ()
-  "Insert a heading built from the first element of `org-bibtex-entries'."
+(defun org-bibtex-write (&optional no-new)
+  "Insert a heading built from the first element of `org-bibtex-entries'. With 
non-nil optional NO-NEW write to heading at point instead of creating new."
+  ;; SWH 2022-11-22 changes to allow for writing heading at point instead of 
inserting new.
   (interactive)
   (when (= (length org-bibtex-entries) 0)
 (error "No entries in `org-bibtex-entries'"))
@@ -712,8 +713,9 @@ Return the number of saved entries."
 (org-special-properties nil) ; avoids errors with `org-entry-put'
 (val (lambda (field) (cdr (assoc field entry
 (togtag (lambda (tag) (org-toggle-tag tag 'on
-(org-insert-heading)
-(insert (funcall org-bibtex-headline-format-function entry))
+(unless no-new
+  (org-insert-heading)
+  (insert (funcall org-bibtex-headline-format-function entry)))
 (org-bibtex-put "TITLE" (funcall val :title))
 (org-bibtex-put org-bibtex-type-property-name
(downcase (funcall val :type)))

Re: Completely hide properties drawer in 9.6

[Sterling Hooten]

So I've got a solution somewhat working but I'd like some help simplifying it.

The good news is that it's significantly faster than the previous Org
implementation I was using (the one you wrote on stack exchange), and
can now handle folding a file with 30k lines in around .065s for
hiding and .005s for showing. It does seem to be dependent on garbage
collection settings though.

I'm just going to walk through my solution and then ask some specific
questions about how to fix it.

Most of the work is actually in dealing with invisible edits and then
handling =org-return= when called in a heading with a hidden property
drawer.

Starting with a basic minor mode:

#+begin_src emacs-lisp
  (define-minor-mode org-hide-properties-mode
"Toggle `org-hide-properties-mode'. Hide all property drawers in buffer and 
any further created."
:init-value nil
:lighter (" HP")
(if org-hide-properties-mode
(progn 
  (org-fold-add-folding-spec 'property-drawer-hidden-edges
 '((:ellipsis . nil)
   (:isearch-open . t)
   (:front-sticky . nil)
   (:rear-sticky . nil))
 nil
 'append)
  (org-fold-add-folding-spec 'property-drawer-hidden-interior
 '((:ellipsis . nil)
   (:isearch-open . t)
   (:front-sticky . t)
   (:rear-sticky . t))
 nil
 'append)
  (advice-add 'org-insert-property-drawer :after 
#'org-fold-hide-property-drawer)
  (org-fold-hide-property-drawer-all))
  (org-fold-show-property-drawer-all)
  (advice-remove 'org-insert-property-drawer 
#'org-fold-hide-property-drawer)))
#+end_src

In order to both have newly added properties automatically adopt the
invisibility text-property the interior characters of the properties
box needs to be sticky. But this conflicted with being able to type
with the point before the hidden text. To satisfy both these
requirements I applied =property-drawer-hidden-interior= as a folding
spec which was sticky to all but the first two and last two characters
of the property drawer. Then I applied =property-drawer-hidden-edges=
to these two remaining chunks. In this way you can add in new
properties and they'll be invisible, but typing at the edges appears
visibly.

Is there an alternative way of doing this without two separate folding
specs? Maybe an additional key for ":interior-sticky" which is sticky
on all characters sharing the same spec, but not sticky at the front
and rear?

I couldn't quite figure out what the :fragile setting was.

#+begin_src emacs-lisp
(defun org-fold--hide-property-drawers (begin end)
"Hide all property drawers betweeen BEGIN and END."
(org-with-wide-buffer
  (goto-char begin)
  (while (and (< (point) end)
  (re-search-forward org-property-drawer-re end t))
(with-silent-modifications
 ;; Hide interior of drawer to be sticky so org-entry-put will 
respect invisibility
 (org-fold-region (+ (match-beginning 0) 1) (- (match-end 0) 2) t 
'property-drawer-hidden-interior)
 ;; Hide first two and last two characters without sticky
 (org-fold-region (- (match-beginning 0) 1) (+ (match-beginning 0) 
1) t 'property-drawer-hidden-edges)
 (org-fold-region (- (match-end 0) 2) (match-end 0) t 
'property-drawer-hidden-edges)

(defun org-fold--show-property-drawers (begin end)
"Unhide all property drawers in buffer between BEG and END"
(org-fold-region begin end nil
 'property-drawer-hidden-edges)
;; SWH 2022-12-15 HACK because I'm using two diferent folding specs
(org-fold-region begin end nil
 'property-drawer-hidden-interior))

(defun org-fold-hide-property-drawer-all ()  
"Hide all property drawers in visible part of current buffer."
(org-fold--hide-property-drawers (point-min) (point-max)))
  
(defun org-fold-show-property-drawer-all ()
"Unhide all property drawers visible part of current buffer"
(org-fold--show-property-drawers (point-min) (point-max)))

(defun org-fold-toggle-property-drawer-all ()
"Show or hide all property drawers in buffer."
(interactive)
(require 'org-macs)
(if org-properties-hidden-p
(progn 
  (setq org-properties-hidden-p nil)
  (org-fold-show-property-drawer-all))
  (setq org-properties-hidden-p t)
  (org-fold-hide-property-drawer-all)))
#+end_src

Folding headings or subtrees:

#+begin_src emacs-lisp
  (defun org-fold-show-property-drawer (&option

Re: Completely hide properties drawer in 9.6

[Sterling Hooten]

I was able to get a very rough version working to fold and hide 
the properties drawer completely at point.

#+begin_src emacs-lisp
  (defun swh-org-hide-properties-drawer ()
"Completely hide the property drawer in heading at point."
(org-fold-add-folding-spec 'org-hide-property-drawer
   '((:ellipsis . nil)
 (:isearch-open . t))
   nil
   'append)
(let* ((block (org-get-property-block))
   ;; Extremely hacky way to get the whole property drawer
   (beg (- (car block) 14))
   (end (+ (cdr block) 5)))
  (with-silent-modifications
(org-fold-region beg end t 'org-hide-property-drawer
#+end_src

Is there a better way to get the beginning and end 
of the property drawer more robustly?

This is a step in the right direction, but it seems too sensitive to accidental 
editing.

This should be impervious to most types of user interaction,
and prevent accidentally deleting or abutting anything
up against either the start or end of the properties box.

I tried setting `org-fold-catch-invisible-edits' to different values, but even
with set to nil I can backward delete and erase some invisible
text. Is this dependent on their being an ellipsis? Or is there
another way to make this impervious to most edits?

When adding a property with `org-property-put’ this would need
to be called again, is there a way to do that robustly?

> On 2022-12-14, at 12:14, Ihor Radchenko  wrote:
> 
> I tried my best to explain the design in the top comment. You may also
> look into `org-fold-initialize'.

Re: Completely hide properties drawer in 9.6

[Sterling Hooten]

> On 2022-12-14, at 06:15, Ihor Radchenko  wrote:
> 
> I do have this implemented in local branch. Just did not push it
> upstream as it was rejected.

Could you share that here so I could test it?

> Note that Emacs 29 does not have issues with large numbers of overlays.

I’m using the MacOS version and not building Emacs myself so I can’t try this 
out at the moment.

> This is handled by org-fold-core.el library.
> Note that you can use the library to introduce new fold types.
> Editing inside folds is optimized. All other concerns are also handled
> within org-fold-core.

I will read through this and try it out.

Completely hide properties drawer in 9.6

[Sterling Hooten]

First of all, thanks to everyone for working on 9.6!

I'm trying to clone Lotus notes in Org mode. This requires certain
properties to be generated for every heading (e.g., ID, creation time,
modification time, etc.). But these are mainly used for searching and
sorting, and rarely of interest to me during editing. For this reason
I would like to /completely/ hide the properties box (not just the
individual properties).

This has been discussed multiple times before in this list. The
general sentiment has been to reject adding it to Org as hiding things
ompletely from the user might be confusing, but it's also been
brought up enough times independently to demonstrate a latent demand
for this behavior, and a working example would be very useful for
those of us who wish to have a cleaner interface and editing
experience.

Alternative suggestions have been to change the face to make the
properties box appear smaller, but this interferes with movement from
the heading to the body, and the ability to toggle the visibility of a
property box would make this is redundant.

In  Ihor suggested
adding `org-custom-properties-hide-emptied-drawers', this was
rejected, but is pretty close to the desired behavior, except it's
still global. I was unable to get this working in release 9.6.

Prior to 9.6 I overwrote org-cycle-heading to hide the properties box
using overlays. This worked alright but had slowdowns on larger files
and affected typing cadence noticeably. Org 9.6 is now relying on text
properties instead of overlays, and I would like help replicating (and
improving) that behavior.

Part of the complication comes from updating modified timestamps in
the properties of each heading. This drives the need for hiding to be
fast so as not to bog down the editing experience when triggered by
`after-change-functions'.

The constant updating of a modification timestamp system puts some
constraints on the property hiding system. From working in 9.5 I
believe the relevant requirements are:

• Ability to add or modify properties with `org-entry-put' and
  maintain the state of being hidden or visible
  • E.g., Updating the ":MODIFIED:" property to a hidden property
drawer should keep the drawer hidden.
• Toggle on and off hiding globally (perhaps in a minor mode)
• Moving headings around with `org-metaup' etc preserves the
  visibility state
  • A hidden property drawer should stay hidden when the outline is
rearranged
• Resistant to changes with undo etc
  • Undoing a change should cause the contet to revert to the same
visibility state it had at time of change.
• Don't overwrite when editing normally
  • With the point at the end of a heading and immediately before an
ellipsis, typing "j" should result in the character being inserted
with the face of the heading, and not overwrite or otherwise
disturb the invisible text. `org-fold-catch-invisible-edits' might
be relevant here.
• Reveal or hide just the property drawer of the heading at point and
  not the whole buffer
  • This is partially for speed reasons (to handle modified timestamp
updates), but also because revealing all property drawers is
extremely annoying when the surrounding headings in the buffer
have headings with >10 properties.
• Fast enough to use with `after-change-functions'
  • The modification timestamp setup I'm using is triggered by buffer
changes, and thus when typing runs after every keystroke. To keep
this smooth and not be a bottleneck it would need `(benchmark 100
'org-hide-properties)' to be less than .1s on an Org file with
>20,000 lines and >1,000 headings.
• Ability to reveal just a subset of property lines
  • e.g., `org-bibtex' creates many properties to store bibliographic
data. It would be useful to have enough control over hiding of
properties such that if only ":BIB_FIELD:" properties of.
• Hide ellipsis unless body text is present
  • If all the headings have properties then the default folding
behavior will place ellipses at the end of each heading, thus
destroying the informational value of the ellipsis as an
indication of folded but displayable content. It would be useful
if instead the ellipsis would only be present when there's body
text.

Any help would be greatly appreciated as things have changed in 9.6
and I'm not sure where the opportunities are to leverage the
improvements (e.g., are text properties now the way? Or should I still
implement it with faces?).

Thanks!

[FR] Display heading alongside id in org-insert-link completion

[Sterling Hooten]

Hi,

When trying to insert a stored 'id' link with org-insert-link the
candidates are just the 'id' numbers themselves. This doesn't help
when trying to determine what the 'id' actually points to. Is there a
way for org-insert-link to display headings or other information
(maybe the tags or other heading properties) wih a completion
framework like Ivy?

Thanks!
Sterling