Hello! See attached many patches as I do love scope creep! As usual feel free to apply any patches you're happy with so I don't have to keep resending them (except patches marked WIP of course).
Tests pass after each patch on emacs 30.2. Tests pass after final patch on emacs 28 and 29. Tests pass after final patch with TZ set to UTC, Europe/Istanbul, and America/New_York. The first 4 patches aren't even related to this bug report but do include some nice agenda sorting fixes. Patch 5 is a fixed up version of the test I sent previously. Patch 6 is a minor cleanup. Patch 7 actually fixes the reported bug! But don't apply that one! It's marked WIP. As we transition to using the org-element API in more places, we are eventually going to have to figure out what to do about the agenda. My WIP patch shows one way to get access to the org element API but I assume it comes with some serious performance issues (still have no clue how parsing/caching works so please let me know if my assumption is correct). I haven't done any investigations yet and would just like to ask: How are we planning on shoving org-element into the agenda? Have we already done that somewhere? If using org-element like this will be too difficult, I can come up with a solution that is more inline with how the other agenda logic works. Ihor Radchenko <[email protected]> writes: > Morgan Smith <[email protected]> writes: > >> (let ((set-agenda-with-priority >> (lambda (priority) > > Also, see cl-flet. Fixed! > Why "scheduled tomorrow"? Normally, agenda does not display tasks > scheduled in the future. Just felt like an easy way to demonstrate relative timestamps. They are in the agenda because they are TODO's and the agenda is of TODO type. > Also, why does both yesterday come after deadline today? The former has > deadline yesterday. Fixed! It took me way to long to figure out why my code was sorting stuff wrong. SCHEDULED and DEADLINE have to be on the same line, not separate.
>From 82817a4c7087492be441c031639b4d18b3d2344b Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Fri, 28 Nov 2025 19:50:16 -0500 Subject: [PATCH 1/7] test-org-agenda/sorting: New test * testing/lisp/test-org-agenda.el (test-org-agenda/sorting): New test. --- testing/lisp/test-org-agenda.el | 96 +++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/testing/lisp/test-org-agenda.el b/testing/lisp/test-org-agenda.el index a8a11af0f..3408b4a1f 100644 --- a/testing/lisp/test-org-agenda.el +++ b/testing/lisp/test-org-agenda.el @@ -670,6 +670,102 @@ test-org-agenda/skip-scheduled-repeats-after-deadline (buffer-string)))))) (org-test-agenda--kill-all-agendas)))) +(ert-deftest test-org-agenda/sorting () + "Test if `org-agenda' sorts according to `org-agenda-sorting-strategy'." + :expected-result :failed + ;; FIXME: test the following + ;; urgency-up + ;; category-up + ;; category-keep + ;; user-defined-up + (cl-flet ((call-agenda-with-priority (priority) + (let ((org-agenda-custom-commands + `(("f" "no fluff" todo "" + ((org-agenda-tags-column -10) + (org-agenda-overriding-header "") + (org-agenda-prefix-format "") + (org-agenda-sorting-strategy ',priority)))))) + (org-agenda nil "f") + (string-trim + (substring-no-properties + (buffer-string)))))) + (org-test-agenda-with-agenda + "#+TODO: TODO WAIT DONE +* TODO a :a: +* WAIT a :b: +* TODO b :a: +* WAIT b :b:" + (should + (string-equal + (call-agenda-with-priority '(tag-up alpha-up)) + "TODO a :a: +TODO b :a: +WAIT a :b: +WAIT b :b:")) + (should + (string-equal + (call-agenda-with-priority '(alpha-up tag-down)) + ;; This result seems unexpected. It is because the + ;; alphabetical sort is done on the full heading title + ;; including tags (ex. "a :b:"). + ;; FIXME: This is probably not what users want. + "TODO a :a: +WAIT a :b: +TODO b :a: +WAIT b :b:")) + (should + (string-equal + (call-agenda-with-priority '(todo-state-up alpha-down)) + "TODO b :a: +TODO a :a: +WAIT b :b: +WAIT a :b:"))) + (org-test-agenda-with-agenda + "* TODO [#A] foo +:PROPERTIES: +:Effort: 10m +:END: +* TODO [#B] bar +:PROPERTIES: +:Effort: 5m +:END: +* TODO [#C] habit [2/4] +:PROPERTIES: +:STYLE: habit +:END: +* TODO ping [1/4] +:PROPERTIES: +:Effort: 1y +:END: +* TODO pong [3/4] +:PROPERTIES: +:Effort: 10m +:END:" + (should + (string-equal + (call-agenda-with-priority '(effort-up priority-down)) + "TODO [#B] bar +TODO [#A] foo +TODO pong [3/4] +TODO ping [1/4] +TODO [#C] habit [2/4]")) + (should + (string-equal + (call-agenda-with-priority '(habit-up priority-up)) + "TODO [#C] habit [2/4] +TODO [#B] bar +TODO ping [1/4] +TODO pong [3/4] +TODO [#A] foo")) + (should + (string-equal + (call-agenda-with-priority '(stats-up priority-up)) + "TODO [#B] bar +TODO [#A] foo +TODO ping [1/4] +TODO [#C] habit [2/4] +TODO pong [3/4]"))))) + (ert-deftest test-org-agenda/tags-sorting () "Test if `org-agenda' sorts tags according to `org-tags-sort-function'." (let ((string-length< (lambda (s1 s2) base-commit: 1661cd153f16444a38a262c36bca3ecf03edf558 -- 2.51.2
>From 90b4f7fd50c2081cb3b2d3bd0b3dc16b8a3e1694 Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Sun, 16 Nov 2025 15:09:05 -0500 Subject: [PATCH 2/7] org-entries-lessp: Simplify and add error handling * lisp/org-agenda.el (org-entries-lessp): Simplify. Fix checkdoc warning. Add explicit error for trying to use `org-agenda-cmp-user-defined' when it is not a function. Add explicit error for unknown sorting strategies. --- lisp/org-agenda.el | 110 ++++++++++++++++++++------------------------- 1 file changed, 48 insertions(+), 62 deletions(-) diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index 0c530e2ff..8c819514a 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -7603,68 +7603,54 @@ org-cmp-habit-p ((and (not ha) hb) +1)))) (defun org-entries-lessp (a b) - "Predicate for sorting agenda entries." - ;; The following variables will be used when the form is evaluated. - ;; So even though the compiler complains, keep them. - (let ((ss org-agenda-sorting-strategy-selected)) - (org-dlet - ((timestamp-up (and (org-em 'timestamp-up 'timestamp-down ss) - (org-cmp-ts a b ""))) - (timestamp-down (if timestamp-up (- timestamp-up) nil)) - (scheduled-up (and (org-em 'scheduled-up 'scheduled-down ss) - (org-cmp-ts a b "scheduled"))) - (scheduled-down (if scheduled-up (- scheduled-up) nil)) - (deadline-up (and (org-em 'deadline-up 'deadline-down ss) - (org-cmp-ts a b "deadline"))) - (deadline-down (if deadline-up (- deadline-up) nil)) - (tsia-up (and (org-em 'tsia-up 'tsia-down ss) - (org-cmp-ts a b "timestamp_ia"))) - (tsia-down (if tsia-up (- tsia-up) nil)) - (ts-up (and (org-em 'ts-up 'ts-down ss) - (org-cmp-ts a b "timestamp"))) - (ts-down (if ts-up (- ts-up) nil)) - (time-up (and (org-em 'time-up 'time-down ss) - (org-cmp-time a b))) - (time-down (if time-up (- time-up) nil)) - (stats-up (and (org-em 'stats-up 'stats-down ss) - (org-cmp-values a b 'org-stats))) - (stats-down (if stats-up (- stats-up) nil)) - (priority-up (and (org-em 'priority-up 'priority-down ss) - (org-cmp-values a b 'priority))) - (priority-down (if priority-up (- priority-up) nil)) - (urgency-up (and (org-em 'urgency-up 'urgency-down ss) - (org-cmp-values a b 'urgency))) - (urgency-down (if urgency-up (- urgency-up) nil)) - (effort-up (and (org-em 'effort-up 'effort-down ss) - (org-cmp-effort a b))) - (effort-down (if effort-up (- effort-up) nil)) - (category-up (and (or (org-em 'category-up 'category-down ss) - (memq 'category-keep ss)) - (org-cmp-category a b))) - (category-down (if category-up (- category-up) nil)) - (category-keep (if category-up +1 nil)) - (tag-up (and (org-em 'tag-up 'tag-down ss) - (org-cmp-tag a b))) - (tag-down (if tag-up (- tag-up) nil)) - (todo-state-up (and (org-em 'todo-state-up 'todo-state-down ss) - (org-cmp-todo-state a b))) - (todo-state-down (if todo-state-up (- todo-state-up) nil)) - (habit-up (and (org-em 'habit-up 'habit-down ss) - (org-cmp-habit-p a b))) - (habit-down (if habit-up (- habit-up) nil)) - (alpha-up (and (org-em 'alpha-up 'alpha-down ss) - (org-cmp-alpha a b))) - (alpha-down (if alpha-up (- alpha-up) nil)) - (need-user-cmp (org-em 'user-defined-up 'user-defined-down ss)) - user-defined-up user-defined-down) - (when (and need-user-cmp org-agenda-cmp-user-defined - (functionp org-agenda-cmp-user-defined)) - (setq user-defined-up - (funcall org-agenda-cmp-user-defined a b) - user-defined-down (if user-defined-up (- user-defined-up) nil))) - (cdr (assoc - (eval (cons 'or org-agenda-sorting-strategy-selected) t) - '((-1 . t) (1 . nil) (nil . nil))))))) + "Predicate for sorting agenda entries A and B." + (catch :org-entries-lessp-return + (dolist (strategy org-agenda-sorting-strategy-selected) + (when-let* + ((result + (cl-case strategy + (timestamp-up (org-cmp-ts a b "")) + (timestamp-down (org-cmp-ts b a "")) + (scheduled-up (org-cmp-ts a b "scheduled")) + (scheduled-down (org-cmp-ts b a "scheduled")) + (deadline-up (org-cmp-ts a b "deadline")) + (deadline-down (org-cmp-ts b a "deadline")) + (tsia-up (org-cmp-ts a b "timestamp_ia")) + (tsia-down (org-cmp-ts b a "timestamp_ia")) + (ts-up (org-cmp-ts a b "timestamp")) + (ts-down (org-cmp-ts b a "timestamp")) + (time-up (org-cmp-time a b)) + (time-down (org-cmp-time b a)) + (stats-up (org-cmp-values a b 'org-stats)) + (stats-down (org-cmp-values b a 'org-stats)) + (priority-up (org-cmp-values a b 'priority)) + (priority-down (org-cmp-values b a 'priority)) + (urgency-up (org-cmp-values a b 'urgency)) + (urgency-down (org-cmp-values b a 'urgency)) + (effort-up (org-cmp-effort a b)) + (effort-down (org-cmp-effort b a)) + (category-up (org-cmp-category a b)) + (category-down (org-cmp-category b a)) + (category-keep (and (org-cmp-category a b) +1)) + (tag-up (org-cmp-tag a b)) + (tag-down (org-cmp-tag b a)) + (todo-state-up (org-cmp-todo-state a b)) + (todo-state-down (org-cmp-todo-state b a)) + (habit-up (org-cmp-habit-p a b)) + (habit-down (org-cmp-habit-p b a)) + (alpha-up (org-cmp-alpha a b)) + (alpha-down (org-cmp-alpha b a)) + (user-defined-up (unless (functionp org-agenda-cmp-user-defined) + (error "Please set `org-agenda-cmp-user-defined' to a function or remove `user-defined-up' from `org-agenda-sorting-strategy'")) + (funcall org-agenda-cmp-user-defined a b)) + (user-defined-down (unless (functionp org-agenda-cmp-user-defined) + (error "Please set `org-agenda-cmp-user-defined' to a function or remove `user-defined-down' from `org-agenda-sorting-strategy'")) + (funcall org-agenda-cmp-user-defined b a)) + (t (error "Invalid value %S in `org-agenda-sorting-strategy'" strategy))))) + (cond + ((eq -1 result) (throw :org-entries-lessp-return t)) + ((eq 1 result) (throw :org-entries-lessp-return nil))))) + nil)) ;;; Agenda restriction lock -- 2.51.2
>From 2733797441b4e64a132abfd08b7fedf71e438377 Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Fri, 28 Nov 2025 16:59:58 -0500 Subject: [PATCH 3/7] org-agenda: Use `most-positive-fixnum', not arbitrary big numbers For relative effort calculation we set unset values to 32767 minutes which is roughly 22.75 days. This would be incorrect if the user has effort values larger then that. * lisp/org-agenda.el (org-agenda-finalize-entries, org-cmp-effort) (org-cmp-ts, org-agenda-compare-effort): Instead of using arbitrary big numbers, use `most-positive-fixnum'. --- lisp/org-agenda.el | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index 8c819514a..08d64e400 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -7378,7 +7378,7 @@ org-agenda-finalize-entries (setq list (org-agenda-limit-entries list 'effort-minutes max-effort (lambda (e) (or e (if org-agenda-sort-noeffort-is-high - 32767 -1)))))) + most-positive-fixnum -1)))))) (when max-todo (setq list (org-agenda-limit-entries list 'todo-state max-todo))) (when max-tags @@ -7495,7 +7495,7 @@ org-cmp-values (defsubst org-cmp-effort (a b) "Compare the effort values of string A and B." - (let* ((def (if org-agenda-sort-noeffort-is-high 32767 -1)) + (let* ((def (if org-agenda-sort-noeffort-is-high most-positive-fixnum -1)) ;; `effort-minutes' property is not directly accessible from ;; the strings, but is stored as a property in `txt'. (ea (or (get-text-property @@ -7585,7 +7585,7 @@ org-cmp-ts \"timestamp_ia\", compare within each of these type. When TYPE is the empty string, compare all timestamps without respect of their type." - (let* ((def (if org-agenda-sort-notime-is-late 99999999 -1)) + (let* ((def (if org-agenda-sort-notime-is-late most-positive-fixnum -1)) (ta (or (and (string-match type (or (get-text-property 1 'type a) "")) (get-text-property 1 'ts-date a)) def)) @@ -8471,7 +8471,7 @@ org-agenda-compare-effort ;; current line but is stored as a property in `txt'. (let ((effort (get-text-property 0 'effort-minutes (org-get-at-bol 'txt)))) (funcall op - (or effort (if org-agenda-sort-noeffort-is-high 32767 -1)) + (or effort (if org-agenda-sort-noeffort-is-high most-positive-fixnum -1)) value))) (defun org-agenda-filter-expand-tags (filter &optional no-operator) -- 2.51.2
>From 46f03ff21834320187badb362de8fd73914f4c21 Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Fri, 28 Nov 2025 19:53:24 -0500 Subject: [PATCH 4/7] org-agenda: Apply stats text property to entire heading When cookies appeared at the very end of a heading then the text property would be absent from the org agenda entry. * lisp/org.el (org-refresh-stats-properties): Apply property to entire heading. * testing/lisp/test-org-agenda.el (test-org-agenda/sorting): Expect test to pass now. --- lisp/org.el | 3 ++- testing/lisp/test-org-agenda.el | 1 - 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lisp/org.el b/lisp/org.el index c07e21a07..802f966f0 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -8407,7 +8407,8 @@ org-refresh-stats-properties (stats (cond ((not denominator) numerator) ;percent ((= denominator 0) 0) (t (/ (* numerator 100) denominator))))) - (put-text-property (point) (progn (org-end-of-subtree t t) (point)) + (put-text-property (line-beginning-position) + (progn (org-end-of-subtree t t) (point)) 'org-stats stats))))))) (defun org-refresh-effort-properties () diff --git a/testing/lisp/test-org-agenda.el b/testing/lisp/test-org-agenda.el index 3408b4a1f..be17a558c 100644 --- a/testing/lisp/test-org-agenda.el +++ b/testing/lisp/test-org-agenda.el @@ -672,7 +672,6 @@ test-org-agenda/skip-scheduled-repeats-after-deadline (ert-deftest test-org-agenda/sorting () "Test if `org-agenda' sorts according to `org-agenda-sorting-strategy'." - :expected-result :failed ;; FIXME: test the following ;; urgency-up ;; category-up -- 2.51.2
>From eda87a2ea3af312178da5fbe0d83bfae6d0ef630 Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Fri, 7 Nov 2025 15:33:05 -0500 Subject: [PATCH 5/7] test-org-agenda/sorting/time: New test * testing/lisp/test-org-agenda.el (test-org-agenda/sorting): New test. --- testing/lisp/test-org-agenda.el | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/testing/lisp/test-org-agenda.el b/testing/lisp/test-org-agenda.el index be17a558c..312104ab1 100644 --- a/testing/lisp/test-org-agenda.el +++ b/testing/lisp/test-org-agenda.el @@ -765,6 +765,73 @@ test-org-agenda/sorting TODO [#C] habit [2/4] TODO pong [3/4]"))))) +(ert-deftest test-org-agenda/sorting/time () + "Test if `org-agenda' sorts according to `org-agenda-sorting-strategy'." + :expected-result :failed + ;; FIXME: test the following + ;; timestamp-up + ;; tsia-up + ;; ts-up + ;; time-up + (cl-flet ((call-agenda-with-priority (priority) + (let ((org-agenda-custom-commands + `(("f" "no fluff" todo "" + ((org-agenda-todo-keyword-format "") + (org-agenda-overriding-header "") + (org-agenda-prefix-format "") + (org-agenda-sorting-strategy ',priority)))))) + (org-agenda nil "f") + (string-trim + (substring-no-properties + (buffer-string)))))) + (org-test-at-time "2025-11-07" + (org-test-agenda-with-agenda + "* TODO scheduled yesterday +SCHEDULED: <2025-11-06> +* TODO scheduled today +SCHEDULED: <2025-11-07> +* TODO scheduled tomorrow +SCHEDULED: <2025-11-08> +* TODO deadline yesterday +DEADLINE: <2025-11-06> +* TODO deadline today +DEADLINE: <2025-11-07> +* TODO deadline tomorrow +DEADLINE: <2025-11-08> +* TODO both yesterday +SCHEDULED: <2025-11-06> DEADLINE: <2025-11-06> +* TODO both today +SCHEDULED: <2025-11-07> DEADLINE: <2025-11-07> +* TODO both tomorrow +SCHEDULED: <2025-11-08> DEADLINE: <2025-11-08> +* TODO nothing" + (should + (string-equal + (call-agenda-with-priority '(deadline-up scheduled-up)) + "both yesterday +deadline yesterday +both today +deadline today +both tomorrow +deadline tomorrow +scheduled yesterday +scheduled today +scheduled tomorrow +nothing")) + (should + (string-equal + (call-agenda-with-priority '(scheduled-down deadline-down)) + "nothing +deadline tomorrow +deadline today +deadline yesterday +scheduled tomorrow +both tomorrow +scheduled today +both today +scheduled yesterday +both yesterday")))))) + (ert-deftest test-org-agenda/tags-sorting () "Test if `org-agenda' sorts tags according to `org-tags-sort-function'." (let ((string-length< (lambda (s1 s2) -- 2.51.2
>From 6914376ddbb0def15263c0ce4ec056ac3b4713ed Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Sun, 16 Nov 2025 17:18:42 -0500 Subject: [PATCH 6/7] lisp/org-agenda.el (org-cmp-ts): Simplify * lisp/org-agenda.el (org-cmp-ts): Simplify. --- lisp/org-agenda.el | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index 08d64e400..60d1e30bd 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -7585,15 +7585,14 @@ org-cmp-ts \"timestamp_ia\", compare within each of these type. When TYPE is the empty string, compare all timestamps without respect of their type." - (let* ((def (if org-agenda-sort-notime-is-late most-positive-fixnum -1)) - (ta (or (and (string-match type (or (get-text-property 1 'type a) "")) - (get-text-property 1 'ts-date a)) - def)) - (tb (or (and (string-match type (or (get-text-property 1 'type b) "")) - (get-text-property 1 'ts-date b)) - def))) - (cond ((if ta (and tb (< ta tb)) tb) -1) - ((if tb (and ta (< tb ta)) ta) +1)))) + (cl-flet ((get-timestamp (entry) + (or (and (string-match type (or (get-text-property 1 'type entry) "")) + (get-text-property 1 'ts-date entry)) + (if org-agenda-sort-notime-is-late most-positive-fixnum -1)))) + (let ((ta (get-timestamp a)) + (tb (get-timestamp b))) + (cond ((< ta tb) -1) + ((< tb ta) +1))))) (defsubst org-cmp-habit-p (a b) "Compare the todo states of strings A and B." -- 2.51.2
>From df393e2a8c4a5bc58144bad1d104173bf5a6daa6 Mon Sep 17 00:00:00 2001 From: Morgan Smith <[email protected]> Date: Sun, 16 Nov 2025 18:00:08 -0500 Subject: [PATCH 7/7] WIP: try harder to get time in org-cmp-ts --- lisp/org-agenda.el | 13 +++++++++++++ testing/lisp/test-org-agenda.el | 1 - 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el index 60d1e30bd..9bcc998f2 100644 --- a/lisp/org-agenda.el +++ b/lisp/org-agenda.el @@ -7588,6 +7588,19 @@ org-cmp-ts (cl-flet ((get-timestamp (entry) (or (and (string-match type (or (get-text-property 1 'type entry) "")) (get-text-property 1 'ts-date entry)) + (when-let* ((epom (org-element-at-point (get-text-property 1 'org-hd-marker entry))) + (timestamp + (cond + ((string-equal type "") + (or (org-entry-get epom "SCHEDULED") + (org-entry-get epom "DEADLINE") + (org-entry-get epom "TIMESTAMP") + (org-entry-get epom "TIMESTAMP_IA"))) + ((string-equal type "scheduled") (org-entry-get epom "SCHEDULED")) + ((string-equal type "deadline") (org-entry-get epom "DEADLINE")) + ((string-equal type "timestamp_ia") (org-entry-get epom "TIMESTAMP_IA")) + ((string-equal type "timestamp") (org-entry-get epom "TIMESTAMP"))))) + (org-time-string-to-absolute timestamp)) (if org-agenda-sort-notime-is-late most-positive-fixnum -1)))) (let ((ta (get-timestamp a)) (tb (get-timestamp b))) diff --git a/testing/lisp/test-org-agenda.el b/testing/lisp/test-org-agenda.el index 312104ab1..7c5dd89df 100644 --- a/testing/lisp/test-org-agenda.el +++ b/testing/lisp/test-org-agenda.el @@ -767,7 +767,6 @@ test-org-agenda/sorting (ert-deftest test-org-agenda/sorting/time () "Test if `org-agenda' sorts according to `org-agenda-sorting-strategy'." - :expected-result :failed ;; FIXME: test the following ;; timestamp-up ;; tsia-up -- 2.51.2
