Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

>> So technically, a standalone DEADLINE + repeater isn't allowed -- a
>> repeating task must always have a start date.
>
> May we then use org-deadline-warning-days/timestamp warntime spec as DTSTART?
> VALARM component is not fitting for warning days anyway.
>
>> But still, maybe we should stick to the requirement, and only export
>> repeater on SCHEDULED. That would simplify the implementation. The
>> downside is that repeating deadlines won't show up in iCalendar, which
>> seems undesirable.
>
> Agree. We should better stick to the spec.

I took a closer look into how other programs handle RRULE, DTSTART, DUE.

I tried the following CalDav servers:

Nextcloud, radicale

And the following clients:

Tasks.org, Thunderbird, Evolution. (I did not use Nextcloud client
because it doesn't support repeating tasks, even though the Nextcloud
server does).

Thunderbird and Evolution clients do not allow creating repeating tasks
without start date -- if you try to do so, they will force you to
specify one.

Tasks.org client does allow repeating tasks with only a deadline (no
start date).  Nextcloud and radicale servers happily accept the
repeating deadline from Tasks.org without start date. When I download
the ICS file from the server, the VTODO contains RRULE and DUE, but not
DTSTART. When I validate the ICS file with icalendar.org [1], it accepts
the ICS as valid, even though it seemingly violates the spec by missing
DTSTART.

So, it seems there is some inconsistency about this in the iCalendar
ecosystem.

I have not yet reached a firm conclusion on the best solution, but am
leaning towards your suggestion to use org-deadline-warning-days for
DTSTART in this case. I'll try to have a more concrete, updated patch on
this ready in a couple weeks or so.


[1] https://icalendar.org/validator.html

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> So the RRULE applies to both DTSTART and DUE, and the repeats continue
> past DUE.
>
> But, another thing to note from the definition of DTSTART [2]:
>
>   This property [DTSTART] is REQUIRED in all types of recurring
>   calendar components that specify the "RRULE" property.
>
> So technically, a standalone DEADLINE + repeater isn't allowed -- a
> repeating task must always have a start date.

May we then use org-deadline-warning-days/timestamp warntime spec as DTSTART?
VALARM component is not fitting for warning days anyway.

> But still, maybe we should stick to the requirement, and only export
> repeater on SCHEDULED. That would simplify the implementation. The
> downside is that repeating deadlines won't show up in iCalendar, which
> seems undesirable.

Agree. We should better stick to the spec.

>> If we want to leave as many options as possible to the users, we can (1)
>> Implement ICALENAR_DUE property that will set DUE explicitly on export;
>> (2) ICALENDAR_DUE may allow special values that will indicate how to
>> treat Org DEADLINEs - make them into DUE, use Org DEADLINE as a
>> bound for SCHEDULED repeater, or ignore DEADLINE completely.
>
> A couple of these behaviors can already be achieved by customizing
> `org-icalendar-use-deadline' (making DUE or ignoring). For using
> DEADLINE as a bound, we could potentially add another option for that.

Yup. That's what I meant.

>> Is there any reason for this? May we instead export to a single VEVENT
>> with appropriate RDATE list?
>
> I guess if there are multiple timestamps with repeaters, it's easier to
> export these as separate VEVENT, because it's not possible to have
> multiple RRULE in one VEVENT.
>
> But, your suggestion earlier in thread could also solve this: in case of
> different repeaters, we can use RDATE to generate occurrences manually
> sufficiently far into future (with defcustom for "how far").

RDATE is exactly what I had in mind here.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> The question is: does iCalendar allow something like
>
> DTSTART;TZID=America/New_York:19970105T083000
> RRULE:FREQ=YEARLY
> DUE;TZID=America/New_York:20070105T083000
>
> and repeats past DUE?
>
> If not, we have to choose when exporting from Org source - either to
> keep DUE or not.

>From the defintion of RRULE [1]: 

  If the duration of the recurring component is specified with the
  "DTEND" or "DUE" property, then the same exact duration will apply
  to all the members of the generated recurrence set.

So the RRULE applies to both DTSTART and DUE, and the repeats continue
past DUE.

But, another thing to note from the definition of DTSTART [2]:

  This property [DTSTART] is REQUIRED in all types of recurring
  calendar components that specify the "RRULE" property.

So technically, a standalone DEADLINE + repeater isn't allowed -- a
repeating task must always have a start date.

But my impression is that not all iCalendar programs respect this. In
particular, Tasks.org app with Nextcloud server seemed to allow a
standalone repeating deadline. But I will check this more carefully, and
also in a couple more programs (radicale, Evolution).

But still, maybe we should stick to the requirement, and only export
repeater on SCHEDULED. That would simplify the implementation. The
downside is that repeating deadlines won't show up in iCalendar, which
seems undesirable.

[1] https://www.rfc-editor.org/rfc/rfc5545#section-3.8.5.3:
[2] https://www.rfc-editor.org/rfc/rfc5545#section-3.8.2.4

> If we want to leave as many options as possible to the users, we can (1)
> Implement ICALENAR_DUE property that will set DUE explicitly on export;
> (2) ICALENDAR_DUE may allow special values that will indicate how to
> treat Org DEADLINEs - make them into DUE, use Org DEADLINE as a
> bound for SCHEDULED repeater, or ignore DEADLINE completely.

A couple of these behaviors can already be achieved by customizing
`org-icalendar-use-deadline' (making DUE or ignoring). For using
DEADLINE as a bound, we could potentially add another option for that.

> Is there any reason for this? May we instead export to a single VEVENT
> with appropriate RDATE list?

I guess if there are multiple timestamps with repeaters, it's easier to
export these as separate VEVENT, because it's not possible to have
multiple RRULE in one VEVENT.

But, your suggestion earlier in thread could also solve this: in case of
different repeaters, we can use RDATE to generate occurrences manually
sufficiently far into future (with defcustom for "how far").

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> Ihor Radchenko  writes:
>
>> Another scenario we may need to consider is when schedule has a repeater
>> while deadline does not, and vice versa. The former scenario is probably
>> valid - a VTODO with limited number of occurrences.
>
> That is an interesting idea; and we can use the UNTIL or COUNT keywords
> in RRULE to implement it.
>
> However, it doesn't seem completely faithful to the way the TODO ends up
> in the Org Agenda (or does Org have some option to use DEADLINE to bound
> a repeating SCHEDULED in this way?)

DUE in iCalendar and DEADLINE in Org are not exactly the same. So, of
course, there is a room for ambiguity.

The question is: does iCalendar allow something like

DTSTART;TZID=America/New_York:19970105T083000
RRULE:FREQ=YEARLY
DUE;TZID=America/New_York:20070105T083000

and repeats past DUE?

If not, we have to choose when exporting from Org source - either to
keep DUE or not.

> I think the most faithful way to represent different SCHEDULED and
> DEADLINE repeaters is to export 2 separate VTODOs, each with different
> RRULE. Then the exported iCalendar will look just like the Org
> Agenda. It is also in line with how ox-icalendar exports multiple
> timestamps to separate VEVENTs.

If we want to leave as many options as possible to the users, we can (1)
Implement ICALENAR_DUE property that will set DUE explicitly on export;
(2) ICALENDAR_DUE may allow special values that will indicate how to
treat Org DEADLINEs - make them into DUE, use Org DEADLINE as a
bound for SCHEDULED repeater, or ignore DEADLINE completely.

> That said, I am not really happy with this solution either. The fact
> that ox-icalendar can create multiple VEVENT per entry already creates
> headaches for any setup doing bidirectional sync between Org and
> iCalendar, such as with org-caldav, ical2org.awk, or ical2orgpy. And I
> am hesitant to make this problem worse, by making it happen for VTODO as
> well.

Is there any reason for this? May we instead export to a single VEVENT
with appropriate RDATE list?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> Another scenario we may need to consider is when schedule has a repeater
> while deadline does not, and vice versa. The former scenario is probably
> valid - a VTODO with limited number of occurrences.

That is an interesting idea; and we can use the UNTIL or COUNT keywords
in RRULE to implement it.

However, it doesn't seem completely faithful to the way the TODO ends up
in the Org Agenda (or does Org have some option to use DEADLINE to bound
a repeating SCHEDULED in this way?)

I think the most faithful way to represent different SCHEDULED and
DEADLINE repeaters is to export 2 separate VTODOs, each with different
RRULE. Then the exported iCalendar will look just like the Org
Agenda. It is also in line with how ox-icalendar exports multiple
timestamps to separate VEVENTs.

That said, I am not really happy with this solution either. The fact
that ox-icalendar can create multiple VEVENT per entry already creates
headaches for any setup doing bidirectional sync between Org and
iCalendar, such as with org-caldav, ical2org.awk, or ical2orgpy. And I
am hesitant to make this problem worse, by making it happen for VTODO as
well.

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> Attached is the (I think) final version of the patch. I'll install it
> soon, unless I hear otherwise.

I have no further comments. Thanks!

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> Looks reasonable, but I have one comment on the code.
> We should not use user-defined hooks for things that must be executed.
> Instead, we should better explicitly call the necessary functions.

Thanks, I updated the patch to explicitly call the function.

Also, I tweaked the coding-system-for-write to be a bit safer, in case
of edge cases where utf-8 doesn't work -- I think RFC 5545 just says
it's the default charset.

Attached is the (I think) final version of the patch. I'll install it
soon, unless I hear otherwise.

PS I haven't forgotten your feedback on the original VTODO-related
patches (thanks for that review). I'll work on that next, but it might
take me a bit longer.

>From aa59625cd08dcee767f42ad8d45d8902aa8d38bd Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sat, 1 Apr 2023 16:53:35 -0700
Subject: [PATCH] ox-icalendar: Use consistent CRLF line endings

Fixes issue where the ox-icalendar export uses an inconsistent mix of
dos and unix style line endings.

* lisp/ox-icalendar.el (org-icalendar-fold-string): No longer converts
to CRLF, instead delegating that to `org-icalendar--post-process-file'.
(org-icalendar--post-process-file): New function to handle exported
file post-processing.  Converts EOL to CRLF, and then runs
`org-icalendar-after-save-hook'.
(org-icalendar-export-to-ics, org-icalendar-export-current-agenda,
org-icalendar--combine-files): Call `org-icalendar--post-process-file'
instead of running `org-icalendar-after-save-hook' directly.
* testing/lisp/test-ox-icalendar.el: New file for unit tests of
ox-icalendar.  Add an initial test for CRLF line endings.

See also:

https://list.orgmode.org/87o7oetneo.fsf@localhost/T/#m3e3eb80f9fc51ba75854b33ebfe9ecdefa2ded24

https://list.orgmode.org/orgmode/87ilgljv6i.fsf@localhost/
---
 etc/ORG-NEWS  | 12 +
 lisp/ox-icalendar.el  | 27 ---
 testing/lisp/test-ox-icalendar.el | 44 +++
 3 files changed, 74 insertions(+), 9 deletions(-)
 create mode 100644 testing/lisp/test-ox-icalendar.el

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ac233a986..9f7d01707 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -23,6 +23,18 @@ If you still want to use python-mode with ob-python, you might
 consider [[https://gitlab.com/jackkamm/ob-python-mode-mode][ob-python-mode-mode]], where the code to support python-mode
 has been ported to.
 
+*** =ox-icalendar.el= line ending fix may affect downstream packages
+
+iCalendar export now uses dos-style CRLF ("\r\n") line endings
+throughout, as required by the iCalendar specification (RFC 5545).
+Previously, the export used an inconsistent mix of dos and unix line
+endings.
+
+This might cause errors in external packages that parse output from
+ox-icalendar.  In particular, older versions of org-caldav may
+encounter issues, and users are advised to update to the most recent
+version of org-caldav.  See [[https://github.com/dengste/org-caldav/commit/618bf4cdc9be140ca1993901d017b7f18297f1b8][this org-caldav commit]] for more information.
+
 ** New and changed options
 *** New ~org-cite-natbib-export-bibliography~ option defining fallback bibliography style
 
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 81a77a770..ccc237721 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -540,12 +540,23 @@ (defun org-icalendar-fold-string (s)
 	;; line, real contents must be split at 74 chars.
 	(while (< (setq chunk-end (+ chunk-start 74)) len)
 	  (setq folded-line
-		(concat folded-line "\r\n "
+		(concat folded-line "\n "
 			(substring line chunk-start chunk-end))
 		chunk-start chunk-end))
-	(concat folded-line "\r\n " (substring line chunk-start))
-(org-split-string s "\n") "\r\n")))
-
+	(concat folded-line "\n " (substring line chunk-start))
+(org-split-string s "\n") "\n")))
+
+(defun org-icalendar--post-process-file (file)
+  "Post-process the exported iCalendar FILE.
+Converts line endings to dos-style CRLF as per RFC 5545, then
+runs `org-icalendar-after-save-hook'."
+  (with-temp-buffer
+(insert-file-contents file)
+(let ((coding-system-for-write (coding-system-change-eol-conversion
+last-coding-system-used 'dos)))
+  (write-region nil nil file)))
+  (run-hook-with-args 'org-icalendar-after-save-hook file)
+  nil)
 
 
 ;;; Filters
@@ -932,8 +943,7 @@ (defun org-icalendar-export-to-ics
 (org-export-to-file 'icalendar outfile
   async subtreep visible-only body-only
   '(:ascii-charset utf-8 :ascii-links-to-notes nil)
-  '(lambda (file)
-	 (run-hook-with-args 'org-icalendar-after-save-hook file) nil
+  #'org-icalendar--post-process-file)))
 
 ;;;###autoload
 (defun org-icalendar-export-agenda-files ( async)
@@ -1019,7 +1029,7 @@ (defun org-icalendar-export-current-agenda (file)
 	(or (org-string-nw-p org-icalendar-timezone) (format-time-string "%Z"))
 	

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

>> We likely want (according to 34.10.1 Basic Concepts of Coding Systems):
>
> I attach a new patch, which takes the approach of converting to
> utf-8-dos in `org-icalendar-after-save-hook', instead of converting
> newlines in `org-icalendar-fold-string'.
>
> I think this way is simpler, and should be more robust across locales.
>
> Note, this means the string returned by `org-export-as' won't contain
> CRLF. Instead, the newlines are converted during post-process.

Looks reasonable, but I have one comment on the code.

> +(add-hook 'org-icalendar-after-save-hook #'org-icalendar--convert-eol -90)

We should not use user-defined hooks for things that must be executed.
Imagine that a user customizes the hook after loading ox-icalendar and
removes the call to `org-icalendar--convert-eol'?

Instead, we should better explicitly call the necessary functions.
I was told this by Emacs devs.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> So, we should probably override `org-export-coding-system', even when it
> is set. iCalendar demands UTF8 anyway.

Also, ox-icalendar already sets ":ascii-charset utf-8" in the ext-plist
during export.

> We likely want (according to 34.10.1 Basic Concepts of Coding Systems):

I attach a new patch, which takes the approach of converting to
utf-8-dos in `org-icalendar-after-save-hook', instead of converting
newlines in `org-icalendar-fold-string'.

I think this way is simpler, and should be more robust across locales.

Note, this means the string returned by `org-export-as' won't contain
CRLF. Instead, the newlines are converted during post-process.

>From 04761429f82bfd2aee63f4978afec3449abaa37d Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sat, 1 Apr 2023 16:53:35 -0700
Subject: [PATCH] ox-icalendar: Use consistent CRLF line endings

Fixes issue where the ox-icalendar export uses an inconsistent mix of
dos and unix style line endings.

* lisp/ox-icalendar.el (org-icalendar-fold-string): No longer converts
to CRLF, instead delegating that to `org-icalendar--convert-eol'.
(org-icalendar--convert-eol): New function to convert EOL to CRLF. It
runs early in `org-icalendar-after-save-hook'.
* testing/lisp/test-ox-icalendar.el: New file for unit tests of
ox-icalendar.  Add an initial test for CRLF line endings.

See also:

https://list.orgmode.org/87o7oetneo.fsf@localhost/T/#m3e3eb80f9fc51ba75854b33ebfe9ecdefa2ded24

https://list.orgmode.org/orgmode/87ilgljv6i.fsf@localhost/
---
 etc/ORG-NEWS  | 12 +
 lisp/ox-icalendar.el  | 14 +++---
 testing/lisp/test-ox-icalendar.el | 44 +++
 3 files changed, 67 insertions(+), 3 deletions(-)
 create mode 100644 testing/lisp/test-ox-icalendar.el

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ac233a986..9f7d01707 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -23,6 +23,18 @@ If you still want to use python-mode with ob-python, you might
 consider [[https://gitlab.com/jackkamm/ob-python-mode-mode][ob-python-mode-mode]], where the code to support python-mode
 has been ported to.
 
+*** =ox-icalendar.el= line ending fix may affect downstream packages
+
+iCalendar export now uses dos-style CRLF ("\r\n") line endings
+throughout, as required by the iCalendar specification (RFC 5545).
+Previously, the export used an inconsistent mix of dos and unix line
+endings.
+
+This might cause errors in external packages that parse output from
+ox-icalendar.  In particular, older versions of org-caldav may
+encounter issues, and users are advised to update to the most recent
+version of org-caldav.  See [[https://github.com/dengste/org-caldav/commit/618bf4cdc9be140ca1993901d017b7f18297f1b8][this org-caldav commit]] for more information.
+
 ** New and changed options
 *** New ~org-cite-natbib-export-bibliography~ option defining fallback bibliography style
 
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 81a77a770..7f675b5d0 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -540,12 +540,20 @@ (defun org-icalendar-fold-string (s)
 	;; line, real contents must be split at 74 chars.
 	(while (< (setq chunk-end (+ chunk-start 74)) len)
 	  (setq folded-line
-		(concat folded-line "\r\n "
+		(concat folded-line "\n "
 			(substring line chunk-start chunk-end))
 		chunk-start chunk-end))
-	(concat folded-line "\r\n " (substring line chunk-start))
-(org-split-string s "\n") "\r\n")))
+	(concat folded-line "\n " (substring line chunk-start))
+(org-split-string s "\n") "\n")))
 
+(defun org-icalendar--convert-eol (f)
+  "Convert line endings to CRLF as per RFC 5545."
+  (with-temp-buffer
+(insert-file-contents f)
+(let ((coding-system-for-write 'utf-8-dos))
+  (write-region nil nil f
+
+(add-hook 'org-icalendar-after-save-hook #'org-icalendar--convert-eol -90)
 
 
 ;;; Filters
diff --git a/testing/lisp/test-ox-icalendar.el b/testing/lisp/test-ox-icalendar.el
new file mode 100644
index 0..bfc756d51
--- /dev/null
+++ b/testing/lisp/test-ox-icalendar.el
@@ -0,0 +1,44 @@
+;;; test-ox-icalendar.el --- tests for ox-icalendar.el  -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2023  Jack Kamm
+
+;; Author: Jack Kamm 
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see .
+
+;;; Commentary:
+
+;; Tests checking 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> For issue 1, what `org-icalendar-fold-string' does when string already
> contains \r\n, you can see that it produces \r\r\n as follows:
>
> emacs -Q -l ox-icalendar
> M-:
> (org-icalendar-fold-string (org-icalendar-fold-string "Line1\nLine2"))
>
> This is why the patch removes the calls to `org-icalendar-fold-string'
> in `org-icalendar--vevent' and `org-icalendar--vtodo' -- otherwise we
> would add \r multiple times to the same string.
>
> To change this behavior of `org-icalendar-fold-string', we could
> modify the patch to do:
>
>  (defun org-icalendar-fold-string (s)
>"Fold string S according to RFC 5545."
>(replace-regexp-in-string
> -   "\n" "\r\n"
> +   "\r*\n" "\r\n"
>
> which would strip out any extra \r at end of line. Another alternative
> would be to use "\r?\n" instead of "\r*\n".

"\r*\n" looks safer.

> For the second issue -- when `org-export-coding-system' is dos (or
> similar), the file created by `org-export-to-file' will contain
> \r\r\n. This was already the pre-existing behavior, but note the patch
> does cause a minor change here: before the patch just the main body
> will have \r\r\n, but after the patch, the preamble will also have it.

I see.
Looking at
https://icalendar.org/iCalendar-RFC-5545/3-1-4-character-set.html:

There is not a property parameter to declare the charset used in a
property value. The default charset for an iCalendar stream is UTF-8 as
defined in [RFC3629].

So, we should probably override `org-export-coding-system', even when it
is set. iCalendar demands UTF8 anyway.

We likely want (according to 34.10.1 Basic Concepts of Coding Systems):

   The coding system ‘utf-8-emacs’ specifies that the data is
represented in the internal Emacs encoding (*note Text
Representations::).  This is like ‘raw-text’ in that no code conversion
happens, but different in that the result is multibyte data.  The name
‘emacs-internal’ is an alias for ‘utf-8-emacs-unix’ (so it forces no
conversion of end-of-line, unlike ‘utf-8-emacs’, which can decode all 3
kinds of end-of-line conventions).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> I now only have one minor concern about `org-icalendar-fold-string' when
> the original buffer contains DOS line endings. May they mess things up
> producing \r\r\n?

There are 2 issues here: what does `org-icalendar-fold-string' do when
string already contains \r, and what does `org-export-to-file' do when
`org-export-coding-system' or `buffer-file-coding-system' is dos-like.

In both cases, the patch doesn't change the existing behavior -- which
is to produce \r\r\n.

For issue 1, what `org-icalendar-fold-string' does when string already
contains \r\n, you can see that it produces \r\r\n as follows:

emacs -Q -l ox-icalendar
M-:
(org-icalendar-fold-string (org-icalendar-fold-string "Line1\nLine2"))

This is why the patch removes the calls to `org-icalendar-fold-string'
in `org-icalendar--vevent' and `org-icalendar--vtodo' -- otherwise we
would add \r multiple times to the same string.

To change this behavior of `org-icalendar-fold-string', we could
modify the patch to do:

 (defun org-icalendar-fold-string (s)
   "Fold string S according to RFC 5545."
   (replace-regexp-in-string
-   "\n" "\r\n"
+   "\r*\n" "\r\n"

which would strip out any extra \r at end of line. Another alternative
would be to use "\r?\n" instead of "\r*\n".

For the second issue -- when `org-export-coding-system' is dos (or
similar), the file created by `org-export-to-file' will contain
\r\r\n. This was already the pre-existing behavior, but note the patch
does cause a minor change here: before the patch just the main body
will have \r\r\n, but after the patch, the preamble will also have it.

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

>> I am looking at
>> https://icalendar.org/iCalendar-RFC-5545/3-6-2-to-do-component.html, and
>> I note that only BEGIN:VTODO and END:VTODO lines must actually have
>> CRLF. For example,
>> https://icalendar.org/iCalendar-RFC-5545/3-3-11-text.html has no
>> mentions of CRLF, but does talk about escaping staff.
>
> My reading of [1] is that all lines must end with CRLF:

Good to hear that we do not need to worry about the need to mix CRLF and
\n.

I now only have one minor concern about `org-icalendar-fold-string' when
the original buffer contains DOS line endings. May they mess things up
producing \r\r\n? If not, feel free to install the patch.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> Thanks!
> Note that I did not implement my suggestion because I am concerned if
> putting CRLF is safe as every single line ending.
>
> I am looking at
> https://icalendar.org/iCalendar-RFC-5545/3-6-2-to-do-component.html, and
> I note that only BEGIN:VTODO and END:VTODO lines must actually have
> CRLF. For example,
> https://icalendar.org/iCalendar-RFC-5545/3-3-11-text.html has no
> mentions of CRLF, but does talk about escaping staff.

My reading of [1] is that all lines must end with CRLF:

> The iCalendar object is organized into individual lines of text,
> called content lines. Content lines are delimited by a line break,
> which is a CRLF sequence

And in particular, for the component properties after BEGIN:VTODO, [1]
gives the general notation as:

> contentline   = name *(";" param ) ":" value CRLF

For example, the DTSTART notation [2] is:

>  dtstart= "DTSTART" dtstparam ":" dtstval CRLF

And the same is true for all the other properties.

[1] https://icalendar.org/iCalendar-RFC-5545/3-1-content-lines.html
[2] https://icalendar.org/iCalendar-RFC-5545/3-8-2-4-date-time-start.html

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> However you are right that other parts of the iCalendar export have
> inconsistent line endings. Currently, VEVENT and VTODO components have
> the correct CRLF endings, but the other parts of the VCALENDAR do not
> (such as the preamble).
>
> I like your suggestion in the above thread to just wrap the whole
> export in `org-icalendar-fold-string'.  Though I think it's slightly
> nicer to do it in `org-icalendar--vcalendar' instead of
> `org-icalendar-template'.
>
> So, I've attached a standalone patch to do this. It also fixes an issue
> with `org-icalendar-fold-string' where the last newline was missing
> "\r", and adds a unit test.

Thanks!
Note that I did not implement my suggestion because I am concerned if
putting CRLF is safe as every single line ending.

I am looking at
https://icalendar.org/iCalendar-RFC-5545/3-6-2-to-do-component.html, and
I note that only BEGIN:VTODO and END:VTODO lines must actually have
CRLF. For example,
https://icalendar.org/iCalendar-RFC-5545/3-3-11-text.html has no
mentions of CRLF, but does talk about escaping staff.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Ihor Radchenko  writes:

> Side note: here, and in other places, we use "\n" as end of line. Yet,
> for example
> https://icalendar.org/iCalendar-RFC-5545/3-8-2-4-date-time-start.html
> prescribes CRLF (\r\n). Also, see
> https://orgmode.org/list/87ilgljv6i.fsf@localhost
> If you are familiar with iCalendar spec, may you look through the
> ox-icalendar code and check other places where we do not conform to the
> newline spec?

org-icalendar--vtodo is wrapped in org-icalendar-fold-string, so this
"\n" gets converted to CRLF later on.

However you are right that other parts of the iCalendar export have
inconsistent line endings. Currently, VEVENT and VTODO components have
the correct CRLF endings, but the other parts of the VCALENDAR do not
(such as the preamble).

I like your suggestion in the above thread to just wrap the whole
export in `org-icalendar-fold-string'.  Though I think it's slightly
nicer to do it in `org-icalendar--vcalendar' instead of
`org-icalendar-template'.

So, I've attached a standalone patch to do this. It also fixes an issue
with `org-icalendar-fold-string' where the last newline was missing
"\r", and adds a unit test.

Note that fixing the line endings causes a surprising compatibility
issue with org-caldav. I fixed this problem on the org-caldav side, and
made a note in ORG-NEWS.

>From 712a4ef09b63b2f6bdec2a3967712be912dce0d2 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Thu, 30 Mar 2023 22:19:09 -0700
Subject: [PATCH] ox-icalendar: Use consistent CRLF line endings

Fixes issue where the ox-icalendar export uses an inconsistent mix of
dos and unix style line endings.

* lisp/ox-icalendar.el (org-icalendar-fold-string): Don't use "\r"
during the string construction, instead replace "\n" with "\r\n" after
string has been created.  This fixes an issue where the final "\n"
added by `org-element-normalize-string' was missing "\r".
(org-icalendar--vevent): Remove call to `org-icalendar-fold-string'.
(org-icalendar--vtodo): Remove call to `org-icalendar-fold-string'.
(org-icalendar--vcalendar): Wrap in `org-icalendar-fold-string'.
* testing/lisp/test-ox-icalendar.el: New file for unit tests of
ox-icalendar.  Add an initial test for CRLF line endings.

See also:

https://list.orgmode.org/87o7oetneo.fsf@localhost/T/#m3e3eb80f9fc51ba75854b33ebfe9ecdefa2ded24

https://list.orgmode.org/orgmode/87ilgljv6i.fsf@localhost/
---
 etc/ORG-NEWS  |  12 +++
 lisp/ox-icalendar.el  | 159 +++---
 testing/lisp/test-ox-icalendar.el |  46 +
 3 files changed, 138 insertions(+), 79 deletions(-)
 create mode 100644 testing/lisp/test-ox-icalendar.el

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ac233a986..9f7d01707 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -23,6 +23,18 @@ If you still want to use python-mode with ob-python, you might
 consider [[https://gitlab.com/jackkamm/ob-python-mode-mode][ob-python-mode-mode]], where the code to support python-mode
 has been ported to.
 
+*** =ox-icalendar.el= line ending fix may affect downstream packages
+
+iCalendar export now uses dos-style CRLF ("\r\n") line endings
+throughout, as required by the iCalendar specification (RFC 5545).
+Previously, the export used an inconsistent mix of dos and unix line
+endings.
+
+This might cause errors in external packages that parse output from
+ox-icalendar.  In particular, older versions of org-caldav may
+encounter issues, and users are advised to update to the most recent
+version of org-caldav.  See [[https://github.com/dengste/org-caldav/commit/618bf4cdc9be140ca1993901d017b7f18297f1b8][this org-caldav commit]] for more information.
+
 ** New and changed options
 *** New ~org-cite-natbib-export-bibliography~ option defining fallback bibliography style
 
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 81a77a770..06e90d032 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -526,25 +526,27 @@ (defun org-icalendar-cleanup-string (s)
 
 (defun org-icalendar-fold-string (s)
   "Fold string S according to RFC 5545."
-  (org-element-normalize-string
-   (mapconcat
-(lambda (line)
-  ;; Limit each line to a maximum of 75 characters.  If it is
-  ;; longer, fold it by using "\r\n " as a continuation marker.
-  (let ((len (length line)))
-	(if (<= len 75) line
-	  (let ((folded-line (substring line 0 75))
-		(chunk-start 75)
-		chunk-end)
-	;; Since continuation marker takes up one character on the
-	;; line, real contents must be split at 74 chars.
-	(while (< (setq chunk-end (+ chunk-start 74)) len)
-	  (setq folded-line
-		(concat folded-line "\r\n "
-			(substring line chunk-start chunk-end))
-		chunk-start chunk-end))
-	(concat folded-line "\r\n " (substring line chunk-start))
-(org-split-string s "\n") "\r\n")))
+  (replace-regexp-in-string
+   "\n" "\r\n"
+   (org-element-normalize-string
+(mapconcat
+ (lambda (line)
+   ;; Limit each line to a maximum of 75 

Re: [RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Ihor Radchenko]

Jack Kamm  writes:

> For patch 1 (unscheduled tasks):
>
> Currently, ox-icalendar does not allow creating an iCalendar task
> without a scheduled start date. If an Org TODO is missing a SCHEDULED
> timestamp, then ox-icalendar sets today as the scheduled start date for
> the exported task.
>
> Patch 1 changes this by adding a new customization
> org-icalendar-todo-force-scheduling. When non-nil, the start date is set
> to today (same as the current behavior). When nil, unscheduled Org TODOs
> are instead exported without a start date.
>
> I also propose the default value to be nil. Note, this is
> backwards-incompatible with the previous behavior!
>
> But I think it should be the default anyways, because IMO it is the more
> correct and useful behavior. An iCalendar VTODO without a DTSTART
> property is valid, and has the same meaning as an Org TODO without a
> SCHEDULED timestamp. Also, all the iCalendar programs I have tried
> support unscheduled tasks, including Thunderbird, Evolution, Nextcloud,
> and Tasks.org.

I agree that omitting DTSTART will make more sense.

> For patch 2 (repeating timestamps):
>
> I add recurrence rule (RRULE) export for repeating SCHEDULED and
> DEADLINE timestamps in TODOs, similar to how repeating non-TODO events
> are currently handled.
>
> The main complication here is that iCalendar's RRULE applies to both
> DTSTART and DUE properties; by contrast, Org's SCHEDULED and DEADLINE
> timestamps may have different repeaters. I am not sure the best way to
> handle the case where SCHEDULED and DEADLINE have different repeaters,
> so in that case I issue a warning and skip the repeater.

In the case of different repeaters, we can use RDATE
(https://icalendar.org/iCalendar-RFC-5545/3-8-5-2-recurrence-date-times.html)
and generate occurrences manually sufficiently far into future. ("how
far" should be a defcustom).

However, different repeaters for deadline and schedule are most likely a
mistake - we can report it via org-lint and in ox-icalendar, as warning.

Another scenario we may need to consider is when schedule has a repeater
while deadline does not, and vice versa. The former scenario is probably
valid - a VTODO with limited number of occurrences. The latter is likely
a mistake we should raise warning about. It is also not clear how to
represent moving event deadline in iCalendar.

See more inline comments below.

> +(defcustom org-icalendar-todo-force-scheduling nil
> +  "Non-nil means unscheduled tasks are exported as scheduled.
> +The current date is used as the scheduled time for such tasks."
> +  :group 'org-export-icalendar
> +  :type 'boolean)

Please add :package-version and possibly :safe keywords.
We may also refer to `org-icalendar-include-todo' in the docstring.

> -  (org-icalendar-convert-timestamp start "DTSTART" nil timezone) "\n"
> + (when start
> +   (concat (org-icalendar-convert-timestamp
> +start "DTSTART" nil timezone)
> +   "\n"))

Side note: here, and in other places, we use "\n" as end of line. Yet,
for example
https://icalendar.org/iCalendar-RFC-5545/3-8-2-4-date-time-start.html
prescribes CRLF (\r\n). Also, see
https://orgmode.org/list/87ilgljv6i.fsf@localhost
If you are familiar with iCalendar spec, may you look through the
ox-icalendar code and check other places where we do not conform to the
newline spec?

Ideally, we want a set of private functions ensuring proper prescribed
format for all the used iCalendar syntax entries. Otherwise, we will
keep forgetting about these subtleties.

> +(defun org-icalendar--rrule (unit value)
> +  (format "RRULE:FREQ=%s;INTERVAL=%d\n"

\r\n

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[RFC] ox-icalendar: Unscheduled tasks & repeating tasks

[Jack Kamm]

Hello,

The attached 2 patches add support for exporting unscheduled tasks and
repeating tasks to iCalendar, respectively.

For patch 1 (unscheduled tasks):

Currently, ox-icalendar does not allow creating an iCalendar task
without a scheduled start date. If an Org TODO is missing a SCHEDULED
timestamp, then ox-icalendar sets today as the scheduled start date for
the exported task.

Patch 1 changes this by adding a new customization
org-icalendar-todo-force-scheduling. When non-nil, the start date is set
to today (same as the current behavior). When nil, unscheduled Org TODOs
are instead exported without a start date.

I also propose the default value to be nil. Note, this is
backwards-incompatible with the previous behavior!

But I think it should be the default anyways, because IMO it is the more
correct and useful behavior. An iCalendar VTODO without a DTSTART
property is valid, and has the same meaning as an Org TODO without a
SCHEDULED timestamp. Also, all the iCalendar programs I have tried
support unscheduled tasks, including Thunderbird, Evolution, Nextcloud,
and Tasks.org.

For patch 2 (repeating timestamps):

I add recurrence rule (RRULE) export for repeating SCHEDULED and
DEADLINE timestamps in TODOs, similar to how repeating non-TODO events
are currently handled.

The main complication here is that iCalendar's RRULE applies to both
DTSTART and DUE properties; by contrast, Org's SCHEDULED and DEADLINE
timestamps may have different repeaters. I am not sure the best way to
handle the case where SCHEDULED and DEADLINE have different repeaters,
so in that case I issue a warning and skip the repeater.

>From 1bd268ab260d5077d7456c0d64fea36128772f86 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sun, 26 Mar 2023 07:43:53 -0700
Subject: [PATCH 1/2] ox-icalendar: Allow exporting unscheduled VTODOs

* lisp/ox-icalendar.el (org-icalendar-todo-force-scheduling): New
option to revert to previous export behavior of unscheduled TODOs.
(org-icalendar--vtodo): Don't force unscheduled TODOs to have a
scheduled start time of today, unless
`org-icalendar-todo-force-scheduling' is set.
---
 etc/ORG-NEWS | 15 +++
 lisp/ox-icalendar.el | 32 +---
 2 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ac233a986..fb4f82b29 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -23,6 +23,15 @@ If you still want to use python-mode with ob-python, you might
 consider [[https://gitlab.com/jackkamm/ob-python-mode-mode][ob-python-mode-mode]], where the code to support python-mode
 has been ported to.
 
+*** Icalendar export of TODOs no longer forces a start time
+
+For TODOs without a scheduled start time, ox-icalendar no longer
+forces them to have a scheduled start time of today when exporting.
+This makes it possible to create icalendar TODOs without a start time.
+
+To revert to the old behavior, set the new custom option
+~org-icalendar-todo-force-scheduling~ to non-nil.
+
 ** New and changed options
 *** New ~org-cite-natbib-export-bibliography~ option defining fallback bibliography style
 
@@ -111,6 +120,12 @@ backend used for evaluation of ClojureScript.
 official [[https://clojure.org/guides/deps_and_cli][Clojure CLI tools]].
 The command can be customized with ~ob-clojure-cli-command~.
 
+*** New ~org-icalendar-todo-force-scheduling~ option for old ox-icalendar TODO scheduling behavior
+
+Set ~org-icalendar-todo-force-scheduling~ to non-nil to revert to the
+old ox-icalendar TODO export behavior, that forced all exported TODOs
+to have a scheduled start time.
+
 ** New features
 *** Add support for ~logind~ idle time in ~org-user-idle-seconds~
 
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 81a77a770..63aefcc84 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -231,6 +231,12 @@ (defcustom org-icalendar-include-todo nil
 	  (repeat :tag "Specific TODO keywords"
 		  (string :tag "Keyword"
 
+(defcustom org-icalendar-todo-force-scheduling nil
+  "Non-nil means unscheduled tasks are exported as scheduled.
+The current date is used as the scheduled time for such tasks."
+  :group 'org-export-icalendar
+  :type 'boolean)
+
 (defcustom org-icalendar-include-bbdb-anniversaries nil
   "Non-nil means a combined iCalendar file should include anniversaries.
 The anniversaries are defined in the BBDB database."
@@ -776,21 +782,25 @@ (defun org-icalendar--vtodo
 Return VTODO component as a string."
   (let ((start (or (and (memq 'todo-start org-icalendar-use-scheduled)
 			(org-element-property :scheduled entry))
-		   ;; If we can't use a scheduled time for some
-		   ;; reason, start task now.
-		   (let ((now (decode-time)))
-		 (list 'timestamp
-			   (list :type 'active
- :minute-start (nth 1 now)
- :hour-start (nth 2 now)
- :day-start (nth 3 now)
- :month-start (nth 4 now)
- :year-start (nth 5 now)))
+   (when org-icalendar-todo-force-scheduling
+		 ;; If