Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

My preferences would be

1. If a timestamp does not include the TZ, then assume the local TZ
2. If a timestamp does include the TZ, honour that TZ
3. If the timestamp does not include a time component, default to 0:00:0
for the local TZ
4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.

Rationale is that the user should have ability to fully control how
their timestamps are represented. However, adding the TZ is probably
just a pain for the majority of use cases, so defaulting to the local
(wall) TZ is OK provided you can override this consistently by adding
explicit TZ values.

However, there is some devil in the details we need to work out. For
example, should we support both TZ names (like AEDT or Australia/Sydney)
and POSIX style +11/+11:00/+1100, should we add an option to tell org to
always add TZ info in timestamps which include time components and if
so, how complex will this need to be e.g. handle setting a future/past
timestamp which is in a different (daylight savings) offset and what
about the additional complexity in dealing with timestamp calculations
where dates might be in different offsets due to daylight savings -
while all quite possible, it does add significant complexity and this
may have adverse impact on performance. Not to say we shouldn't do it,
just that it will take significant work.

I suspect just the first part won't have major impact - at least no more
than enforcing UTC in org-2ft.

Tim

P.S. when you start to think about it, it is easy to see how Java
screwed up this stuff so badly!


Allen Li  writes:

> On Tue, Oct 31, 2017 at 10:41 PM, Tim Cross  wrote:
>>
>> I think this whole issue really requires a lot more analysis and
>> design. Just removing or cancelling various commits is unlikely to
>> improve matters and could result in new problems.
>
> You're right, but the new behavior was introduced fairly recently.
>
>> For org to work correctly, especially when interacting/interfacing with
>> other systems, such as external calendars, the use of timestamps must
>> handle timezones consistently and accurately. This is the only way that
>> any daylight savings calculations will work consistently as different
>> timezones have different rules for when daylight savings start/finish
>> (and these rules change).
>>
>> If the tests only support UTC/GMT timezone, they are poor tests and need
>> to be adjusted to use whatever the timezone is on the system running the
>> tests.
>>
>> I also wonder if there is some inconsistencies in how timestamps without
>> a time component are being handled. It would be good to know if the
>> issues Alan has observed exist when a full timestamp is used ie. one
>> with HH:MM:SS.s and not just date. If timezones are not been applied
>> consistently when choosing the default i.e. 00:00:00.0 with respect to
>> timezone offset, you will get inconsistencies when moving between
>> displayed (string) and calculated (number/seconds since epoch) values.
>
> I think we first need to agree on how Org mode should handle
> time.  What seems most natural to me is:
>
> * Floating point timestamps are Unix timestamps, seconds since Epoch.
> * Org format time strings are interpreted in the local machine's time zone.
>
> Let us assume that my timezone is UTC-07.  In that case,
> <2017-10-30> should be interpreted as 2017-10-30T00:00:00-0700,
> or 2017-10-30T07:00:00+.
>
> <2017-10-30>1509346800.02017-10-30T00:00:00-0700
>  2017-10-30T07:00:00+
> <2017-10-30 12:34>  1509392040.02017-10-30T12:34:00-0700
>  2017-10-30T19:34:00+
>
> Currently, org-2ft returns:
>
> <2017-10-30>1509321600.02017-10-29T17:00:00-0700
>  2017-10-30T00:00:00+
> <2017-10-30 12:34>  1509366840.02017-10-30T05:34:00-0700
>  2017-10-30T12:34:00+
>
> This is because org-2ft calls org-parse-time-string with t for
> zone, i.e. parse the time string as though it were UTC.  That
> should be apparent from the last column.
>
> Hopefully, we can agree that the former is the desired behavior.
> Working on this assumption, org-2ft should be changed to the
> original behavior, i.e., to not parse time strings as UTC.
>
> The question then becomes, what breaks if we just naively change
> org-2ft?  The new behavior was only added to org-2ft four months
> ago, so worst case is reverting every time-related change in the
> interim puts us back four months in time-related logic.
>
> But looking through the history, most of the time-related changes
> in the interim were to fix regressions caused by the initial
> logic change.  If we revert the original "regression", then those
> interim changes are also not needed.


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>
> My preferences would be
>
> 1. If a timestamp does not include the TZ, then assume the local TZ
> 2. If a timestamp does include the TZ, honour that TZ

Org mode does not support TZ in time strings and adding support isn't
the topic of the current bug.

I want to emphasize the distinction between timestamps and time
strings.  Timestamps are a
float value, Unix timestamps.  Time strings are Org mode specific,
like <2017-10-30 12:34:56>

Timestamps do not have timezone information since they describe an
exact (well, minus leap seconds)
point in time, the number of seconds after the epoch.

> 3. If the timestamp does not include a time component, default to 0:00:0

This is what Org mode does

> for the local TZ

This is what Org mode used to do, now it interprets it as 00:00:00 UTC.

> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>
> Rationale is that the user should have ability to fully control how
> their timestamps are represented. However, adding the TZ is probably
> just a pain for the majority of use cases, so defaulting to the local
> (wall) TZ is OK provided you can override this consistently by adding
> explicit TZ values.
>
> However, there is some devil in the details we need to work out. For
> example, should we support both TZ names (like AEDT or Australia/Sydney)
> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
> always add TZ info in timestamps which include time components and if
> so, how complex will this need to be e.g. handle setting a future/past
> timestamp which is in a different (daylight savings) offset and what
> about the additional complexity in dealing with timestamp calculations
> where dates might be in different offsets due to daylight savings -
> while all quite possible, it does add significant complexity and this
> may have adverse impact on performance. Not to say we shouldn't do it,
> just that it will take significant work.
>
> I suspect just the first part won't have major impact - at least no more
> than enforcing UTC in org-2ft.

Org mode does not support TZ in time strings currently.  While I would
like such a feature very much,
adding TZ support isn't the topic of the current bug, just fixing how
time strings are interpreted.

>
> Tim
>
> P.S. when you start to think about it, it is easy to see how Java
> screwed up this stuff so badly!

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Allen Li  writes:

> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>
>> My preferences would be
>>
>> 1. If a timestamp does not include the TZ, then assume the local TZ
>> 2. If a timestamp does include the TZ, honour that TZ
>
> Org mode does not support TZ in time strings and adding support isn't
> the topic of the current bug.
>

I disagree. The root cause of the bug is due to a lack of clarity
regarding timezones. I also suspect this is also the basic cause of
issues relating to the use of timestamps/time strings and daylight
savings, especially when it comes to performing calculations etc.

> I want to emphasize the distinction between timestamps and time
> strings.  Timestamps are a
> float value, Unix timestamps.  Time strings are Org mode specific,
> like <2017-10-30 12:34:56>

Org mode refers to what you are calling time strings as timestamps. In
reality, there is no difference - one is just a numeric representation
and the other is a string representation. It is good you have clarified
your definition to reduce confusion. However, I think the problems are
arising because of a lack of explicit TZ handling.

>
> Timestamps do not have timezone information since they describe an
> exact (well, minus leap seconds)
> point in time, the number of seconds after the epoch.
>

A point in time is measured in number of seconds since epoch. However,
if you want to use local (wall) time to display that point, you have to
include TZ information when converting that number to a more
meaningful/usable (for humans) format. The point 'now' for me is UTC+1100
and for you (based on your previous post) UTC-0700, so our
representations in string format of this value will be different. Even
on a single machine, it is also relevant. For example, if I have two org
timestamps (your times strings) and I want to calculate the number of
hours between the two, I need to include TZ information as one timestamp
might be during daylight savings time and the other outside daylight
savings time. Any calculation which fails to consider TZ information in
this case will be out by an hour.

If we just take the simple solution and ignore TZ information, we will
break calculations which cross daylight savings 'boarders'. I also
suspect we will get inconsistent behaviour when integrating with other
systems (such as external calendars, ical integration or sharing
timestamp data with others etc).

>> 3. If the timestamp does not include a time component, default to 0:00:0
>
> This is what Org mode does
>
>> for the local TZ
>
> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>
>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>
>> Rationale is that the user should have ability to fully control how
>> their timestamps are represented. However, adding the TZ is probably
>> just a pain for the majority of use cases, so defaulting to the local
>> (wall) TZ is OK provided you can override this consistently by adding
>> explicit TZ values.
>>
>> However, there is some devil in the details we need to work out. For
>> example, should we support both TZ names (like AEDT or Australia/Sydney)
>> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
>> always add TZ info in timestamps which include time components and if
>> so, how complex will this need to be e.g. handle setting a future/past
>> timestamp which is in a different (daylight savings) offset and what
>> about the additional complexity in dealing with timestamp calculations
>> where dates might be in different offsets due to daylight savings -
>> while all quite possible, it does add significant complexity and this
>> may have adverse impact on performance. Not to say we shouldn't do it,
>> just that it will take significant work.
>>
>> I suspect just the first part won't have major impact - at least no more
>> than enforcing UTC in org-2ft.
>
> Org mode does not support TZ in time strings currently.  While I would
> like such a feature very much,
> adding TZ support isn't the topic of the current bug, just fixing how
> time strings are interpreted.
>

We could make the decision that all org timestamps are relative to the
local TZ (ignoring the daylight savings issue) and that would probably
be OK for most users, but I feel the whole issue is due to a lack of
adequate/consistent TZ interpretation. The problem with org-2ft is that
it forces the conversion into UTC when the input time string may
actually be in local time, resulting in a number which is out by
whatever the local offset is. When you then convert back, your string
representation will be out by at least that amount (possibly more
depending on whether TZ info is applied in the conversion back to a
string).

If it wasn't for the benefits of org files being essentially plain text
files, the easiest solution would possibly be to represent timestamps as
a number and use an overlay or similar to convert it to a human readable
value based on whatever the user se

Re: [O] Footnotes-per-element?

[Tory S. Anderson]

THanks for this; it may be a workable solution. The trick is, I 
didn't generate the footnotes myself -- I have concatenated a 
bunch of html -> orgmode pandoc outputs. Still an elisp 
find-and-replace might do what I need. I was just hoping for a 
quick switch to flip for localized footnotes.


Thanks!
- Tory

Kaushal Modi  writes:


On Fri, Oct 27, 2017 at 8:12 AM Tory S. Anderson <
torys.ander...@gmail.com> wrote:

I have a single org file that represents a collection of
articles;
each article starts with a first-level heading and includes 
its
own footnotes, always restarting from 0. I have a ** 
Footnotes
section in each article with the appropriate footnotes, but 
it
isn't respected when I follow one of the footnote 
references;
following [fn:3] in the any article, for example, will take 
me to

[fn:3] of the first article's footnotes. What's the best way
around this, so that each article will use its own footnotes
section?


I just increment the footnotes across all the articles. So, for
example, if article 1 has footnotes 1 and 2, article 2 will 
continue
from footnote number 3. The beauty is that if you export article 
2,

it will auto-reset the footnote reference to number 1.

The key is to not manually number the footnotes and let C-c C-x 
f

(bound to org-footnote-action) do its thing.

Hope that helps. 
--

Kaushal Modi

Re: [O] macro replacement in code block?

[Kaushal Modi]

On Tue, Oct 31, 2017 at 11:21 PM David Rogoff  wrote:

> Thanks - I’ll have to look into noweb.  I’m not familiar with it.
>

This turned out to be much complicated than I expected..

I still don't have a fully working solution (see below). Hopefully someone
can help figure out the reason for "Invalid read syntax: #" error.

Below example Org file will work right away. But it will fail with the
above error the moment second line (for more macros) is added to org-macros
block.

=
1. Eval the =org-macros= block once (=C-c C-c=)
   - Always use that =org-macros= block to define the Org macros for this
file.
   - Hitting =C-c C-c= will insert those macros in this file.
   - This block is reused as a noweb-ref in other blocks.. so keeping just
this
 block as a source for all Org macros will be convenient.
2. Export this file (=C-c C-e h o=)

#+BEGIN_SRC org :noweb-ref org-macros :results output replace :exports none
,#+MACRO: SEC_FOO Foo Topic
#+END_SRC

#+NAME: ehdr
#+BEGIN_SRC emacs-lisp :var macro="n" :noweb yes :results raw :exports none
(with-temp-buffer
  (insert "<>\n")
  (let ((start (point)))
(insert "// ---\n")
(insert (concat "// {{{" macro "}}}"))
(org-mode)
(org-macro-replace-all
 (append org-macro-templates org-export-global-macros))
(buffer-substring-no-properties start (point-max
#+END_SRC
** TODO Fix Invalid read syntax: "#"
This solution /almost/ works.

If the content of the =org-macros= block is changed from:
#+BEGIN_SRC org
,#+MACRO: SEC_FOO Foo Topic
#+END_SRC

To:
#+BEGIN_SRC org
,#+MACRO: SEC_FOO Foo Topic
,#+MACRO: SEC_BAR Bar Topic
#+END_SRC

We get this error:
#+BEGIN_EXAMPLE
Invalid read syntax: "#"
#+END_EXAMPLE

Looks like that happens due to ~(insert "<>\n")~ when the
~org-macros~ ~noweb-ref~ has multiple lines. But I cannot figure out why
having
multiple lines is a problem.
** Section 1: {{{SEC_FOO}}}
#+BEGIN_SRC verilog :noweb yes
<>
parameter A = 1;
#+END_SRC
** COMMENT Section 2: {{{SEC_BAR}}}
#+BEGIN_SRC verilog :noweb yes
<>
parameter A = 1;
#+END_SRC
=

-- 

Kaushal Modi

Re: [O] macro replacement in code block?

[Kaushal Modi]

On Wed, Nov 1, 2017 at 11:51 AM Kaushal Modi  wrote:

> I still don't have a fully working solution (see below). Hopefully someone
> can help figure out the reason for "Invalid read syntax: #" error.
>

OK, I found a workaround. Below solution works:

1. Eval the =org-macros= block once (=C-c C-c=)
   - Always use that =org-macros= block to define the Org macros for this
file.
   - Hitting =C-c C-c= will insert those macros in this file.
   - This block is reused as a noweb-ref in other blocks.. so keeping just
this
 block as a source for all Org macros will be convenient.
2. Then export that file (=C-c C-e h o=)

=
#+BEGIN_SRC org :noweb-ref org-macros :results output replace :exports none
,#+MACRO: SEC_FOO Foo Topic
,#+MACRO: SEC_BAR Bar Topic
#+END_SRC

#+NAME: ehdr
#+BEGIN_SRC emacs-lisp :var macro="n" :noweb yes :results raw :exports none
(with-temp-buffer
  (insert "
<>
\n")
  (let ((start (point)))
(insert "// ---\n")
(insert (concat "// {{{" macro "}}}"))
(org-mode)
(org-macro-replace-all
 (append org-macro-templates org-export-global-macros))
(buffer-substring-no-properties start (point-max
#+END_SRC

* Section 1: {{{SEC_FOO}}}
#+BEGIN_SRC verilog :noweb yes
<>
parameter A = 1;
#+END_SRC

* Section 2: {{{SEC_BAR}}}
#+BEGIN_SRC verilog :noweb yes
<>
parameter A = 1;
#+END_SRC
=
-- 

Kaushal Modi

[O] Multi-line noweb expansion in elisp insert form (Was Re: macro replacement in code block?)

[Kaushal Modi]

Hello all,

While trying to come up with a solution for this problem[1], I ended up
with a new problem :)

On Wed, Nov 1, 2017 at 11:51 AM Kaushal Modi  wrote:

> I still don't have a fully working solution (see below). Hopefully someone
> can help figure out the reason for "Invalid read syntax: #" error.
>
> Below example Org file will work right away. But it will fail with the
> above error the moment second line (for more macros) is added to org-macros
> block.
>

Here's a minimal working example to recreate that specific problem:

- The problem is seen when you C-c C-c the "noworky" block.

=

#+BEGIN_SRC org :noweb-ref org-macros-1-line :results output replace
:exports none
,#+MACRO: AA 1
#+END_SRC

#+RESULTS:
#+MACRO: AA 1

#+BEGIN_SRC org :noweb-ref org-macros-2-lines :results output replace
:exports none
,#+MACRO: BB 2
,#+MACRO: CC 3
#+END_SRC

#+RESULTS:
#+MACRO: BB 2
#+MACRO: CC 3

* This works
#+NAME: works
#+BEGIN_SRC emacs-lisp :var macro="AA" :noweb yes :results raw :exports none
(with-temp-buffer
  (insert "<>\n")
  (insert (concat "{{{" macro "}}}"))
  (org-mode)
  (org-macro-replace-all
   (append org-macro-templates org-export-global-macros))
  (buffer-substring-no-properties (point-min) (point-max)))
#+END_SRC
#+CALL: works()

* This does not work
#+NAME: noworky
#+BEGIN_SRC emacs-lisp :var macro="BB" :noweb yes :results raw :exports none
(with-temp-buffer
  (insert "<>\n")
  (insert "// ---\n")
  (insert (concat "{{{" macro "}}}"))
  (org-mode)
  (org-macro-replace-all
   (append org-macro-templates org-export-global-macros))
  (buffer-substring-no-properties (point-min) (point-max)))
#+END_SRC

If you uncomment the below =#+CALL= line, you will get:
#+BEGIN_EXAMPLE
org-babel-execute:emacs-lisp: Invalid read syntax: "#"
#+END_EXAMPLE

when you try to export (or even if you =C-c C-c= that =noworky= block.

# #+CALL: noworky()
=

[1]: http://lists.gnu.org/archive/html/emacs-orgmode/2017-11/msg7.html

-- 

Kaushal Modi

Re: [O] macro replacement in code block?

[David Rogoff]

Thanks so much.  It sort of works but with a bit of work-around:

1) tangle works fine

2) for export, I had to move the 
> ,#+MACRO: SEC_FOO 
lines out of the code block.  If I didn’t and tried to eval that block, I got 
the error

   No org-babel-execute function for org!

But I think this will do what I need. Of course emacs / org couldn’t have a 
simple preproccessor-like macro :)

  David 

Sent from my iPhone

> On Nov 1, 2017, at 9:17 AM, Kaushal Modi  wrote:
> 
>> On Wed, Nov 1, 2017 at 11:51 AM Kaushal Modi  wrote:
>> I still don't have a fully working solution (see below). Hopefully someone 
>> can help figure out the reason for "Invalid read syntax: #" error.
> 
> OK, I found a workaround. Below solution works:
> 
> 1. Eval the =org-macros= block once (=C-c C-c=)
>- Always use that =org-macros= block to define the Org macros for this 
> file.
>- Hitting =C-c C-c= will insert those macros in this file.
>- This block is reused as a noweb-ref in other blocks.. so keeping just 
> this
>  block as a source for all Org macros will be convenient.
> 2. Then export that file (=C-c C-e h o=)
> 
> =
> #+BEGIN_SRC org :noweb-ref org-macros :results output replace :exports none
> ,#+MACRO: SEC_FOO Foo Topic
> ,#+MACRO: SEC_BAR Bar Topic
> #+END_SRC
> 
> #+NAME: ehdr
> #+BEGIN_SRC emacs-lisp :var macro="n" :noweb yes :results raw :exports none
> (with-temp-buffer
>   (insert "
> <>
> \n")
>   (let ((start (point)))
> (insert "// ---\n")
> (insert (concat "// {{{" macro "}}}"))
> (org-mode)
> (org-macro-replace-all
>  (append org-macro-templates org-export-global-macros))
> (buffer-substring-no-properties start (point-max
> #+END_SRC
> 
> * Section 1: {{{SEC_FOO}}}
> #+BEGIN_SRC verilog :noweb yes
> <>
> parameter A = 1;
> #+END_SRC
> 
> * Section 2: {{{SEC_BAR}}}
> #+BEGIN_SRC verilog :noweb yes
> <>
> parameter A = 1;
> #+END_SRC 
> =
> -- 
> Kaushal Modi

[O] Issue with workaround (referring to sonarqube badges in github README.org)

[Steinar Bang]

I use org-mode syntax for my github README file, because I'm more
familiar with it, and because I use org-mode.

I ran into a problem when trying to use SonarQube status badges from
SonarCloud: the badge images didn't show up, instead the URL to the REST
API call generating the badge was shown[1].

According to a comment response on the issue[2], this is caused by
org-mode, because org-mode only creates image links for URLs containing
a file extension matching a supported image type.

Luckily the same message suggested a workaround: add the
pseudo-page-internal-link "#.svg" to the image URLs.

The results can be viewed on[3].

References:
[1] 
[2] 
[3] 

Re: [O] macro replacement in code block?

[Kaushal Modi]

On Wed, Nov 1, 2017 at 1:32 PM David Rogoff  wrote:

> Thanks so much.  It sort of works but with a bit of work-around:
>

Great!


> 2) for export, I had to move the
>
> ,#+MACRO: SEC_FOO
>
> lines out of the code block.  If I didn’t and tried to eval that block, I
> got the error
>
>No org-babel-execute function for org!
>

You just need to add org as an org babel language:

  (let (ob-lang-alist)
(add-to-list 'ob-lang-alist '(org . t))
(org-babel-do-load-languages 'org-babel-load-languages ob-lang-alist))

You don't necessarily need the let form above, but that is in a form that
allows easily adding more languages in future.
-- 

Kaushal Modi

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>
> Allen Li  writes:
>
>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>>
>>> My preferences would be
>>>
>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>> 2. If a timestamp does include the TZ, honour that TZ
>>
>> Org mode does not support TZ in time strings and adding support isn't
>> the topic of the current bug.
>>
>
> I disagree. The root cause of the bug is due to a lack of clarity
> regarding timezones. I also suspect this is also the basic cause of
> issues relating to the use of timestamps/time strings and daylight
> savings, especially when it comes to performing calculations etc.

Unless I missed something, Org mode does not support explicit
timezones in Org timestamps, and adding such support would take a
large effort [1].

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html

Unfortunately, I do not have time for the task.  Thus, Org mode
will have to make due with only Org timestamps without an
explicit timezone unless a savior appears.

>
>> I want to emphasize the distinction between timestamps and time
>> strings.  Timestamps are a
>> float value, Unix timestamps.  Time strings are Org mode specific,
>> like <2017-10-30 12:34:56>
>
> Org mode refers to what you are calling time strings as timestamps. In
> reality, there is no difference - one is just a numeric representation
> and the other is a string representation. It is good you have clarified
> your definition to reduce confusion. However, I think the problems are
> arising because of a lack of explicit TZ handling.

There's a pretty significant difference.  Unix timestamps are
standardized and used widely by most computers and Emacs itself.
Unix timestamps are time zone agnostics.

Org timestamps (time strings, whatever) are used exclusively by
Org mode.  As Org timestamps do not support explicit timezones,
they are not time zone agnostic.  The interpretation of an Org
timestamp depends on the time zone of the local machine, whereas
the interpretation of a Unix timestamp does not depend on the
time zone of the local machine.

>
>>
>> Timestamps do not have timezone information since they describe an
>> exact (well, minus leap seconds)
>> point in time, the number of seconds after the epoch.
>>
>
> A point in time is measured in number of seconds since epoch. However,
> if you want to use local (wall) time to display that point, you have to
> include TZ information when converting that number to a more
> meaningful/usable (for humans) format.

I'm not sure what you mean.  Time zone information is provided by
the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
and tell it to format it in human readable format in the local
time zone (the default behavior), then it will be done, without
my having to attach time zone information anywhere.

> The point 'now' for me is UTC+1100
> and for you (based on your previous post) UTC-0700, so our
> representations in string format of this value will be different. Even
> on a single machine, it is also relevant. For example, if I have two org
> timestamps (your times strings) and I want to calculate the number of
> hours between the two, I need to include TZ information as one timestamp
> might be during daylight savings time and the other outside daylight
> savings time.

Ideally, except Org mode does not support including explicit TZ
information in Org timestamps.  Thus, Org mode can only current
interpret Org timestamps according to some blessed time zone.
Four months ago, that blessed time zone changed from the local
machine's time zone to UTC.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

Apologies, my reply was partially omitted.  My full reply follows

On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>
> Allen Li  writes:
>
>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>>
>>> My preferences would be
>>>
>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>> 2. If a timestamp does include the TZ, honour that TZ
>>
>> Org mode does not support TZ in time strings and adding support isn't
>> the topic of the current bug.
>>
>
> I disagree. The root cause of the bug is due to a lack of clarity
> regarding timezones. I also suspect this is also the basic cause of
> issues relating to the use of timestamps/time strings and daylight
> savings, especially when it comes to performing calculations etc.

Unless I missed something, Org mode does not support explicit
timezones in Org timestamps, and adding such support would take a
large effort [1].

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html

Unfortunately, I do not have time for the task.  Thus, Org mode
will have to make due with only Org timestamps without an
explicit timezone unless a savior appears.

>
>> I want to emphasize the distinction between timestamps and time
>> strings.  Timestamps are a
>> float value, Unix timestamps.  Time strings are Org mode specific,
>> like <2017-10-30 12:34:56>
>
> Org mode refers to what you are calling time strings as timestamps. In
> reality, there is no difference - one is just a numeric representation
> and the other is a string representation. It is good you have clarified
> your definition to reduce confusion. However, I think the problems are
> arising because of a lack of explicit TZ handling.

There's a pretty significant difference.  Unix timestamps are
standardized and used widely by most computers and Emacs itself.
Unix timestamps are time zone agnostics.

Org timestamps (time strings, whatever) are used exclusively by
Org mode.  As Org timestamps do not support explicit timezones,
they are not time zone agnostic.  The interpretation of an Org
timestamp depends on the time zone of the local machine, whereas
the interpretation of a Unix timestamp does not depend on the
time zone of the local machine.

>
>>
>> Timestamps do not have timezone information since they describe an
>> exact (well, minus leap seconds)
>> point in time, the number of seconds after the epoch.
>>
>
> A point in time is measured in number of seconds since epoch. However,
> if you want to use local (wall) time to display that point, you have to
> include TZ information when converting that number to a more
> meaningful/usable (for humans) format.

I'm not sure what you mean.  Time zone information is provided by
the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
and tell it to format it in human readable format in the local
time zone (the default behavior), then it will be done, without
my having to attach time zone information anywhere.

> The point 'now' for me is UTC+1100
> and for you (based on your previous post) UTC-0700, so our
> representations in string format of this value will be different. Even
> on a single machine, it is also relevant. For example, if I have two org
> timestamps (your times strings) and I want to calculate the number of
> hours between the two, I need to include TZ information as one timestamp
> might be during daylight savings time and the other outside daylight
> savings time.

Ideally, except Org mode does not support including explicit TZ
information in Org timestamps.  Thus, Org mode can only current
interpret Org timestamps according to some blessed time zone.
Four months ago, that blessed time zone changed from the local
machine's time zone to UTC.

> Any calculation which fails to consider TZ information in
> this case will be out by an hour.
>
> If we just take the simple solution and ignore TZ information,

Org timestamps do not have TZ information.  It's not a matter of
not ignoring TZ information, support for TZ information would
have to be added, which as I understand is non-trivial.

> we will
> break calculations which cross daylight savings 'boarders'. I also
> suspect we will get inconsistent behaviour when integrating with other
> systems (such as external calendars, ical integration or sharing
> timestamp data with others etc).

As long as we use Unix timestamps when integrating with other
systems, there is no ambiguity.

>
>>> 3. If the timestamp does not include a time component, default to 0:00:0
>>
>> This is what Org mode does
>>
>>> for the local TZ
>>
>> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>>
>>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>>
>>> Rationale is that the user should have ability to fully control how
>>> their timestamps are represented. However, adding the TZ is probably
>>> just a pain for the majority of use cases, so defaulting to the local
>>> (wall) TZ is OK provided you can override this consistently by adding
>>> e

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nicolas Goaziou]

Hello,

Allen Li  writes:

> I guess the relevant bug is
> http://lists.gnu.org/archive/html/emacs-orgmode/2017-07/msg00097.html,

FWIW, this is not the initial bug.

> From what I can glean from the history,
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 is where things started to go
> wrong.  UTC timezone was added to tests.  From there, a number of
> commits were added to fix regressions.
>
> The problem is that the tests were made to assume UTC timezone and Org
> mode was changed to also use UTC timezone to make the tests pass;
> however, this means that org.el no longer works for non-UTC time
> zones.
>
> That commit's message (112c5ba479d52c3c36de5c7aafd14ab6bc075005) also
> seems suspect.  It claims to be removing DST offset by enforcing UTC.
> That could only be true for people using GMT.  In GMT during DST,
> enforcing UTC would "remove" the DST offset, but that is not true
> anywhere else.
>
> Given my chain of reasoning, the commits following
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 to fix "regressions" is
> really just converting all of Org mode's timestamps to use the
> "timezone adjusted" Unix timestamps introduced by
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005

IIRC, the point is to remove DST in durations, i.e., in the difference
between two dates. One way to do that is to assume UTC in both start end
end date. Most of the commits are about making sure UTC is used whenever
two Org dates are to be used in a duration computation, and nowhere
else.

I think the change to org-2ft was a side-effect, since it is indirectly
used is a duration.

Regards,

-- 
Nicolas Goaziou0x80A93738

Re: [O] ``make test'' failure

[Nicolas Goaziou]

Hello,

Nick Dokos  writes:

> I get one failure in test-org-publish/resolve-external-link.
> Bisecting fingers this commit:
>
> ,
> | $ git bisect good
> | 007bbddbccfd06ab9c97d51bf833a068cb1b07a2 is the first bad commit
> | commit 007bbddbccfd06ab9c97d51bf833a068cb1b07a2
> | Author: Nicolas Goaziou 
> | Date:   Sun Sep 10 00:16:12 2017 +0200
> | 
> | ox-html: Prevent spurious target below headlines
> | 
> | * lisp/ox-html.el (org-html-headline):
> | (org-html-link): Do not insert an additional target.
> | 
> | * lisp/ox-publish.el (org-publish-resolve-external-link): Add an
> |   optional argument.
> | 
> | * lisp/ox.el (org-export-get-reference): Improve docstring.
> | 
> | * testing/examples/pub/a.org:
> | * testing/examples/pub/b.org: New files.
> | * testing/lisp/test-ox-publish.el 
> (test-org-publish/resolve-external-link):
> |   New test.
> | 
> | :04 04 fda080e0db653c9049448200be5ad03369379cc3 
> ec0eb8a5cd9b3c3b204fd7134e515fb360b85c79 M  etc
> | :04 04 39afe6514335e2ccd549fc5b7fafcac344cbb471 
> a0fdf6967cc3ab95b046f81dab08095fe0d302de M  lisp
> | :04 04 120e098640643a963336d77d65929ef5863eeba5 
> 679329e78a4bbc04f6f84bb176821a3084732ab3 M  testing
> `
>
> which *introduced* this test, but of course the test may be at fault,
> rather than the code.

There is a non-deterministic part in that failure since neither our
Build nor my machine can reproduce it.

Could you try the following patch and tell me if the test succeed?

Thank you.

Regards,

-- 
Nicolas Goaziou
>From dc8d8d1b9bf92d9fe826cfe8a13c6ad89c1f9c8e Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou 
Date: Wed, 1 Nov 2017 17:54:12 +0100
Subject: [PATCH] Fix failing test

* testing/lisp/test-ox-publish.el (test-org-publish/resolve-external-link):
  Rewrite test to prevent race condition which could affect results.
* testing/examples/pub/b.org: Add missing keyword.
---
 testing/examples/pub/b.org  |  3 +++
 testing/lisp/test-ox-publish.el | 34 ++
 2 files changed, 21 insertions(+), 16 deletions(-)

diff --git a/testing/examples/pub/b.org b/testing/examples/pub/b.org
index 60387460e..261152e99 100644
--- a/testing/examples/pub/b.org
+++ b/testing/examples/pub/b.org
@@ -2,5 +2,8 @@
 #+date: <2012-03-29 Thu>
 
 * Headline1
+:PROPERTIES:
+:CUSTOM_ID: b1
+:END:
 
 [[file:a.org::#a1]]
diff --git a/testing/lisp/test-ox-publish.el b/testing/lisp/test-ox-publish.el
index a51190578..0c3aaedca 100644
--- a/testing/lisp/test-ox-publish.el
+++ b/testing/lisp/test-ox-publish.el
@@ -366,27 +366,29 @@ Unless set otherwise in PROPERTIES, `:base-directory' is set to
   ;; instead of internal reference, whenever possible.
   (should
(equal
-"a1"
+'("a1" "b1")
 (let* ((ids nil)
+	   (link-transcoder
+	(lambda (l c i)
+	  (let ((option (org-element-property :search-option l))
+		(path (org-element-property :path l)))
+		(push (org-publish-resolve-external-link option path t)
+		  ids)
+		"")))
 	   (backend
 	(org-export-create-backend
-	 :transcoders
-	 '((headline . (lambda (h c i) c))
-	   (paragraph . (lambda (p c i) c))
-	   (section . (lambda (s c i) c))
-	   (link . (lambda (l c i)
-			 (let ((option (org-element-property :search-option l))
-			   (path (org-element-property :path l)))
-			   (when option
-			 (throw :exit (org-publish-resolve-external-link
-	   option path t)
+	 :transcoders `((headline . (lambda (h c i) c))
+			(paragraph . (lambda (p c i) c))
+			(section . (lambda (s c i) c))
+			(link . ,link-transcoder
 	   (publish
 	(lambda (plist filename pub-dir)
-	  (push (catch :exit
-		  (org-publish-org-to backend filename ".test" plist pub-dir))
-		ids
-  (org-test-publish (list :publishing-function (list publish)) #'ignore)
-  (car ids)
+	  (org-publish-org-to backend filename ".test" plist pub-dir
+  (org-test-publish (list :publishing-function (list publish)
+			  :exclude "."
+			  :include '("a.org" "b.org"))
+	#'ignore)
+  (sort ids #'string<)
 
 
 ;;; Tools
-- 
2.14.3

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 1:55 PM, Nicolas Goaziou  wrote:
>
> IIRC, the point is to remove DST in durations, i.e., in the difference
> between two dates. One way to do that is to assume UTC in both start end
> end date. Most of the commits are about making sure UTC is used whenever
> two Org dates are to be used in a duration computation, and nowhere
> else.

Alas, I still can't seem to find the original DST bug.  I'm not sure
using UTC solves DST problems.

For example, in the timezone America/Los_Angeles,

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour

This is what Emacs gives me using the default time zone

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour

This is what Emacs gives me using UTC

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 3 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 3 hours

Using UTC seems strictly wrong to me.

>
> I think the change to org-2ft was a side-effect, since it is indirectly
> used is a duration.
>
> Regards,
>
> --
> Nicolas Goaziou0x80A93738

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Let me try to clarify and see if that helps.

Org is only forcing time into UTC format for functions which deal with
time comparisons. i.e. those which use org-2ft. The reason for doing
this is to try and avoid issues relating to daylight savings timezone
changes when performing comparison operations.

It is not clear to me exactly how this is actually creating a problem
for your scenario as there does not seem to be anywhere in org where
values created by org-2ft are converted back to human readable time
strings. If you can provide more details on how this is a practical
issue in org-mode that might help.

I do think there was an error in your example analysis.

In your examples of

(current-time-string (float-time))
"Mon Oct 30 21:21:31 2017" ; right

(current-time-string (org-time-today))
"Mon Oct 30 00:00:00 2017" ; right

(current-time-string (org-2ft "<2017-10-31>"))
"Mon Oct 30 17:00:00 2017" ; wrong

the last one is only wrong because you have failed to tell
current-time-string that the value you are converting is a UTC time. If
you change the code to

(current-time-string (org-2ft "<2017-10-31>") "UTC")
"Mon Oct 30 00:00:00 2017" ; correct

Now, considering your suggestion to change org-2ft to not use UTC.

If we change org-2ft to not use UTC time, we end up with time
related calculations which cross daylight savings boarders becoming
incorrect. So, if you have a clock table where you want to calculate the
number of hours between two timestamps and one is within daylgiht
savings time and the other is outside it, your calculation will be out
by an hour.

As org-2ft is not used in conversion of 'epoch' time to times strings by
org, I'm not sure reverting the change gains us anything and will likely
adversely impact things like clock table calculations. What may be
justified is to add a note to the manual which clearly states that if
you are using org-2ft in your own code and plan to convert the result
back to a timestamp string, you need to ensure you explicitly tell the
function performing the conversion the input value is in UTC and not
local time.

I agree that enhancing org timestamps to include TZ information would be
a non-trivial chunk of work, I do feel it is the only solid way to
address all of these issues. I suspect there are still edge cases in
time related computations which fail even with the UTC approach and
there are certainly extensions which could benefit from explicit TZ
information (for example, taskgjuggler, which does time calculations for
planning and where such calculations can easily cross multiple daylight
savings TZ changes or include operations covering different timezones
etc).  

Tim

Allen Li  writes:

> Apologies, my reply was partially omitted.  My full reply follows
>
> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>>
>> Allen Li  writes:
>>
>>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:

 My preferences would be

 1. If a timestamp does not include the TZ, then assume the local TZ
 2. If a timestamp does include the TZ, honour that TZ
>>>
>>> Org mode does not support TZ in time strings and adding support isn't
>>> the topic of the current bug.
>>>
>>
>> I disagree. The root cause of the bug is due to a lack of clarity
>> regarding timezones. I also suspect this is also the basic cause of
>> issues relating to the use of timestamps/time strings and daylight
>> savings, especially when it comes to performing calculations etc.
>
> Unless I missed something, Org mode does not support explicit
> timezones in Org timestamps, and adding such support would take a
> large effort [1].
>
> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html
>
> Unfortunately, I do not have time for the task.  Thus, Org mode
> will have to make due with only Org timestamps without an
> explicit timezone unless a savior appears.
>
>>
>>> I want to emphasize the distinction between timestamps and time
>>> strings.  Timestamps are a
>>> float value, Unix timestamps.  Time strings are Org mode specific,
>>> like <2017-10-30 12:34:56>
>>
>> Org mode refers to what you are calling time strings as timestamps. In
>> reality, there is no difference - one is just a numeric representation
>> and the other is a string representation. It is good you have clarified
>> your definition to reduce confusion. However, I think the problems are
>> arising because of a lack of explicit TZ handling.
>
> There's a pretty significant difference.  Unix timestamps are
> standardized and used widely by most computers and Emacs itself.
> Unix timestamps are time zone agnostics.
>
> Org timestamps (time strings, whatever) are used exclusively by
> Org mode.  As Org timestamps do not support explicit timezones,
> they are not time zone agnostic.  The interpretation of an Org
> timestamp depends on the time zone of the local machine, whereas
> the interpretation of a Unix timestamp does not depend on the
> time zone of the local machine.
>
>>
>>>
>>> Timestamps do not

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross  wrote:
>
> Let me try to clarify and see if that helps.
>
> Org is only forcing time into UTC format for functions which deal with
> time comparisons. i.e. those which use org-2ft. The reason for doing
> this is to try and avoid issues relating to daylight savings timezone
> changes when performing comparison operations.
>
> It is not clear to me exactly how this is actually creating a problem
> for your scenario as there does not seem to be anywhere in org where
> values created by org-2ft are converted back to human readable time
> strings. If you can provide more details on how this is a practical
> issue in org-mode that might help.

I think I made the problem quite clear multiple times.

Assuming today is 2017-10-31, SCHEDULED<"" will match results
that have SCHEDULED=<2017-11-01> depending on your local time zone.

>
> I do think there was an error in your example analysis.
>
> In your examples of
>
> (current-time-string (float-time))
> "Mon Oct 30 21:21:31 2017" ; right
>
> (current-time-string (org-time-today))
> "Mon Oct 30 00:00:00 2017" ; right
>
> (current-time-string (org-2ft "<2017-10-31>"))
> "Mon Oct 30 17:00:00 2017" ; wrong
>
> the last one is only wrong because you have failed to tell
> current-time-string that the value you are converting is a UTC time. If
> you change the code to
>
> (current-time-string (org-2ft "<2017-10-31>") "UTC")
> "Mon Oct 30 00:00:00 2017" ; correct

No, that is wrong. Mon Oct 30 00:00:00 2017 UTC != Mon Oct 30 00:00:00
2017 UTC-7

(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>"))
"2017-10-30T17:00:00+0700" ; wrong
(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>") "UTC")
"2017-10-31T00:00:00+" ; wrong

In fact, you haven't solved a thing, since (current-time-string
(org-2ft "<2017-10-31>") "UTC") == (current-time-string (org-2ft
"<2017-10-31>"))  The time is still the same, and still wrong, just
formatted differently if you silently drop the TZ info from the time
string.

>
> Now, considering your suggestion to change org-2ft to not use UTC.
>
> If we change org-2ft to not use UTC time, we end up with time
> related calculations which cross daylight savings boarders becoming
> incorrect. So, if you have a clock table where you want to calculate the
> number of hours between two timestamps and one is within daylgiht
> savings time and the other is outside it, your calculation will be out
> by an hour.

Or rather, we end up with time related calculations which cross
daylight savings becoming correct.

But I am beginning to see the picture; someone wanted time related
calculations crossing DST to be incorrect to appear "right" to time
zone naive applications.

>
> As org-2ft is not used in conversion of 'epoch' time to times strings by
> org, I'm not sure reverting the change gains us anything and will likely
> adversely impact things like clock table calculations. What may be
> justified is to add a note to the manual which clearly states that if
> you are using org-2ft in your own code and plan to convert the result
> back to a timestamp string, you need to ensure you explicitly tell the
> function performing the conversion the input value is in UTC and not
> local time.

Yes, org-2ft documentation should be updated at the very least,
assuming that this behavior is desired.

>
> I agree that enhancing org timestamps to include TZ information would be
> a non-trivial chunk of work, I do feel it is the only solid way to
> address all of these issues. I suspect there are still edge cases in
> time related computations which fail even with the UTC approach and
> there are certainly extensions which could benefit from explicit TZ
> information (for example, taskgjuggler, which does time calculations for
> planning and where such calculations can easily cross multiple daylight
> savings TZ changes or include operations covering different timezones
> etc).
>
> Tim
>
> Allen Li  writes:
>
>> Apologies, my reply was partially omitted.  My full reply follows
>>
>> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>>>
>>> Allen Li  writes:
>>>
 On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>
> My preferences would be
>
> 1. If a timestamp does not include the TZ, then assume the local TZ
> 2. If a timestamp does include the TZ, honour that TZ

 Org mode does not support TZ in time strings and adding support isn't
 the topic of the current bug.

>>>
>>> I disagree. The root cause of the bug is due to a lack of clarity
>>> regarding timezones. I also suspect this is also the basic cause of
>>> issues relating to the use of timestamps/time strings and daylight
>>> savings, especially when it comes to performing calculations etc.
>>
>> Unless I missed something, Org mode does not support explicit
>> timezones in Org timestamps, and adding such support would take a
>> large effort [1].
>>
>> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html
>>
>> Unf

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Allen Li  writes:

> On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross  wrote:
>>
>> Let me try to clarify and see if that helps.
>>
>> Org is only forcing time into UTC format for functions which deal with
>> time comparisons. i.e. those which use org-2ft. The reason for doing
>> this is to try and avoid issues relating to daylight savings timezone
>> changes when performing comparison operations.
>>
>> It is not clear to me exactly how this is actually creating a problem
>> for your scenario as there does not seem to be anywhere in org where
>> values created by org-2ft are converted back to human readable time
>> strings. If you can provide more details on how this is a practical
>> issue in org-mode that might help.
>
> I think I made the problem quite clear multiple times.
>
> Assuming today is 2017-10-31, SCHEDULED<"" will match results
> that have SCHEDULED=<2017-11-01> depending on your local time zone.
>

I get that, what I am not clear about is where you encounter this issue
when using org .eg. is it when entering now as a date for generating an
agenda of past or current scheduled items etc or is this a problem with
some custom functions you have written where you need to enter 'now'
manually and due to the use of UTC in time comparisons are forced to
adjust manually for UTC or something else? This is what is not
clear. From my own use, I don't see the issue you describe with
scheduled items not appearing or appearing when they shouldn't, so don't
know how to reproduce the issue your having i.e. when I do agenda
searches, I don't see deadline or scheduled items showing up when they
should not etc.

This matters because if the issue is the former, then we likely do have
a real bug. However, if it is the latter, then it may be a bug or it may
be just an implementation weakness or it could just be a difference of
opinions on how it should be.

Your examples don't clarify what the issue is. It is obvious that if we
convert times to UTC for calculation purposes that this will affect the
display of those timestamps, but as far as I can tell, org doesn't
convert back and use those converted values in any visible way. Provided
all values are converted consistently, it should not matter. Therefore,
if there is a problem, then something must not be getting converted
correctly, but it is not possible to narrow down where this is without
more info.

If you can provide an ECM then we can avoid assumptions and different
interpretations and focus on the issue.

Tim

--
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 8:27 PM, Tim Cross  wrote:
>
> I get that, what I am not clear about is where you encounter this issue
> when using org .eg. is it when entering now as a date for generating an
> agenda of past or current scheduled items etc or is this a problem with
> some custom functions you have written where you need to enter 'now'
> manually and due to the use of UTC in time comparisons are forced to
> adjust manually for UTC or something else? This is what is not
> clear. From my own use, I don't see the issue you describe with
> scheduled items not appearing or appearing when they shouldn't, so don't
> know how to reproduce the issue your having i.e. when I do agenda
> searches, I don't see deadline or scheduled items showing up when they
> should not etc.

It is when doing any kind of tag search.  So org-agenda, mostly.

I'm not surprised you cannot reproduce it because it is dependent on
your local timezone.

>
> This matters because if the issue is the former, then we likely do have
> a real bug. However, if it is the latter, then it may be a bug or it may
> be just an implementation weakness or it could just be a difference of
> opinions on how it should be.
>
> Your examples don't clarify what the issue is. It is obvious that if we
> convert times to UTC for calculation purposes that this will affect the
> display of those timestamps, but as far as I can tell, org doesn't
> convert back and use those converted values in any visible way. Provided
> all values are converted consistently, it should not matter. Therefore,
> if there is a problem, then something must not be getting converted
> correctly, but it is not possible to narrow down where this is without
> more info.

Ah, perhaps I should make the issue clearer.

When I run this at 2017-11-01 20:55:30

(org-2ft "")
1509594900.0

(org-2ft "<2017-11-01 20:55:30>")
1509569700.0

Therefore when org.el does a tag search for SCHEDULED="", it will
not match <2017-11-01 20:55:30>, assuming that I have frozen my system
clock to that time.

>
> If you can provide an ECM then we can avoid assumptions and different
> interpretations and focus on the issue.

Again, this is dependent on your local time zone.  Try setting your
time zone to America/Los_Angeles.

1. emacs -Q
2. Assume it is 2017-10-10 12:00:00 local time
3. Create an agenda file

* TODO thing
SCHEDULED:<2017-10-10 13:00:00>

4. M-x org-agenda RET m SCHEDULED>""

Expected:

thing shows up

Actual:

thing does not show up

>
> Tim
>
> --
> Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

I found another one (It is 2017-11-01 in local time for me)

(org-time= "<2017-11-01>" "")
nil

This is also local timezone dependent.

Tim made the point that these floats are not intended to be Unix
timestamps and only used for comparison, but this implementation
detail leaks quite heavily.  Furthermore, the value returned by
org-2ft for these comparisons isn't even constant.

For example (org-2ft "<2017-10-10 10:00:00>") returns 1507629600.0 for
me today, but in four days it will return a different value,
1507626000.0.  Surprise, the offset from local time to UTC is not
constant, even on the same machine in the same timezone.

The most immediate impact of this is that all of Org mode's time
comparison functions will be wrong when called during a local time
offset change.  For those of us who are lucky, this may be twice a
year, but for others who are unlucky, it may be multiple times per
year or month, up to the whims of the local government.

This is one of the reasons I think "we're just using them for
comparisons" doesn't excuse org-2ft doing its little time shift.  The
other reason is all of the regressions that change caused, including
this current bug, and yet another is my gut instinct that this is a
poor design choice from a technical perspective as a software
engineer.

[O] Bug: org-parse-time-string et al hit recursion limit [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

Running (org-parse-time-string "<@@@>") results in hitting max-lisp-eval-depth

org-parse-time-string calls org-matcher-time calls org-2ft calls
org-parse-time-string

There is a condition-case in org-2ft that catches the error and
returns 0, but it would be nice if this wasn't implemented via hitting
the recursion limit.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

Sorry for the spam, but I am digging to see how deep the rabbit hole goes.

All five of the first branches in org-matcher-time are wrong (again,
local timezone dependent):

(org-time= "<2017-11-01>" "")
nil
(org-time= "<2017-10-31>" "")
nil
(org-time= "<2017-11-02>" "")
nil
(org-time= "<2017-11-02>" "<+1d>")
nil

For my immediate bug, all that is needed is a small fix to
org-time-today and modifying the call to float-time in
org-matcher-time.

However, there are a lot of float-time calls in Org mode, and I don't
know how many of them need to be modified to return org-2ft time
shifted timestamps instead of UTC Unix timestamps.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

OK, thanks for the additional information. That helps a lot.

We can now narrow down where the issue is.

After having looked (only very casually) at some of the org date/time
related functions, I think there may be quite a few issues. I'm not
convinced the root cause is org-2ft converting to UTC, though I think
that change has probably revealed some of the issues.

There does seem to be some inconsistency or lack of clarity regarding
some of these operations and it could well be that the switch to using
UTC is not the correct approach, but just switching back probably isn't
either What does appear to be required is increased consistency in some
of these functions or where/how they are used. .

To correctly fix this, I feel more analysis is warranted. I'm prepared
to look at this and present a summary and options, but it will have to
wait until after 21st when I start leave from work. It is a complex area
and perhaps my skills won't be up to it, but I should at least be able
to identify the main areas needing attention/decisions. 

Whether we should change back to non-UTC for time comparisons in the
meantime is not something I feel informed enough to make a call. It
really depends on the daylight savings time related issues affecting
clock tables and other functionality.  This is probably something core
org maintainers will need to make a call on.

Tim


Allen Li  writes:

> On Wed, Nov 1, 2017 at 8:27 PM, Tim Cross  wrote:
>>
>> I get that, what I am not clear about is where you encounter this issue
>> when using org .eg. is it when entering now as a date for generating an
>> agenda of past or current scheduled items etc or is this a problem with
>> some custom functions you have written where you need to enter 'now'
>> manually and due to the use of UTC in time comparisons are forced to
>> adjust manually for UTC or something else? This is what is not
>> clear. From my own use, I don't see the issue you describe with
>> scheduled items not appearing or appearing when they shouldn't, so don't
>> know how to reproduce the issue your having i.e. when I do agenda
>> searches, I don't see deadline or scheduled items showing up when they
>> should not etc.
>
> It is when doing any kind of tag search.  So org-agenda, mostly.
>
> I'm not surprised you cannot reproduce it because it is dependent on
> your local timezone.
>
>>
>> This matters because if the issue is the former, then we likely do have
>> a real bug. However, if it is the latter, then it may be a bug or it may
>> be just an implementation weakness or it could just be a difference of
>> opinions on how it should be.
>>
>> Your examples don't clarify what the issue is. It is obvious that if we
>> convert times to UTC for calculation purposes that this will affect the
>> display of those timestamps, but as far as I can tell, org doesn't
>> convert back and use those converted values in any visible way. Provided
>> all values are converted consistently, it should not matter. Therefore,
>> if there is a problem, then something must not be getting converted
>> correctly, but it is not possible to narrow down where this is without
>> more info.
>
> Ah, perhaps I should make the issue clearer.
>
> When I run this at 2017-11-01 20:55:30
>
> (org-2ft "")
> 1509594900.0
>
> (org-2ft "<2017-11-01 20:55:30>")
> 1509569700.0
>
> Therefore when org.el does a tag search for SCHEDULED="", it will
> not match <2017-11-01 20:55:30>, assuming that I have frozen my system
> clock to that time.
>
>>
>> If you can provide an ECM then we can avoid assumptions and different
>> interpretations and focus on the issue.
>
> Again, this is dependent on your local time zone.  Try setting your
> time zone to America/Los_Angeles.
>
> 1. emacs -Q
> 2. Assume it is 2017-10-10 12:00:00 local time
> 3. Create an agenda file
>
> * TODO thing
> SCHEDULED:<2017-10-10 13:00:00>
>
> 4. M-x org-agenda RET m SCHEDULED>""
>
> Expected:
>
> thing shows up
>
> Actual:
>
> thing does not show up
>
>>
>> Tim
>>
>> --
>> Tim Cross


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 9:56 PM, Tim Cross  wrote:
>
> OK, thanks for the additional information. That helps a lot.
>
> We can now narrow down where the issue is.
>
> After having looked (only very casually) at some of the org date/time
> related functions, I think there may be quite a few issues. I'm not
> convinced the root cause is org-2ft converting to UTC, though I think
> that change has probably revealed some of the issues.
>
> There does seem to be some inconsistency or lack of clarity regarding
> some of these operations and it could well be that the switch to using
> UTC is not the correct approach, but just switching back probably isn't
> either What does appear to be required is increased consistency in some
> of these functions or where/how they are used. .

Thanks.  I think I have been harping a bit too hard on org-2ft, but
indeed it seems to be a bit more systemic an issue.

> To correctly fix this, I feel more analysis is warranted. I'm prepared
> to look at this and present a summary and options, but it will have to
> wait until after 21st when I start leave from work. It is a complex area
> and perhaps my skills won't be up to it, but I should at least be able
> to identify the main areas needing attention/decisions.

My initial approach would be to do some refactoring here, especially
among org-2ft, org-matcher-time, and org-parse-time-string, each of
which calls the others in a cycle and each share a part of the logic
for interpreting Org mode timestamps.

I'm not familiar with refactoring FOSS code via mailed patches, nor if
Org maintainers would welcome such patches, but I would be willing to
do some refactoring here.

>
> Whether we should change back to non-UTC for time comparisons in the
> meantime is not something I feel informed enough to make a call. It
> really depends on the daylight savings time related issues affecting
> clock tables and other functionality.  This is probably something core
> org maintainers will need to make a call on.
>
> Tim