Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Allen Li  writes:

> On Wed, Nov 1, 2017 at 9:56 PM, Tim Cross  wrote:
>>
>
>> To correctly fix this, I feel more analysis is warranted. I'm prepared
>> to look at this and present a summary and options, but it will have to
>> wait until after 21st when I start leave from work. It is a complex area
>> and perhaps my skills won't be up to it, but I should at least be able
>> to identify the main areas needing attention/decisions.
>
> My initial approach would be to do some refactoring here, especially
> among org-2ft, org-matcher-time, and org-parse-time-string, each of
> which calls the others in a cycle and each share a part of the logic
> for interpreting Org mode timestamps.
>
> I'm not familiar with refactoring FOSS code via mailed patches, nor if
> Org maintainers would welcome such patches, but I would be willing to
> do some refactoring here.
>

I think what I will do is start with adding/extending the tests relating
to timestamps and clock tables. This should

- help ensure I understand the required functionality
- may help identify existing bugs
- help ensure any refactoring does not have undesired side effects or
  loss of functionality

I also suspect it will be a good way for core org maintainers to verify
I'm on the right track and haven't missed anything before making any
changes to the code base. I'm not a big TDD advocate, but when working
with an unfamiliar code base, I've found developing tests first is a
good approach to ensure you really do understand existing and required
functionality.

regards,

Tim

-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nick Dokos]

Allen Li  writes:

> ...
> I'm not familiar with refactoring FOSS code via mailed patches, nor if
> Org maintainers would welcome such patches, but I would be willing to
> do some refactoring here.
>

See http://orgmode.org/worg/org-contribute.html

-- 
Nick

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

In a couple of weeks, I will have some spare time and I am happy to look
at the time related functions in org-mode to see if we can refactor them
to make them a bit clearer and possibly more efficient. I would be
hoping for input from the list as guidance experience has taught me that
date/time stuff can often have some subtle corner cases which are
easily missed.

regards,

Tim

Nicolas Goaziou  writes:

> Hello,
>
> Allen Li  writes:
>
>> Alas, I still can't seem to find the original DST bug.  I'm not sure
>> using UTC solves DST problems.
>>
>> For example, in the timezone America/Los_Angeles,
>>
>> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
>> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
>> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour
>>
>> This is what Emacs gives me using the default time zone
>>
>> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
>> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
>> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour
>>
>> This is what Emacs gives me using UTC
>>
>> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 3 hours
>> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
>> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 3 hours
>>
>> Using UTC seems strictly wrong to me.
>
> You're right. Using UTC doesn't solve any DST bug, despite what
> I initially thought. I think we just need to remove the whole set of
> changes about UTC in `parse-time-string'. We also need to adapt tests in
> test-org-clock since the same time difference could have different
> meanings depending on the time zone. I can do that later, if no one
> objects. WDYT?n
>
> Refactoring time functions in Org is still useful, though.
>
> Regards,


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nicolas Goaziou]

Hello,

Allen Li  writes:

> Alas, I still can't seem to find the original DST bug.  I'm not sure
> using UTC solves DST problems.
>
> For example, in the timezone America/Los_Angeles,
>
> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour
>
> This is what Emacs gives me using the default time zone
>
> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour
>
> This is what Emacs gives me using UTC
>
> <2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 3 hours
> <2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
> <2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 3 hours
>
> Using UTC seems strictly wrong to me.

You're right. Using UTC doesn't solve any DST bug, despite what
I initially thought. I think we just need to remove the whole set of
changes about UTC in `parse-time-string'. We also need to adapt tests in
test-org-clock since the same time difference could have different
meanings depending on the time zone. I can do that later, if no one
objects. WDYT?n

Refactoring time functions in Org is still useful, though.

Regards,

-- 
Nicolas Goaziou

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 9:56 PM, Tim Cross  wrote:
>
> OK, thanks for the additional information. That helps a lot.
>
> We can now narrow down where the issue is.
>
> After having looked (only very casually) at some of the org date/time
> related functions, I think there may be quite a few issues. I'm not
> convinced the root cause is org-2ft converting to UTC, though I think
> that change has probably revealed some of the issues.
>
> There does seem to be some inconsistency or lack of clarity regarding
> some of these operations and it could well be that the switch to using
> UTC is not the correct approach, but just switching back probably isn't
> either What does appear to be required is increased consistency in some
> of these functions or where/how they are used. .

Thanks.  I think I have been harping a bit too hard on org-2ft, but
indeed it seems to be a bit more systemic an issue.

> To correctly fix this, I feel more analysis is warranted. I'm prepared
> to look at this and present a summary and options, but it will have to
> wait until after 21st when I start leave from work. It is a complex area
> and perhaps my skills won't be up to it, but I should at least be able
> to identify the main areas needing attention/decisions.

My initial approach would be to do some refactoring here, especially
among org-2ft, org-matcher-time, and org-parse-time-string, each of
which calls the others in a cycle and each share a part of the logic
for interpreting Org mode timestamps.

I'm not familiar with refactoring FOSS code via mailed patches, nor if
Org maintainers would welcome such patches, but I would be willing to
do some refactoring here.

>
> Whether we should change back to non-UTC for time comparisons in the
> meantime is not something I feel informed enough to make a call. It
> really depends on the daylight savings time related issues affecting
> clock tables and other functionality.  This is probably something core
> org maintainers will need to make a call on.
>
> Tim

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

OK, thanks for the additional information. That helps a lot.

We can now narrow down where the issue is.

After having looked (only very casually) at some of the org date/time
related functions, I think there may be quite a few issues. I'm not
convinced the root cause is org-2ft converting to UTC, though I think
that change has probably revealed some of the issues.

There does seem to be some inconsistency or lack of clarity regarding
some of these operations and it could well be that the switch to using
UTC is not the correct approach, but just switching back probably isn't
either What does appear to be required is increased consistency in some
of these functions or where/how they are used. .

To correctly fix this, I feel more analysis is warranted. I'm prepared
to look at this and present a summary and options, but it will have to
wait until after 21st when I start leave from work. It is a complex area
and perhaps my skills won't be up to it, but I should at least be able
to identify the main areas needing attention/decisions. 

Whether we should change back to non-UTC for time comparisons in the
meantime is not something I feel informed enough to make a call. It
really depends on the daylight savings time related issues affecting
clock tables and other functionality.  This is probably something core
org maintainers will need to make a call on.

Tim


Allen Li  writes:

> On Wed, Nov 1, 2017 at 8:27 PM, Tim Cross  wrote:
>>
>> I get that, what I am not clear about is where you encounter this issue
>> when using org .eg. is it when entering now as a date for generating an
>> agenda of past or current scheduled items etc or is this a problem with
>> some custom functions you have written where you need to enter 'now'
>> manually and due to the use of UTC in time comparisons are forced to
>> adjust manually for UTC or something else? This is what is not
>> clear. From my own use, I don't see the issue you describe with
>> scheduled items not appearing or appearing when they shouldn't, so don't
>> know how to reproduce the issue your having i.e. when I do agenda
>> searches, I don't see deadline or scheduled items showing up when they
>> should not etc.
>
> It is when doing any kind of tag search.  So org-agenda, mostly.
>
> I'm not surprised you cannot reproduce it because it is dependent on
> your local timezone.
>
>>
>> This matters because if the issue is the former, then we likely do have
>> a real bug. However, if it is the latter, then it may be a bug or it may
>> be just an implementation weakness or it could just be a difference of
>> opinions on how it should be.
>>
>> Your examples don't clarify what the issue is. It is obvious that if we
>> convert times to UTC for calculation purposes that this will affect the
>> display of those timestamps, but as far as I can tell, org doesn't
>> convert back and use those converted values in any visible way. Provided
>> all values are converted consistently, it should not matter. Therefore,
>> if there is a problem, then something must not be getting converted
>> correctly, but it is not possible to narrow down where this is without
>> more info.
>
> Ah, perhaps I should make the issue clearer.
>
> When I run this at 2017-11-01 20:55:30
>
> (org-2ft "")
> 1509594900.0
>
> (org-2ft "<2017-11-01 20:55:30>")
> 1509569700.0
>
> Therefore when org.el does a tag search for SCHEDULED="", it will
> not match <2017-11-01 20:55:30>, assuming that I have frozen my system
> clock to that time.
>
>>
>> If you can provide an ECM then we can avoid assumptions and different
>> interpretations and focus on the issue.
>
> Again, this is dependent on your local time zone.  Try setting your
> time zone to America/Los_Angeles.
>
> 1. emacs -Q
> 2. Assume it is 2017-10-10 12:00:00 local time
> 3. Create an agenda file
>
> * TODO thing
> SCHEDULED:<2017-10-10 13:00:00>
>
> 4. M-x org-agenda RET m SCHEDULED>""
>
> Expected:
>
> thing shows up
>
> Actual:
>
> thing does not show up
>
>>
>> Tim
>>
>> --
>> Tim Cross


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

Sorry for the spam, but I am digging to see how deep the rabbit hole goes.

All five of the first branches in org-matcher-time are wrong (again,
local timezone dependent):

(org-time= "<2017-11-01>" "")
nil
(org-time= "<2017-10-31>" "")
nil
(org-time= "<2017-11-02>" "")
nil
(org-time= "<2017-11-02>" "<+1d>")
nil

For my immediate bug, all that is needed is a small fix to
org-time-today and modifying the call to float-time in
org-matcher-time.

However, there are a lot of float-time calls in Org mode, and I don't
know how many of them need to be modified to return org-2ft time
shifted timestamps instead of UTC Unix timestamps.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

I found another one (It is 2017-11-01 in local time for me)

(org-time= "<2017-11-01>" "")
nil

This is also local timezone dependent.

Tim made the point that these floats are not intended to be Unix
timestamps and only used for comparison, but this implementation
detail leaks quite heavily.  Furthermore, the value returned by
org-2ft for these comparisons isn't even constant.

For example (org-2ft "<2017-10-10 10:00:00>") returns 1507629600.0 for
me today, but in four days it will return a different value,
1507626000.0.  Surprise, the offset from local time to UTC is not
constant, even on the same machine in the same timezone.

The most immediate impact of this is that all of Org mode's time
comparison functions will be wrong when called during a local time
offset change.  For those of us who are lucky, this may be twice a
year, but for others who are unlucky, it may be multiple times per
year or month, up to the whims of the local government.

This is one of the reasons I think "we're just using them for
comparisons" doesn't excuse org-2ft doing its little time shift.  The
other reason is all of the regressions that change caused, including
this current bug, and yet another is my gut instinct that this is a
poor design choice from a technical perspective as a software
engineer.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 8:27 PM, Tim Cross  wrote:
>
> I get that, what I am not clear about is where you encounter this issue
> when using org .eg. is it when entering now as a date for generating an
> agenda of past or current scheduled items etc or is this a problem with
> some custom functions you have written where you need to enter 'now'
> manually and due to the use of UTC in time comparisons are forced to
> adjust manually for UTC or something else? This is what is not
> clear. From my own use, I don't see the issue you describe with
> scheduled items not appearing or appearing when they shouldn't, so don't
> know how to reproduce the issue your having i.e. when I do agenda
> searches, I don't see deadline or scheduled items showing up when they
> should not etc.

It is when doing any kind of tag search.  So org-agenda, mostly.

I'm not surprised you cannot reproduce it because it is dependent on
your local timezone.

>
> This matters because if the issue is the former, then we likely do have
> a real bug. However, if it is the latter, then it may be a bug or it may
> be just an implementation weakness or it could just be a difference of
> opinions on how it should be.
>
> Your examples don't clarify what the issue is. It is obvious that if we
> convert times to UTC for calculation purposes that this will affect the
> display of those timestamps, but as far as I can tell, org doesn't
> convert back and use those converted values in any visible way. Provided
> all values are converted consistently, it should not matter. Therefore,
> if there is a problem, then something must not be getting converted
> correctly, but it is not possible to narrow down where this is without
> more info.

Ah, perhaps I should make the issue clearer.

When I run this at 2017-11-01 20:55:30

(org-2ft "")
1509594900.0

(org-2ft "<2017-11-01 20:55:30>")
1509569700.0

Therefore when org.el does a tag search for SCHEDULED="", it will
not match <2017-11-01 20:55:30>, assuming that I have frozen my system
clock to that time.

>
> If you can provide an ECM then we can avoid assumptions and different
> interpretations and focus on the issue.

Again, this is dependent on your local time zone.  Try setting your
time zone to America/Los_Angeles.

1. emacs -Q
2. Assume it is 2017-10-10 12:00:00 local time
3. Create an agenda file

* TODO thing
SCHEDULED:<2017-10-10 13:00:00>

4. M-x org-agenda RET m SCHEDULED>""

Expected:

thing shows up

Actual:

thing does not show up

>
> Tim
>
> --
> Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Allen Li  writes:

> On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross  wrote:
>>
>> Let me try to clarify and see if that helps.
>>
>> Org is only forcing time into UTC format for functions which deal with
>> time comparisons. i.e. those which use org-2ft. The reason for doing
>> this is to try and avoid issues relating to daylight savings timezone
>> changes when performing comparison operations.
>>
>> It is not clear to me exactly how this is actually creating a problem
>> for your scenario as there does not seem to be anywhere in org where
>> values created by org-2ft are converted back to human readable time
>> strings. If you can provide more details on how this is a practical
>> issue in org-mode that might help.
>
> I think I made the problem quite clear multiple times.
>
> Assuming today is 2017-10-31, SCHEDULED<"" will match results
> that have SCHEDULED=<2017-11-01> depending on your local time zone.
>

I get that, what I am not clear about is where you encounter this issue
when using org .eg. is it when entering now as a date for generating an
agenda of past or current scheduled items etc or is this a problem with
some custom functions you have written where you need to enter 'now'
manually and due to the use of UTC in time comparisons are forced to
adjust manually for UTC or something else? This is what is not
clear. From my own use, I don't see the issue you describe with
scheduled items not appearing or appearing when they shouldn't, so don't
know how to reproduce the issue your having i.e. when I do agenda
searches, I don't see deadline or scheduled items showing up when they
should not etc.

This matters because if the issue is the former, then we likely do have
a real bug. However, if it is the latter, then it may be a bug or it may
be just an implementation weakness or it could just be a difference of
opinions on how it should be.

Your examples don't clarify what the issue is. It is obvious that if we
convert times to UTC for calculation purposes that this will affect the
display of those timestamps, but as far as I can tell, org doesn't
convert back and use those converted values in any visible way. Provided
all values are converted consistently, it should not matter. Therefore,
if there is a problem, then something must not be getting converted
correctly, but it is not possible to narrow down where this is without
more info.

If you can provide an ECM then we can avoid assumptions and different
interpretations and focus on the issue.

Tim

--
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross  wrote:
>
> Let me try to clarify and see if that helps.
>
> Org is only forcing time into UTC format for functions which deal with
> time comparisons. i.e. those which use org-2ft. The reason for doing
> this is to try and avoid issues relating to daylight savings timezone
> changes when performing comparison operations.
>
> It is not clear to me exactly how this is actually creating a problem
> for your scenario as there does not seem to be anywhere in org where
> values created by org-2ft are converted back to human readable time
> strings. If you can provide more details on how this is a practical
> issue in org-mode that might help.

I think I made the problem quite clear multiple times.

Assuming today is 2017-10-31, SCHEDULED<"" will match results
that have SCHEDULED=<2017-11-01> depending on your local time zone.

>
> I do think there was an error in your example analysis.
>
> In your examples of
>
> (current-time-string (float-time))
> "Mon Oct 30 21:21:31 2017" ; right
>
> (current-time-string (org-time-today))
> "Mon Oct 30 00:00:00 2017" ; right
>
> (current-time-string (org-2ft "<2017-10-31>"))
> "Mon Oct 30 17:00:00 2017" ; wrong
>
> the last one is only wrong because you have failed to tell
> current-time-string that the value you are converting is a UTC time. If
> you change the code to
>
> (current-time-string (org-2ft "<2017-10-31>") "UTC")
> "Mon Oct 30 00:00:00 2017" ; correct

No, that is wrong. Mon Oct 30 00:00:00 2017 UTC != Mon Oct 30 00:00:00
2017 UTC-7

(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>"))
"2017-10-30T17:00:00+0700" ; wrong
(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>") "UTC")
"2017-10-31T00:00:00+" ; wrong

In fact, you haven't solved a thing, since (current-time-string
(org-2ft "<2017-10-31>") "UTC") == (current-time-string (org-2ft
"<2017-10-31>"))  The time is still the same, and still wrong, just
formatted differently if you silently drop the TZ info from the time
string.

>
> Now, considering your suggestion to change org-2ft to not use UTC.
>
> If we change org-2ft to not use UTC time, we end up with time
> related calculations which cross daylight savings boarders becoming
> incorrect. So, if you have a clock table where you want to calculate the
> number of hours between two timestamps and one is within daylgiht
> savings time and the other is outside it, your calculation will be out
> by an hour.

Or rather, we end up with time related calculations which cross
daylight savings becoming correct.

But I am beginning to see the picture; someone wanted time related
calculations crossing DST to be incorrect to appear "right" to time
zone naive applications.

>
> As org-2ft is not used in conversion of 'epoch' time to times strings by
> org, I'm not sure reverting the change gains us anything and will likely
> adversely impact things like clock table calculations. What may be
> justified is to add a note to the manual which clearly states that if
> you are using org-2ft in your own code and plan to convert the result
> back to a timestamp string, you need to ensure you explicitly tell the
> function performing the conversion the input value is in UTC and not
> local time.

Yes, org-2ft documentation should be updated at the very least,
assuming that this behavior is desired.

>
> I agree that enhancing org timestamps to include TZ information would be
> a non-trivial chunk of work, I do feel it is the only solid way to
> address all of these issues. I suspect there are still edge cases in
> time related computations which fail even with the UTC approach and
> there are certainly extensions which could benefit from explicit TZ
> information (for example, taskgjuggler, which does time calculations for
> planning and where such calculations can easily cross multiple daylight
> savings TZ changes or include operations covering different timezones
> etc).
>
> Tim
>
> Allen Li  writes:
>
>> Apologies, my reply was partially omitted.  My full reply follows
>>
>> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>>>
>>> Allen Li  writes:
>>>
 On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>
> My preferences would be
>
> 1. If a timestamp does not include the TZ, then assume the local TZ
> 2. If a timestamp does include the TZ, honour that TZ

 Org mode does not support TZ in time strings and adding support isn't
 the topic of the current bug.

>>>
>>> I disagree. The root cause of the bug is due to a lack of clarity
>>> regarding timezones. I also suspect this is also the basic cause of
>>> issues relating to the use of timestamps/time strings and daylight
>>> savings, especially when it comes to performing calculations etc.
>>
>> Unless I missed something, Org mode does not support explicit
>> timezones in Org timestamps, and adding such support 

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Let me try to clarify and see if that helps.

Org is only forcing time into UTC format for functions which deal with
time comparisons. i.e. those which use org-2ft. The reason for doing
this is to try and avoid issues relating to daylight savings timezone
changes when performing comparison operations.

It is not clear to me exactly how this is actually creating a problem
for your scenario as there does not seem to be anywhere in org where
values created by org-2ft are converted back to human readable time
strings. If you can provide more details on how this is a practical
issue in org-mode that might help.

I do think there was an error in your example analysis.

In your examples of

(current-time-string (float-time))
"Mon Oct 30 21:21:31 2017" ; right

(current-time-string (org-time-today))
"Mon Oct 30 00:00:00 2017" ; right

(current-time-string (org-2ft "<2017-10-31>"))
"Mon Oct 30 17:00:00 2017" ; wrong

the last one is only wrong because you have failed to tell
current-time-string that the value you are converting is a UTC time. If
you change the code to

(current-time-string (org-2ft "<2017-10-31>") "UTC")
"Mon Oct 30 00:00:00 2017" ; correct

Now, considering your suggestion to change org-2ft to not use UTC.

If we change org-2ft to not use UTC time, we end up with time
related calculations which cross daylight savings boarders becoming
incorrect. So, if you have a clock table where you want to calculate the
number of hours between two timestamps and one is within daylgiht
savings time and the other is outside it, your calculation will be out
by an hour.

As org-2ft is not used in conversion of 'epoch' time to times strings by
org, I'm not sure reverting the change gains us anything and will likely
adversely impact things like clock table calculations. What may be
justified is to add a note to the manual which clearly states that if
you are using org-2ft in your own code and plan to convert the result
back to a timestamp string, you need to ensure you explicitly tell the
function performing the conversion the input value is in UTC and not
local time.

I agree that enhancing org timestamps to include TZ information would be
a non-trivial chunk of work, I do feel it is the only solid way to
address all of these issues. I suspect there are still edge cases in
time related computations which fail even with the UTC approach and
there are certainly extensions which could benefit from explicit TZ
information (for example, taskgjuggler, which does time calculations for
planning and where such calculations can easily cross multiple daylight
savings TZ changes or include operations covering different timezones
etc).  

Tim

Allen Li  writes:

> Apologies, my reply was partially omitted.  My full reply follows
>
> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>>
>> Allen Li  writes:
>>
>>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:

 My preferences would be

 1. If a timestamp does not include the TZ, then assume the local TZ
 2. If a timestamp does include the TZ, honour that TZ
>>>
>>> Org mode does not support TZ in time strings and adding support isn't
>>> the topic of the current bug.
>>>
>>
>> I disagree. The root cause of the bug is due to a lack of clarity
>> regarding timezones. I also suspect this is also the basic cause of
>> issues relating to the use of timestamps/time strings and daylight
>> savings, especially when it comes to performing calculations etc.
>
> Unless I missed something, Org mode does not support explicit
> timezones in Org timestamps, and adding such support would take a
> large effort [1].
>
> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html
>
> Unfortunately, I do not have time for the task.  Thus, Org mode
> will have to make due with only Org timestamps without an
> explicit timezone unless a savior appears.
>
>>
>>> I want to emphasize the distinction between timestamps and time
>>> strings.  Timestamps are a
>>> float value, Unix timestamps.  Time strings are Org mode specific,
>>> like <2017-10-30 12:34:56>
>>
>> Org mode refers to what you are calling time strings as timestamps. In
>> reality, there is no difference - one is just a numeric representation
>> and the other is a string representation. It is good you have clarified
>> your definition to reduce confusion. However, I think the problems are
>> arising because of a lack of explicit TZ handling.
>
> There's a pretty significant difference.  Unix timestamps are
> standardized and used widely by most computers and Emacs itself.
> Unix timestamps are time zone agnostics.
>
> Org timestamps (time strings, whatever) are used exclusively by
> Org mode.  As Org timestamps do not support explicit timezones,
> they are not time zone agnostic.  The interpretation of an Org
> timestamp depends on the time zone of the local machine, whereas
> the interpretation of a 

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 1:55 PM, Nicolas Goaziou  wrote:
>
> IIRC, the point is to remove DST in durations, i.e., in the difference
> between two dates. One way to do that is to assume UTC in both start end
> end date. Most of the commits are about making sure UTC is used whenever
> two Org dates are to be used in a duration computation, and nowhere
> else.

Alas, I still can't seem to find the original DST bug.  I'm not sure
using UTC solves DST problems.

For example, in the timezone America/Los_Angeles,

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour

This is what Emacs gives me using the default time zone

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 4 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 2 hour

This is what Emacs gives me using UTC

<2017-11-05 01:00:00> -> <2017-11-05 04:00:00> = 3 hours
<2017-10-10 01:00:00> -> <2017-10-10 04:00:00> = 3 hours
<2017-03-12 01:00:00> -> <2017-03-12 04:00:00> = 3 hours

Using UTC seems strictly wrong to me.

>
> I think the change to org-2ft was a side-effect, since it is indirectly
> used is a duration.
>
> Regards,
>
> --
> Nicolas Goaziou0x80A93738

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nicolas Goaziou]

Hello,

Allen Li  writes:

> I guess the relevant bug is
> http://lists.gnu.org/archive/html/emacs-orgmode/2017-07/msg00097.html,

FWIW, this is not the initial bug.

> From what I can glean from the history,
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 is where things started to go
> wrong.  UTC timezone was added to tests.  From there, a number of
> commits were added to fix regressions.
>
> The problem is that the tests were made to assume UTC timezone and Org
> mode was changed to also use UTC timezone to make the tests pass;
> however, this means that org.el no longer works for non-UTC time
> zones.
>
> That commit's message (112c5ba479d52c3c36de5c7aafd14ab6bc075005) also
> seems suspect.  It claims to be removing DST offset by enforcing UTC.
> That could only be true for people using GMT.  In GMT during DST,
> enforcing UTC would "remove" the DST offset, but that is not true
> anywhere else.
>
> Given my chain of reasoning, the commits following
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 to fix "regressions" is
> really just converting all of Org mode's timestamps to use the
> "timezone adjusted" Unix timestamps introduced by
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005

IIRC, the point is to remove DST in durations, i.e., in the difference
between two dates. One way to do that is to assume UTC in both start end
end date. Most of the commits are about making sure UTC is used whenever
two Org dates are to be used in a duration computation, and nowhere
else.

I think the change to org-2ft was a side-effect, since it is indirectly
used is a duration.

Regards,

-- 
Nicolas Goaziou0x80A93738

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

Apologies, my reply was partially omitted.  My full reply follows

On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>
> Allen Li  writes:
>
>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>>
>>> My preferences would be
>>>
>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>> 2. If a timestamp does include the TZ, honour that TZ
>>
>> Org mode does not support TZ in time strings and adding support isn't
>> the topic of the current bug.
>>
>
> I disagree. The root cause of the bug is due to a lack of clarity
> regarding timezones. I also suspect this is also the basic cause of
> issues relating to the use of timestamps/time strings and daylight
> savings, especially when it comes to performing calculations etc.

Unless I missed something, Org mode does not support explicit
timezones in Org timestamps, and adding such support would take a
large effort [1].

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html

Unfortunately, I do not have time for the task.  Thus, Org mode
will have to make due with only Org timestamps without an
explicit timezone unless a savior appears.

>
>> I want to emphasize the distinction between timestamps and time
>> strings.  Timestamps are a
>> float value, Unix timestamps.  Time strings are Org mode specific,
>> like <2017-10-30 12:34:56>
>
> Org mode refers to what you are calling time strings as timestamps. In
> reality, there is no difference - one is just a numeric representation
> and the other is a string representation. It is good you have clarified
> your definition to reduce confusion. However, I think the problems are
> arising because of a lack of explicit TZ handling.

There's a pretty significant difference.  Unix timestamps are
standardized and used widely by most computers and Emacs itself.
Unix timestamps are time zone agnostics.

Org timestamps (time strings, whatever) are used exclusively by
Org mode.  As Org timestamps do not support explicit timezones,
they are not time zone agnostic.  The interpretation of an Org
timestamp depends on the time zone of the local machine, whereas
the interpretation of a Unix timestamp does not depend on the
time zone of the local machine.

>
>>
>> Timestamps do not have timezone information since they describe an
>> exact (well, minus leap seconds)
>> point in time, the number of seconds after the epoch.
>>
>
> A point in time is measured in number of seconds since epoch. However,
> if you want to use local (wall) time to display that point, you have to
> include TZ information when converting that number to a more
> meaningful/usable (for humans) format.

I'm not sure what you mean.  Time zone information is provided by
the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
and tell it to format it in human readable format in the local
time zone (the default behavior), then it will be done, without
my having to attach time zone information anywhere.

> The point 'now' for me is UTC+1100
> and for you (based on your previous post) UTC-0700, so our
> representations in string format of this value will be different. Even
> on a single machine, it is also relevant. For example, if I have two org
> timestamps (your times strings) and I want to calculate the number of
> hours between the two, I need to include TZ information as one timestamp
> might be during daylight savings time and the other outside daylight
> savings time.

Ideally, except Org mode does not support including explicit TZ
information in Org timestamps.  Thus, Org mode can only current
interpret Org timestamps according to some blessed time zone.
Four months ago, that blessed time zone changed from the local
machine's time zone to UTC.

> Any calculation which fails to consider TZ information in
> this case will be out by an hour.
>
> If we just take the simple solution and ignore TZ information,

Org timestamps do not have TZ information.  It's not a matter of
not ignoring TZ information, support for TZ information would
have to be added, which as I understand is non-trivial.

> we will
> break calculations which cross daylight savings 'boarders'. I also
> suspect we will get inconsistent behaviour when integrating with other
> systems (such as external calendars, ical integration or sharing
> timestamp data with others etc).

As long as we use Unix timestamps when integrating with other
systems, there is no ambiguity.

>
>>> 3. If the timestamp does not include a time component, default to 0:00:0
>>
>> This is what Org mode does
>>
>>> for the local TZ
>>
>> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>>
>>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>>
>>> Rationale is that the user should have ability to fully control how
>>> their timestamps are represented. However, adding the TZ is probably
>>> just a pain for the majority of use cases, so defaulting to the local
>>> 

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross  wrote:
>
> Allen Li  writes:
>
>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>>
>>> My preferences would be
>>>
>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>> 2. If a timestamp does include the TZ, honour that TZ
>>
>> Org mode does not support TZ in time strings and adding support isn't
>> the topic of the current bug.
>>
>
> I disagree. The root cause of the bug is due to a lack of clarity
> regarding timezones. I also suspect this is also the basic cause of
> issues relating to the use of timestamps/time strings and daylight
> savings, especially when it comes to performing calculations etc.

Unless I missed something, Org mode does not support explicit
timezones in Org timestamps, and adding such support would take a
large effort [1].

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html

Unfortunately, I do not have time for the task.  Thus, Org mode
will have to make due with only Org timestamps without an
explicit timezone unless a savior appears.

>
>> I want to emphasize the distinction between timestamps and time
>> strings.  Timestamps are a
>> float value, Unix timestamps.  Time strings are Org mode specific,
>> like <2017-10-30 12:34:56>
>
> Org mode refers to what you are calling time strings as timestamps. In
> reality, there is no difference - one is just a numeric representation
> and the other is a string representation. It is good you have clarified
> your definition to reduce confusion. However, I think the problems are
> arising because of a lack of explicit TZ handling.

There's a pretty significant difference.  Unix timestamps are
standardized and used widely by most computers and Emacs itself.
Unix timestamps are time zone agnostics.

Org timestamps (time strings, whatever) are used exclusively by
Org mode.  As Org timestamps do not support explicit timezones,
they are not time zone agnostic.  The interpretation of an Org
timestamp depends on the time zone of the local machine, whereas
the interpretation of a Unix timestamp does not depend on the
time zone of the local machine.

>
>>
>> Timestamps do not have timezone information since they describe an
>> exact (well, minus leap seconds)
>> point in time, the number of seconds after the epoch.
>>
>
> A point in time is measured in number of seconds since epoch. However,
> if you want to use local (wall) time to display that point, you have to
> include TZ information when converting that number to a more
> meaningful/usable (for humans) format.

I'm not sure what you mean.  Time zone information is provided by
the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
and tell it to format it in human readable format in the local
time zone (the default behavior), then it will be done, without
my having to attach time zone information anywhere.

> The point 'now' for me is UTC+1100
> and for you (based on your previous post) UTC-0700, so our
> representations in string format of this value will be different. Even
> on a single machine, it is also relevant. For example, if I have two org
> timestamps (your times strings) and I want to calculate the number of
> hours between the two, I need to include TZ information as one timestamp
> might be during daylight savings time and the other outside daylight
> savings time.

Ideally, except Org mode does not support including explicit TZ
information in Org timestamps.  Thus, Org mode can only current
interpret Org timestamps according to some blessed time zone.
Four months ago, that blessed time zone changed from the local
machine's time zone to UTC.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

Allen Li  writes:

> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>>
>> My preferences would be
>>
>> 1. If a timestamp does not include the TZ, then assume the local TZ
>> 2. If a timestamp does include the TZ, honour that TZ
>
> Org mode does not support TZ in time strings and adding support isn't
> the topic of the current bug.
>

I disagree. The root cause of the bug is due to a lack of clarity
regarding timezones. I also suspect this is also the basic cause of
issues relating to the use of timestamps/time strings and daylight
savings, especially when it comes to performing calculations etc.

> I want to emphasize the distinction between timestamps and time
> strings.  Timestamps are a
> float value, Unix timestamps.  Time strings are Org mode specific,
> like <2017-10-30 12:34:56>

Org mode refers to what you are calling time strings as timestamps. In
reality, there is no difference - one is just a numeric representation
and the other is a string representation. It is good you have clarified
your definition to reduce confusion. However, I think the problems are
arising because of a lack of explicit TZ handling.

>
> Timestamps do not have timezone information since they describe an
> exact (well, minus leap seconds)
> point in time, the number of seconds after the epoch.
>

A point in time is measured in number of seconds since epoch. However,
if you want to use local (wall) time to display that point, you have to
include TZ information when converting that number to a more
meaningful/usable (for humans) format. The point 'now' for me is UTC+1100
and for you (based on your previous post) UTC-0700, so our
representations in string format of this value will be different. Even
on a single machine, it is also relevant. For example, if I have two org
timestamps (your times strings) and I want to calculate the number of
hours between the two, I need to include TZ information as one timestamp
might be during daylight savings time and the other outside daylight
savings time. Any calculation which fails to consider TZ information in
this case will be out by an hour.

If we just take the simple solution and ignore TZ information, we will
break calculations which cross daylight savings 'boarders'. I also
suspect we will get inconsistent behaviour when integrating with other
systems (such as external calendars, ical integration or sharing
timestamp data with others etc).

>> 3. If the timestamp does not include a time component, default to 0:00:0
>
> This is what Org mode does
>
>> for the local TZ
>
> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>
>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>
>> Rationale is that the user should have ability to fully control how
>> their timestamps are represented. However, adding the TZ is probably
>> just a pain for the majority of use cases, so defaulting to the local
>> (wall) TZ is OK provided you can override this consistently by adding
>> explicit TZ values.
>>
>> However, there is some devil in the details we need to work out. For
>> example, should we support both TZ names (like AEDT or Australia/Sydney)
>> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
>> always add TZ info in timestamps which include time components and if
>> so, how complex will this need to be e.g. handle setting a future/past
>> timestamp which is in a different (daylight savings) offset and what
>> about the additional complexity in dealing with timestamp calculations
>> where dates might be in different offsets due to daylight savings -
>> while all quite possible, it does add significant complexity and this
>> may have adverse impact on performance. Not to say we shouldn't do it,
>> just that it will take significant work.
>>
>> I suspect just the first part won't have major impact - at least no more
>> than enforcing UTC in org-2ft.
>
> Org mode does not support TZ in time strings currently.  While I would
> like such a feature very much,
> adding TZ support isn't the topic of the current bug, just fixing how
> time strings are interpreted.
>

We could make the decision that all org timestamps are relative to the
local TZ (ignoring the daylight savings issue) and that would probably
be OK for most users, but I feel the whole issue is due to a lack of
adequate/consistent TZ interpretation. The problem with org-2ft is that
it forces the conversion into UTC when the input time string may
actually be in local time, resulting in a number which is out by
whatever the local offset is. When you then convert back, your string
representation will be out by at least that amount (possibly more
depending on whether TZ info is applied in the conversion back to a
string).

If it wasn't for the benefits of org files being essentially plain text
files, the easiest solution would possibly be to represent timestamps as
a number and use an overlay or similar to convert it to a 

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross  wrote:
>
> My preferences would be
>
> 1. If a timestamp does not include the TZ, then assume the local TZ
> 2. If a timestamp does include the TZ, honour that TZ

Org mode does not support TZ in time strings and adding support isn't
the topic of the current bug.

I want to emphasize the distinction between timestamps and time
strings.  Timestamps are a
float value, Unix timestamps.  Time strings are Org mode specific,
like <2017-10-30 12:34:56>

Timestamps do not have timezone information since they describe an
exact (well, minus leap seconds)
point in time, the number of seconds after the epoch.

> 3. If the timestamp does not include a time component, default to 0:00:0

This is what Org mode does

> for the local TZ

This is what Org mode used to do, now it interprets it as 00:00:00 UTC.

> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>
> Rationale is that the user should have ability to fully control how
> their timestamps are represented. However, adding the TZ is probably
> just a pain for the majority of use cases, so defaulting to the local
> (wall) TZ is OK provided you can override this consistently by adding
> explicit TZ values.
>
> However, there is some devil in the details we need to work out. For
> example, should we support both TZ names (like AEDT or Australia/Sydney)
> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
> always add TZ info in timestamps which include time components and if
> so, how complex will this need to be e.g. handle setting a future/past
> timestamp which is in a different (daylight savings) offset and what
> about the additional complexity in dealing with timestamp calculations
> where dates might be in different offsets due to daylight savings -
> while all quite possible, it does add significant complexity and this
> may have adverse impact on performance. Not to say we shouldn't do it,
> just that it will take significant work.
>
> I suspect just the first part won't have major impact - at least no more
> than enforcing UTC in org-2ft.

Org mode does not support TZ in time strings currently.  While I would
like such a feature very much,
adding TZ support isn't the topic of the current bug, just fixing how
time strings are interpreted.

>
> Tim
>
> P.S. when you start to think about it, it is easy to see how Java
> screwed up this stuff so badly!

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

My preferences would be

1. If a timestamp does not include the TZ, then assume the local TZ
2. If a timestamp does include the TZ, honour that TZ
3. If the timestamp does not include a time component, default to 0:00:0
for the local TZ
4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.

Rationale is that the user should have ability to fully control how
their timestamps are represented. However, adding the TZ is probably
just a pain for the majority of use cases, so defaulting to the local
(wall) TZ is OK provided you can override this consistently by adding
explicit TZ values.

However, there is some devil in the details we need to work out. For
example, should we support both TZ names (like AEDT or Australia/Sydney)
and POSIX style +11/+11:00/+1100, should we add an option to tell org to
always add TZ info in timestamps which include time components and if
so, how complex will this need to be e.g. handle setting a future/past
timestamp which is in a different (daylight savings) offset and what
about the additional complexity in dealing with timestamp calculations
where dates might be in different offsets due to daylight savings -
while all quite possible, it does add significant complexity and this
may have adverse impact on performance. Not to say we shouldn't do it,
just that it will take significant work.

I suspect just the first part won't have major impact - at least no more
than enforcing UTC in org-2ft.

Tim

P.S. when you start to think about it, it is easy to see how Java
screwed up this stuff so badly!


Allen Li  writes:

> On Tue, Oct 31, 2017 at 10:41 PM, Tim Cross  wrote:
>>
>> I think this whole issue really requires a lot more analysis and
>> design. Just removing or cancelling various commits is unlikely to
>> improve matters and could result in new problems.
>
> You're right, but the new behavior was introduced fairly recently.
>
>> For org to work correctly, especially when interacting/interfacing with
>> other systems, such as external calendars, the use of timestamps must
>> handle timezones consistently and accurately. This is the only way that
>> any daylight savings calculations will work consistently as different
>> timezones have different rules for when daylight savings start/finish
>> (and these rules change).
>>
>> If the tests only support UTC/GMT timezone, they are poor tests and need
>> to be adjusted to use whatever the timezone is on the system running the
>> tests.
>>
>> I also wonder if there is some inconsistencies in how timestamps without
>> a time component are being handled. It would be good to know if the
>> issues Alan has observed exist when a full timestamp is used ie. one
>> with HH:MM:SS.s and not just date. If timezones are not been applied
>> consistently when choosing the default i.e. 00:00:00.0 with respect to
>> timezone offset, you will get inconsistencies when moving between
>> displayed (string) and calculated (number/seconds since epoch) values.
>
> I think we first need to agree on how Org mode should handle
> time.  What seems most natural to me is:
>
> * Floating point timestamps are Unix timestamps, seconds since Epoch.
> * Org format time strings are interpreted in the local machine's time zone.
>
> Let us assume that my timezone is UTC-07.  In that case,
> <2017-10-30> should be interpreted as 2017-10-30T00:00:00-0700,
> or 2017-10-30T07:00:00+.
>
> <2017-10-30>1509346800.02017-10-30T00:00:00-0700
>  2017-10-30T07:00:00+
> <2017-10-30 12:34>  1509392040.02017-10-30T12:34:00-0700
>  2017-10-30T19:34:00+
>
> Currently, org-2ft returns:
>
> <2017-10-30>1509321600.02017-10-29T17:00:00-0700
>  2017-10-30T00:00:00+
> <2017-10-30 12:34>  1509366840.02017-10-30T05:34:00-0700
>  2017-10-30T12:34:00+
>
> This is because org-2ft calls org-parse-time-string with t for
> zone, i.e. parse the time string as though it were UTC.  That
> should be apparent from the last column.
>
> Hopefully, we can agree that the former is the desired behavior.
> Working on this assumption, org-2ft should be changed to the
> original behavior, i.e., to not parse time strings as UTC.
>
> The question then becomes, what breaks if we just naively change
> org-2ft?  The new behavior was only added to org-2ft four months
> ago, so worst case is reverting every time-related change in the
> interim puts us back four months in time-related logic.
>
> But looking through the history, most of the time-related changes
> in the interim were to fix regressions caused by the initial
> logic change.  If we revert the original "regression", then those
> interim changes are also not needed.


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Tue, Oct 31, 2017 at 10:41 PM, Tim Cross  wrote:
>
> I think this whole issue really requires a lot more analysis and
> design. Just removing or cancelling various commits is unlikely to
> improve matters and could result in new problems.

You're right, but the new behavior was introduced fairly recently.

> For org to work correctly, especially when interacting/interfacing with
> other systems, such as external calendars, the use of timestamps must
> handle timezones consistently and accurately. This is the only way that
> any daylight savings calculations will work consistently as different
> timezones have different rules for when daylight savings start/finish
> (and these rules change).
>
> If the tests only support UTC/GMT timezone, they are poor tests and need
> to be adjusted to use whatever the timezone is on the system running the
> tests.
>
> I also wonder if there is some inconsistencies in how timestamps without
> a time component are being handled. It would be good to know if the
> issues Alan has observed exist when a full timestamp is used ie. one
> with HH:MM:SS.s and not just date. If timezones are not been applied
> consistently when choosing the default i.e. 00:00:00.0 with respect to
> timezone offset, you will get inconsistencies when moving between
> displayed (string) and calculated (number/seconds since epoch) values.

I think we first need to agree on how Org mode should handle
time.  What seems most natural to me is:

* Floating point timestamps are Unix timestamps, seconds since Epoch.
* Org format time strings are interpreted in the local machine's time zone.

Let us assume that my timezone is UTC-07.  In that case,
<2017-10-30> should be interpreted as 2017-10-30T00:00:00-0700,
or 2017-10-30T07:00:00+.

<2017-10-30>1509346800.02017-10-30T00:00:00-0700
 2017-10-30T07:00:00+
<2017-10-30 12:34>  1509392040.02017-10-30T12:34:00-0700
 2017-10-30T19:34:00+

Currently, org-2ft returns:

<2017-10-30>1509321600.02017-10-29T17:00:00-0700
 2017-10-30T00:00:00+
<2017-10-30 12:34>  1509366840.02017-10-30T05:34:00-0700
 2017-10-30T12:34:00+

This is because org-2ft calls org-parse-time-string with t for
zone, i.e. parse the time string as though it were UTC.  That
should be apparent from the last column.

Hopefully, we can agree that the former is the desired behavior.
Working on this assumption, org-2ft should be changed to the
original behavior, i.e., to not parse time strings as UTC.

The question then becomes, what breaks if we just naively change
org-2ft?  The new behavior was only added to org-2ft four months
ago, so worst case is reverting every time-related change in the
interim puts us back four months in time-related logic.

But looking through the history, most of the time-related changes
in the interim were to fix regressions caused by the initial
logic change.  If we revert the original "regression", then those
interim changes are also not needed.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross]

I think this whole issue really requires a lot more analysis and
design. Just removing or cancelling various commits is unlikely to
improve matters and could result in new problems.

For org to work correctly, especially when interacting/interfacing with
other systems, such as external calendars, the use of timestamps must
handle timezones consistently and accurately. This is the only way that
any daylight savings calculations will work consistently as different
timezones have different rules for when daylight savings start/finish
(and these rules change).

If the tests only support UTC/GMT timezone, they are poor tests and need
to be adjusted to use whatever the timezone is on the system running the
tests.

I also wonder if there is some inconsistencies in how timestamps without
a time component are being handled. It would be good to know if the
issues Alan has observed exist when a full timestamp is used ie. one
with HH:MM:SS.s and not just date. If timezones are not been applied
consistently when choosing the default i.e. 00:00:00.0 with respect to
timezone offset, you will get inconsistencies when moving between
displayed (string) and calculated (number/seconds since epoch) values.

Tim


Allen Li  writes:

>  On Tue, Oct 31, 2017 at 11:52 AM, Nicolas Goaziou
>  wrote:
>> Allen Li  writes:
>>
>>> Can you clarify on the issues the UTC timezone fixes?
>>
>> At the moment, I can only give you a pointer, which is commit
>> 97a1a498956da2e1961df5a0506df4cbb98fff52. Some other commits followed
>> this one in maint and master.
>>
>> You may want to check the ML for the initial bug report.
>
> Bisecting on maint shows that this regression happened at
> cc5a9bf538a4a7eb1b84d368336c46cead106e01
>
> I tested using this command:
>
> find . -name "*.elc" -delete; emacs -Q --batch --eval "(add-to-list
> 'load-path default-directory)" --eval "(require 'org)" --eval "(princ
> (current-time-string (org-2ft \"<2017-10-31>\")))"
>
> I guess the relevant bug is
> http://lists.gnu.org/archive/html/emacs-orgmode/2017-07/msg00097.html,
> but oddly enough, I cannot reproduce that bug at
> cc5a9bf538a4a7eb1b84d368336c46cead106e01~1
>
> From what I can glean from the history,
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 is where things started to go
> wrong.  UTC timezone was added to tests.  From there, a number of
> commits were added to fix regressions.
>
> The problem is that the tests were made to assume UTC timezone and Org
> mode was changed to also use UTC timezone to make the tests pass;
> however, this means that org.el no longer works for non-UTC time
> zones.
>
> That commit's message (112c5ba479d52c3c36de5c7aafd14ab6bc075005) also
> seems suspect.  It claims to be removing DST offset by enforcing UTC.
> That could only be true for people using GMT.  In GMT during DST,
> enforcing UTC would "remove" the DST offset, but that is not true
> anywhere else.
>
> Given my chain of reasoning, the commits following
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005 to fix "regressions" is
> really just converting all of Org mode's timestamps to use the
> "timezone adjusted" Unix timestamps introduced by
> 112c5ba479d52c3c36de5c7aafd14ab6bc075005


-- 
Tim Cross

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

 On Tue, Oct 31, 2017 at 11:52 AM, Nicolas Goaziou
 wrote:
> Allen Li  writes:
>
>> Can you clarify on the issues the UTC timezone fixes?
>
> At the moment, I can only give you a pointer, which is commit
> 97a1a498956da2e1961df5a0506df4cbb98fff52. Some other commits followed
> this one in maint and master.
>
> You may want to check the ML for the initial bug report.

Bisecting on maint shows that this regression happened at
cc5a9bf538a4a7eb1b84d368336c46cead106e01

I tested using this command:

find . -name "*.elc" -delete; emacs -Q --batch --eval "(add-to-list
'load-path default-directory)" --eval "(require 'org)" --eval "(princ
(current-time-string (org-2ft \"<2017-10-31>\")))"

I guess the relevant bug is
http://lists.gnu.org/archive/html/emacs-orgmode/2017-07/msg00097.html,
but oddly enough, I cannot reproduce that bug at
cc5a9bf538a4a7eb1b84d368336c46cead106e01~1

>From what I can glean from the history,
112c5ba479d52c3c36de5c7aafd14ab6bc075005 is where things started to go
wrong.  UTC timezone was added to tests.  From there, a number of
commits were added to fix regressions.

The problem is that the tests were made to assume UTC timezone and Org
mode was changed to also use UTC timezone to make the tests pass;
however, this means that org.el no longer works for non-UTC time
zones.

That commit's message (112c5ba479d52c3c36de5c7aafd14ab6bc075005) also
seems suspect.  It claims to be removing DST offset by enforcing UTC.
That could only be true for people using GMT.  In GMT during DST,
enforcing UTC would "remove" the DST offset, but that is not true
anywhere else.

Given my chain of reasoning, the commits following
112c5ba479d52c3c36de5c7aafd14ab6bc075005 to fix "regressions" is
really just converting all of Org mode's timestamps to use the
"timezone adjusted" Unix timestamps introduced by
112c5ba479d52c3c36de5c7aafd14ab6bc075005

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nicolas Goaziou]

Allen Li  writes:

> Can you clarify on the issues the UTC timezone fixes?

At the moment, I can only give you a pointer, which is commit
97a1a498956da2e1961df5a0506df4cbb98fff52. Some other commits followed
this one in maint and master.

You may want to check the ML for the initial bug report.

Regards,

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Tue, Oct 31, 2017 at 11:23 AM, Nicolas Goaziou
 wrote:
> Hello,
>
> Allen Li  writes:
>
>> Removing the t for zone fixes it
>
> [...]
>
>> I will also note that the FIXME comment in org-parse-time-string
>> suggests that it too is not handling timezones correctly.  In fact,
>> perhaps org-parse-time-string should not take a zone argument, since
>> Org does not support timezones thus the only valid value for zone is
>> nil.  I suspect that org-display-custom-time, another caller that
>> passes t for zone, is also timezone incorrect.
>
> Both changes were introduced to fix some issues with daylight saving
> time, in particular in clock reports. It is not possible to simply
> suggest reverting them without considering the underlying issues.
>
> I agree there are issues to fix. Unfortunately, the solution you suggest
> is not sufficient.

Let me clarify the exact behavior that needs to be fixed.

Assuming today is 2017-10-31, SCHEDULED<"" will match results
that have SCHEDULED=<2017-11-01> depending on your local time zone.

Can you clarify on the issues the UTC timezone fixes?  Because my
understanding at this point is that Org timestamps should be
interpreted as localtime, yet they are being interpreted as UTC time.
I can't see how, e.g.,  if I am in timezone UTC-5, I want all my Org
timestamps <2017-11-01> to be interpreted as <2017-10-31 19:00>.

Assuming that is in fact the behavior we want for some reason, then
, or rather everywhere (float-time) is used in Org mode, should
receive a corresponding time shift.

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Nicolas Goaziou]

Hello,

Allen Li  writes:

> Removing the t for zone fixes it

[...]

> I will also note that the FIXME comment in org-parse-time-string
> suggests that it too is not handling timezones correctly.  In fact,
> perhaps org-parse-time-string should not take a zone argument, since
> Org does not support timezones thus the only valid value for zone is
> nil.  I suspect that org-display-custom-time, another caller that
> passes t for zone, is also timezone incorrect.

Both changes were introduced to fix some issues with daylight saving
time, in particular in clock reports. It is not possible to simply
suggest reverting them without considering the underlying issues.

I agree there are issues to fix. Unfortunately, the solution you suggest
is not sufficient.

> tl;dr time is hard.

I agree.


Regards,

-- 
Nicolas Goaziou

Re: [O] Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li]

On Mon, Oct 30, 2017 at 5:40 PM, Allen Li  wrote:
> (current-time-string (time-to-seconds (org-2ft "<2017-10-31>")))
> "Sun Oct 29 17:00:00 2017"
>
> This seems wrong
>
> In org-2ft
>
> (org-parse-time-string s nil t)
>
> The t means use UTC instead of Emacs local time.
>
> However, Org translates  into (float-time), which is in Emacs localtime.
>
> 1. SCHEDULED>"" compares a UTC time against a local time.
> 2. Either org-2ft should be fixed to be localtime, or  should be
> (float-time) in UTC.
>
> I don't know how Org internals works, but my experience so far has
> been that Emacs and *nix in general is very naive about timezones; a
> naked timestamp is assumed to be localtime if it does not have
> accompanying timezone information.
>
> Thus, it seems to be more correct to change org-2ft to parse times as
> localtime.  However, I don't know if UTC timestamps are assumed by
> other parts of Org internals, in which case fixing  (float-time)
> would be safest.

My initial analysis was wrong because I wasn't thinking clearly.

Just to set a stake in the ground: timestamps are seconds since epoch
and timezone neutral.  Emacs time values are also timezone neutral:
(sec-high sec-low microsec picosec)

(current-time-string (float-time))
"Mon Oct 30 21:21:31 2017" ; right

(current-time-string (org-time-today))
"Mon Oct 30 00:00:00 2017" ; right

(current-time-string (org-2ft "<2017-10-31>"))
"Mon Oct 30 17:00:00 2017" ; wrong

Removing the t for zone fixes it

(defun org-2ft (s)
  "Convert S to a floating point time.
If S is already a number, just return it.  If it is a string, parse
it as a time string and apply `float-time' to it.  If S is nil, just return 0."
  (cond
   ((numberp s) s)
   ((stringp s)
(condition-case nil
(float-time (apply #'encode-time (org-parse-time-string s)))
  (error 0.)))
   (t 0.)))

(current-time-string (org-2ft "<2017-10-31>"))
"Tue Oct 31 00:00:00 2017" ; now right

I will also note that the FIXME comment in org-parse-time-string
suggests that it too is not handling timezones correctly.  In fact,
perhaps org-parse-time-string should not take a zone argument, since
Org does not support timezones thus the only valid value for zone is
nil.  I suspect that org-display-custom-time, another caller that
passes t for zone, is also timezone incorrect.

That is to say, nothing in Org allows passing in a custom timezone,
let alone a UTC timezone; thus, every caller that passes a non-nil
zone to org-parse-time-string is getting an incorrect result.

tl;dr time is hard.