Re: [O] bug in org-habits

[Bastien Guerry]

Hi John and Achim,

John Wiegley  writes:

>> Achim Gratz  writes:
>
>> I don't think so. Search for end of entry can be complex in itself and you
>> would never know if the properties you find by looking back aren't belonging
>> to an entry one level down unless you scanned the whole span again. Also,
>> properties can be any size and you have to search for the beginning of the
>> drawer.
>
> Good point.  Still, the option I'd still choose would be:
>
> org-allow-floating-properties
>
>   If set to `non-nil', the :PROPERTIES: drawer may begin anywhere within
>   an entry. Note that this will slow down many operations due to the
>   additional scan needed, and may lead to incorrect result if similar text
>   is found elsewhere within an entry, which is the recommended and default
>   value for this variable is nil.

I think Achim's points above are very valid, and the flexibility
offered by the option above should be very carefully examined.

I'm trying hard to put aside one day or two for org-mode next week,
I'll come back on this issue then.

Thanks,

-- 
 Bastien

Re: [O] bug in org-habits

[John Wiegley]

> Bastien Guerry  writes:

> I think Achim's points above are very valid, and the flexibility offered by
> the option above should be very carefully examined.

I spoke to Nicolas directly and he mentioned that a goal for syntax regularity
is to make it possible to reliably read and manipulate Org files outside of
Emacs.

For this I *am* willing to give up order independence of PROPERTIES. Having a
customization option would needlessly increases the number of possibilities
external processors must consider, and so I retract my request.

John

Re: [O] bug in org-habits

[Stelian Iancu]

On 04/11/15 02:01, John Wiegley wrote:

Nicolas Goaziou  writes:

Also, supporting both locations means that users will pay the full price in
entries without a property drawer, even if they chose the fast path, i.e.,
drawer at the beginning of the entry, in the first place. This kind of
defeats some of the advantages of the current state.


It wouldn't be "users": it would be people intentionally opting to allow
floating properties. _I_ would be paying the price, and I will pay it happily
to keep 8.2 behavior.

John



I would also like to have the 8.2 behavior. If it's easier, it would be 
fine to always have the drawer either in the beginning or in the end and 
nowhere else (and have an option to chose with the default in the 
beginning).


S.

Re: [O] bug in org-habits

[Eric S Fraga]

On Tuesday,  3 Nov 2015 at 20:01, John Wiegley wrote:

[...]

> I'm having a hard time buying the performance argument, since I've been using
> Org-mode for many years, and never has this been a problem. You're making me
> pay a cost (enforced structure), for a value only one of us perceives.

John,

others *have* found issues with performance.

In 8.2, org struggled with many of my documents, some of which are very
large.  Most, if not all, of the performance problems I was having have
disappeared since then (thank you Nicolas!).  I cannot attribute the
improvements directly to the move to a fixed structure for headline
meta-data but I can easily believe that this move has contributed due to
anecdotal experience with complex sections within my org files.

This does not mean I am against your suggestion of a variable
controlling this behaviour.  Maybe implement it and post a patch for the
community to try out?

I am perfectly happy with the current format, however, so would be
unlikely to try it.  (sorry)

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.50.2, Org release_8.3.2-209-gba4d33

Re: [O] bug in org-habits

[John Wiegley]

> Puneeth Chaganti  writes:

>> Actually there has been introduced a constraint on the ordering planning
>> lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
>> This at least invalidates to use PROPERTIES before SCHEDULED afaics.

> Yes, that is correct and you can use the `org-repair-property-drawers`
> utility function provided to fix your org trees.

I would very much like to see this constraint removed from 8.3. I have always
preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
are arranged.

In fact, this one aspect of 8.3 has kept me at 8.2. I do not understand the
need for such an abitrary restriction.

John

Re: [O] bug in org-habits

[Puneeth Chaganti]

On Tue, Nov 3, 2015 at 3:26 PM, Marco Wahl  wrote:
>
>
> Actually there has been introduced a constraint on the ordering planning
> lines and property drawers in 8.3.  See http://orgmode.org/Changes.html.
>
> This at least invalidates to use PROPERTIES before SCHEDULED afaics.


Yes, that is correct and you can use the `org-repair-property-drawers`
utility function provided to fix your org trees.

-- 
Puneeth

Re: [O] bug in org-habits

[Marco Wahl]

"Mark A. Hershberger"  writes:

> org-habit.el is sensitive to the ordering PROPERTIES and SCHEDULED
> and expects them /in only/ the following order:
>
>  * TODO habit name
>  SCHEDULED: <2015-11-03 Tue 07:00 .+1d>
>  :PROPERTIES:
>  :LAST_REPEAT: [2015-11-02 Mon 07:54]
>  :STYLE: habit
>  :END:
>
> Any other ordering, or doubling of PROPERTIES (e.g. STYLE in one, LAST
> REPEAT in another) and the habit will show up as a simple scheduled item
> and not a habit in the agenda view.
>
> org-version: 8.3.2

Actually there has been introduced a constraint on the ordering planning
lines and property drawers in 8.3.  See http://orgmode.org/Changes.html.

This at least invalidates to use PROPERTIES before SCHEDULED afaics.


HTH
-- 
Marco Wahl
GPG: 0x49010A040A3AE6F2

Re: [O] bug in org-habits

[John Wiegley]

> Nicolas Goaziou  writes:

> This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).

Ah, I had misspoken. It's not SCHEDULE-before-PROPERTIES that has broken for
me, it is the inability to have PROPERTIES at the very end of the entry.

Why must the properties drawer appear before the entry's body?

John

Re: [O] bug in org-habits

[Nicolas Goaziou]

Hello,

John Wiegley  writes:

>> Nicolas Goaziou  writes:
>
>> This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).
>
> Ah, I had misspoken. It's not SCHEDULE-before-PROPERTIES that has broken for
> me, it is the inability to have PROPERTIES at the very end of the entry.
>
> Why must the properties drawer appear before the entry's body?

IIRC, you already asked the question. 

It is for efficiency reasons. Properties are an important feature in
Org, letting them anywhere in a potentially long entry doesn't scale
well.


Regards,

-- 
Nicolas Goaziou

Re: [O] bug in org-habits

[Achim Gratz]

John Wiegley writes:
> Thanks for discussing this with me, Nicolas. I appreciate there may be
> technical complexities involved. Could we special-case allow PROPERTIES to be
> the *very last thing* in an entry? I don't need it to float anywhere else. I
> just like it to be at the end.

Well, that's precisely the thing that doesn't scale and that Nicolas
wanted to avoid.  Putting the properties at the beginning of an entry
makes the search pretty much constant time and if you find something
else at the start of the entry then you know there aren't any and can go
on (this is pretty important for making sure property inheritance works
correctly, among other things).  If you could put them _anywhere_ else,
you'd have to keep searching until you either find them or you've
exhausted the span of the entry.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [O] bug in org-habits

[John Wiegley]

> Achim Gratz  writes:

> I don't think so. Search for end of entry can be complex in itself and you
> would never know if the properties you find by looking back aren't belonging
> to an entry one level down unless you scanned the whole span again. Also,
> properties can be any size and you have to search for the beginning of the
> drawer.

Good point.  Still, the option I'd still choose would be:

org-allow-floating-properties

  If set to `non-nil', the :PROPERTIES: drawer may begin anywhere within
  an entry. Note that this will slow down many operations due to the
  additional scan needed, and may lead to incorrect result if similar text
  is found elsewhere within an entry, which is the recommended and default
  value for this variable is nil.

John

Re: [O] bug in org-habits

[John Wiegley]

> Jonathan Leech-Pepin  writes:

> Wouldn't last item in entry scale without issues? Find end of headline
> (start of next or end of buffer) and search backwards. If first element from
> end is a property drawer you have it, otherwise you still know there is
> none.

That sounds even better than what I mentioned about re-search-forward, and
O(1) time complexity as well.

John

Re: [O] bug in org-habits

[John Wiegley]

> Nicolas Goaziou  writes:

> I'd rather not have syntax too much customizable, for portability, ease of
> maintenance, too. There are already too many mistakes in that area.

Thanks for discussing this with me, Nicolas. I appreciate there may be
technical complexities involved. Could we special-case allow PROPERTIES to be
the *very last thing* in an entry? I don't need it to float anywhere else. I
just like it to be at the end.

> If you need to end your entry with a drawer, couldn't you put any of them
> there? You can even have one there named "PROPERTIES".

But it wouldn't be the true PROPERTIES, would it? Most of my entries look like
this (and I have many thousands of them):

** TODO Update auto insurance cards :Home:ATTACH:
SCHEDULED: <2016-03-11 Fri +6m>
- State "DONE"   from "TODO"   [2015-09-11 Fri 11:28]
- State "WAITING"from "TODO"   [2015-07-22 Wed 19:48]
- State "DONE"   from "STARTED"[2014-12-13 Sat 15:56]
- State "DONE"   from "TODO"   [2014-03-31 Mon 03:11]
- State "CANCELED"   from "TODO"   [2013-12-11 Wed 19:23]
- State "CANCELED"   from "TODO"   [2013-05-21 Tue 12:30]
- State "DONE"   from "TODO"   [2012-05-17 Thu 22:24]
- State "CANCELED"   from "TODO"   [2011-04-23 Sat 22:04]
- State "TODO"   from "DONE"   [2010-10-23 Sat 21:54]
- State "DONE"   from "TODO"   [2010-10-08 Fri 14:26]
:PROPERTIES:
:ID:   B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
:CREATED:  [2010-05-08 Sat 17:34]
:LAST_REPEAT: [2015-09-11 Fri 11:28]
:Attachments: AutoInsuranceIdCards.pdf
:END:

I suppose I've just become familiar with seeing :PROPERTIES: as the "period"
at the end of the entry, and I'll like to keep it there if we can devise a
technical answer that does not make life more difficult for you.

John

Re: [O] bug in org-habits

[John Wiegley]

> Nicolas Goaziou  writes:

> It is for efficiency reasons. Properties are an important feature in Org,
> letting them anywhere in a potentially long entry doesn't scale well.

Since it scales for my use case, can I please have a customization variable to
relax this restriction? I prefer PROPERTIES drawers at the end, and would
prefer not to trade something I want for something that doesn't affect me.

Thanks, John

Re: [O] bug in org-habits

[Nicolas Goaziou]

John Wiegley  writes:

> Since it scales for my use case, can I please have a customization variable to
> relax this restriction?

I'd rather not have syntax too much customizable, for portability, ease
of maintenance, too. There are already too many mistakes in that area.

Also, please note that previous behaviour didn't handle some
pathological cases (e.g., property drawer in a comment block, or past an
inlinetask). Fixing that properly, i.e. parsing the section all the way
down to the drawer, would probably have had an impact in your use case.

The current behaviour is technically much more reliable.

> I prefer PROPERTIES drawers at the end, and would prefer not to trade
> something I want for something that doesn't affect me.

If you need to end your entry with a drawer, couldn't you put any of
them there? You can even have one there named "PROPERTIES".

Regards,

Re: [O] bug in org-habits

[John Wiegley]

> Nicolas Goaziou  writes:

> As a matter of fact, going to the end of an entry is not negligible, because
> of inlinetasks. Also, it is not really O(1) since it depends on the size of
> the entry. To get an idea, on my computer, moving past a 500 lines entry
> takes around 0.001s. I can imagine slower computers, or larger entries,
> neither being unheard of, having a slightly noticeable delay.

I'm having a hard time buying the performance argument, since I've been using
Org-mode for many years, and never has this been a problem. You're making me
pay a cost (enforced structure), for a value only one of us perceives.

> Also, supporting both locations means that users will pay the full price in
> entries without a property drawer, even if they chose the fast path, i.e.,
> drawer at the beginning of the entry, in the first place. This kind of
> defeats some of the advantages of the current state.

It wouldn't be "users": it would be people intentionally opting to allow
floating properties. _I_ would be paying the price, and I will pay it happily
to keep 8.2 behavior.

John

Re: [O] bug in org-habits

[Nicolas Goaziou]

John Wiegley  writes:

> Thanks for discussing this with me, Nicolas. I appreciate there may be
> technical complexities involved. Could we special-case allow PROPERTIES to be
> the *very last thing* in an entry? I don't need it to float anywhere else. I
> just like it to be at the end.

I think it's a false good idea.

As a matter of fact, going to the end of an entry is not negligible,
because of inlinetasks. Also, it is not really O(1) since it depends on
the size of the entry. To get an idea, on my computer, moving past a 500
lines entry takes around 0.001s. I can imagine slower computers, or
larger entries, neither being unheard of, having a slightly noticeable
delay.

Also, supporting both locations means that users will pay the full price
in entries without a property drawer, even if they chose the fast path,
i.e., drawer at the beginning of the entry, in the first place. This
kind of defeats some of the advantages of the current state.

There is also a meaner problem. Unlike to beginning of entry, end of
entry is a moving target. Org sometimes automatically adds stuff there
(e.g., footnotes when `org-footnote-section' is nil). So do other parts
of Emacs (file local variables come to mind). Preventing insertions in
this area can be tedious, if possible at all. 

OTOH, the mechanism protecting the beginning of the entry, i.e,
`org-end-of-meta-data', is already implemented since planning lines
require it anyway.

Again, speed _was_ an issue, as testified by, e.g.,
org-use-property-inheritance's docstring. Current solution is both
robust and fast, and is well worth the high price of an incompatible
change, considering its central place in Org.

>> If you need to end your entry with a drawer, couldn't you put any of them
>> there? You can even have one there named "PROPERTIES".
>
> But it wouldn't be the true PROPERTIES, would it? Most of my entries look like
> this (and I have many thousands of them):

It wouldn't, of course. But at least you would get your "period" at the
end of the entry. It doesn't matter much if it is empty since its
contents are hidden anyway.

> ** TODO Update auto insurance cards :Home:ATTACH:
> SCHEDULED: <2016-03-11 Fri +6m>
> - State "DONE"   from "TODO"   [2015-09-11 Fri 11:28]
> - State "WAITING"from "TODO"   [2015-07-22 Wed 19:48]
> - State "DONE"   from "STARTED"[2014-12-13 Sat 15:56]
> - State "DONE"   from "TODO"   [2014-03-31 Mon 03:11]
> - State "CANCELED"   from "TODO"   [2013-12-11 Wed 19:23]
> - State "CANCELED"   from "TODO"   [2013-05-21 Tue 12:30]
> - State "DONE"   from "TODO"   [2012-05-17 Thu 22:24]
> - State "CANCELED"   from "TODO"   [2011-04-23 Sat 22:04]
> - State "TODO"   from "DONE"   [2010-10-23 Sat 21:54]
> - State "DONE"   from "TODO"   [2010-10-08 Fri 14:26]
> :PROPERTIES:
> :ID:   B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
> :CREATED:  [2010-05-08 Sat 17:34]
> :LAST_REPEAT: [2015-09-11 Fri 11:28]
> :Attachments: AutoInsuranceIdCards.pdf
> :END:

The following is not ugly either

  ** TODO Update auto insurance cards :Home:ATTACH:
  SCHEDULED: <2016-03-11 Fri +6m>
  :PROPERTIES:
  :ID:   B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
  :CREATED:  [2010-05-08 Sat 17:34]
  :LAST_REPEAT: [2015-09-11 Fri 11:28]
  :Attachments: AutoInsuranceIdCards.pdf
  :END:
  - State "DONE"   from "TODO"   [2015-09-11 Fri 11:28]
  - State "WAITING"from "TODO"   [2015-07-22 Wed 19:48]
  - State "DONE"   from "STARTED"[2014-12-13 Sat 15:56]
  - State "DONE"   from "TODO"   [2014-03-31 Mon 03:11]
  - State "CANCELED"   from "TODO"   [2013-12-11 Wed 19:23]
  - State "CANCELED"   from "TODO"   [2013-05-21 Tue 12:30]
  - State "DONE"   from "TODO"   [2012-05-17 Thu 22:24]
  - State "CANCELED"   from "TODO"   [2011-04-23 Sat 22:04]
  - State "TODO"   from "DONE"   [2010-10-23 Sat 21:54]
  - State "DONE"   from "TODO"   [2010-10-08 Fri 14:26]

even more so considering it really appears as 

  ** TODO Update auto insurance cards :Home:ATTACH:
  SCHEDULED: <2016-03-11 Fri +6m>
  :PROPERTIES:
  - State "DONE"   from "TODO"   [2015-09-11 Fri 11:28]
  - State "WAITING"from "TODO"   [2015-07-22 Wed 19:48]
  - State "DONE"   from "STARTED"[2014-12-13 Sat 15:56]
  - State "DONE"   from "TODO"   [2014-03-31 Mon 03:11]
  - State "CANCELED"   from "TODO"   [2013-12-11 Wed 19:23]
  - State "CANCELED"   from "TODO"   [2013-05-21 Tue 12:30]
  - State "DONE"   from "TODO"   [2012-05-17 Thu 22:24]
  - State "CANCELED"   from "TODO"   [2011-04-23 Sat 22:04]
  - State "TODO"   from "DONE"   [2010-10-23 Sat 21:54]
  - State "DONE"   from "TODO"   [2010-10-08 Fri 14:26]

and with a fake "THE-END" drawer

  ** TODO Update auto insurance cards :Home:ATTACH:
  SCHEDULED: 

Re: [O] bug in org-habits

[John Wiegley]

> Achim Gratz  writes:

> Well, that's precisely the thing that doesn't scale and that Nicolas wanted
> to avoid. Putting the properties at the beginning of an entry makes the
> search pretty much constant time and if you find something else at the start
> of the entry then you know there aren't any and can go on (this is pretty
> important for making sure property inheritance works correctly, among other
> things). If you could put them _anywhere_ else, you'd have to keep searching
> until you either find them or you've exhausted the span of the entry.

As a user, I'm willing to pay that cost. Also, I never have other property
drawers. If it were just doing (re-search-forward "^:PROPERTIES:$"
(end-of-entry)), then it wouldn't matter where the properties drawer was, so
long as it's understood that the search may be wrong if the user has such a
string appearing elsewhere.

John

Re: [O] bug in org-habits

[Jonathan Leech-Pepin]

On November 3, 2015 4:31:11 PM EST, Achim Gratz  wrote:
>John Wiegley writes:
>> Thanks for discussing this with me, Nicolas. I appreciate there may
>be
>> technical complexities involved. Could we special-case allow
>PROPERTIES to be
>> the *very last thing* in an entry? I don't need it to float anywhere
>else. I
>> just like it to be at the end.
>
>Well, that's precisely the thing that doesn't scale and that Nicolas
>wanted to avoid.  Putting the properties at the beginning of an entry
>makes the search pretty much constant time and if you find something
>else at the start of the entry then you know there aren't any and can
>go
>on (this is pretty important for making sure property inheritance works
>correctly, among other things).  If you could put them _anywhere_ else,
>you'd have to keep searching until you either find them or you've
>exhausted the span of the entry.

Wouldn't last item in entry scale without issues? Find end of headline (start 
of next or end of buffer) and search backwards. If first element from end is a 
property drawer you have it, otherwise you still know there is none.

Jon

>
>Regards,
>Achim.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [O] bug in org-habits

[Achim Gratz]

John Wiegley writes:
>> Jonathan Leech-Pepin  writes:
>> Wouldn't last item in entry scale without issues? Find end of headline
>> (start of next or end of buffer) and search backwards. If first element from
>> end is a property drawer you have it, otherwise you still know there is
>> none.
>
> That sounds even better than what I mentioned about re-search-forward, and
> O(1) time complexity as well.

I don't think so.  Search for end of entry can be complex in itself and
you would never know if the properties you find by looking back aren't
belonging to an entry one level down unless you scanned the whole span
again.  Also, properties can be any size and you have to search for the
beginning of the drawer.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [O] bug in org-habits

[Stelian Iancu]

On 03/11/15 14:46, Marco Wahl wrote:

John Wiegley  writes:


Puneeth Chaganti  writes:



Actually there has been introduced a constraint on the ordering planning
lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
This at least invalidates to use PROPERTIES before SCHEDULED afaics.



Yes, that is correct and you can use the `org-repair-property-drawers`
utility function provided to fix your org trees.


I would very much like to see this constraint removed from 8.3. I have always
preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
are arranged.


Actually what you describe _is_ the expected order for the 8.3 files.


My issue is a bit different.

I have an org file with calendar appointments. I also have attachments 
to the appointments. The attachment appears in a PROPERTIES drawer.


Now if I have the timestamp (plain one) before the drawer, I cannot open 
the attachment. Pressing C-c C-a o just inserts another PROPERTIES 
drawer (with another ID) above the timestamp.


If, however, I move the timestamp after the PROPERTIES drawer, I can 
successfully open the attachment and the second drawer doesn't appear.


Also, when the second drawer appears, a new empty folder is created in 
the data directory of my org directory.


IMHO this is a bug, as at the moment I have to have the timestamp after 
the drawer.


Org version is Org-mode version 8.3.2 (release_8.3.2-253-g9b5757 @ 
/Users/si/elisp/org-mode/lisp/)

Re: [O] bug in org-habits

[Nicolas Goaziou]

Hello,

Stelian Iancu  writes:

> I have an org file with calendar appointments. I also have attachments
> to the appointments. The attachment appears in a PROPERTIES drawer.
>
> Now if I have the timestamp (plain one) before the drawer, I cannot
> open the attachment. Pressing C-c C-a o just inserts another
> PROPERTIES drawer (with another ID) above the timestamp.
>
> If, however, I move the timestamp after the PROPERTIES drawer, I can
> successfully open the attachment and the second drawer doesn't appear.
>
> Also, when the second drawer appears, a new empty folder is created in
> the data directory of my org directory.
>
> IMHO this is a bug, as at the moment I have to have the timestamp
> after the drawer.

This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).

If you insert a timestamp between SCHEDULED and PROPERTIES, the latter
is no longer recognized as a PROPERTIES DRAWER (it becomes, in fact,
a regular drawer which happens to be named "PROPERTIES") and you lose
any information attached to it.

You can get this information back by putting the PROPERTIES drawer (the
special one) at its expected location.

Regards,

-- 
Nicolas Goaziou

Re: [O] bug in org-habits

[Marco Wahl]

John Wiegley  writes:

>> Puneeth Chaganti  writes:
>
>>> Actually there has been introduced a constraint on the ordering planning
>>> lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
>>> This at least invalidates to use PROPERTIES before SCHEDULED afaics.
>
>> Yes, that is correct and you can use the `org-repair-property-drawers`
>> utility function provided to fix your org trees.
>
> I would very much like to see this constraint removed from 8.3. I have always
> preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
> are arranged.

Actually what you describe _is_ the expected order for the 8.3 files.

> In fact, this one aspect of 8.3 has kept me at 8.2. I do not understand the
> need for such an abitrary restriction.

So this constraint on the order should not hold you back to move on to
8.3.

I can report that in my personal org practice this constraint never got
in my way.  Further I can't imagine a use case when the user can benefit
of a SCHEDULE line below a property drawer, but this might be a bit
personal.

I can say nothing about the technical view of having that constraint.


Best regards,
-- 
Marco Wahl
GPG: 0x49010A040A3AE6F2