Re: Clarification on blank lines following list items

[Ihor Radchenko]

Ihor Radchenko  writes:

> "Tom Alexander"  writes:
>
>> I am noticing the list items have some very context-sensitive specific 
>> behavior regarding ownership of the trailing blank lines. I was hoping to 
>> get some clarification on this (namely, are my observations correct, am I 
>> stumbling across a bug, or have I not dug deep enough to suss out the real 
>> rules?). The org-mode documentation states:
>>
>>> With the exception of list items and footnote definitions blank lines 
>>> belong to the preceding element with the narrowest possible scope
>>
>> but it does not state who ends up owning those blank lines.
>
> I can see how this explanation steered you into wrong line of thoughts.
> It should better be explained from the widest scope to the narrowest
> scope, not the opposite.

I tried to clarify things in
https://git.sr.ht/~bzg/worg/commit/ac9de71c1e73ef3a1d63e58e364fc55e83f0214e

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Clarification on blank lines following list items

[Tom Alexander]

Thanks!

-- 
Tom Alexander

On Tue, Aug 22, 2023, at 4:26 AM, Ihor Radchenko wrote:
> Ihor Radchenko  writes:
>
>>> I am seeing dual-ownership of the post-blank in the examples below, but at 
>>> the same time if I put a plain-list inside a footnote definition, the 
>>> footnote definition ends up with sole custody of the post-blank.
>>
>> It is a bug in the list parser.
>
> Fixed, on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=53c9d91d3
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: Clarification on blank lines following list items

[Ihor Radchenko]

Ihor Radchenko  writes:

>> I am seeing dual-ownership of the post-blank in the examples below, but at 
>> the same time if I put a plain-list inside a footnote definition, the 
>> footnote definition ends up with sole custody of the post-blank.
>
> It is a bug in the list parser.

Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=53c9d91d3

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Clarification on blank lines following list items

[Ihor Radchenko]

"Tom Alexander"  writes:

> 1. Is the following statement true? "Two elements can count the same 
> character in their post-blank?"

This statement ought to be false.

> I am seeing dual-ownership of the post-blank in the examples below, but at 
> the same time if I put a plain-list inside a footnote definition, the 
> footnote definition ends up with sole custody of the post-blank.

It is a bug in the list parser.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Clarification on blank lines following list items

[Tom Alexander]

Thank you so much for explaining all of that! There is some good information 
there I was missing. I think the most important bit I was missing is the 
post-blank stuff. I was only looking at begin->end but I think digging into the 
post-blank is what makes this consistent.

I've got 2 separate questions:

1. Is the following statement true? "Two elements can count the same character 
in their post-blank?"
I am seeing dual-ownership of the post-blank in the examples below, but at the 
same time if I put a plain-list inside a footnote definition, the footnote 
definition ends up with sole custody of the post-blank.

2. I'm still not sure about some behavior I'm seeing. I think it would be 
easiest to see if we focus on exactly 1 blank line:

```
1. bar
2. baz
   < this blank line here
ipsum
```

In this example, the blank line gets counted in the post-blank for the 
plain-list but not for the item:
```
plain-list: post-blank 1 | begin 1 end 16 | contents-begin 1 contents-end 15
item: post-blank 0 | begin 1 end 8 | contents-begin 4 contents-end 8
paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 
8
item: post-blank 0 | begin 8 end 15 | contents-begin 11 contents-end 15
paragraph: post-blank 0 | begin 11 end 15 | contents-begin 11 
contents-end 15
paragraph: post-blank 0 | begin 16 end 22 | contents-begin 16 contents-end 22
```

but if we take that plain-list and nest it inside another plain-list:
```
1. foo
   1. bar
   2. baz
   < this blank line here
2. lorem
ipsum
```

The blank line gets counted as a post-blank for both the item "foo" and the 
item "baz":
```
plain-list: post-blank 0 | begin 1 end 38 | contents-begin 1 contents-end 38
item: post-blank 1 | begin 1 end 29 | contents-begin 4 contents-end 28
paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 
8
plain-list: post-blank 0 | begin 8 end 29 | contents-begin 8 
contents-end 29
item: post-blank 0 | begin 8 end 18 | contents-begin 14 
contents-end 18
paragraph: post-blank 0 | begin 14 end 18 | contents-begin 14 
contents-end 18
item: post-blank 1 | begin 18 end 29 | contents-begin 24 
contents-end 28
paragraph: post-blank 0 | begin 24 end 28 | contents-begin 24 
contents-end 28
item: post-blank 0 | begin 29 end 38 | contents-begin 32 contents-end 38
paragraph: post-blank 0 | begin 32 end 38 | contents-begin 32 
contents-end 38
paragraph: post-blank 0 | begin 38 end 44 | contents-begin 38 contents-end 44
```

Meaning the post-blank did this movement:
```
plain-list: post-blank 0
item: post-blank 1   <---<<<-\
paragraph: post-blank 0  |
plain-list: post-blank 0 >>--|
item: post-blank 0   |
paragraph: post-blank 0  |
item: post-blank 1   <---<---/
paragraph: post-blank 0
item: post-blank 0
paragraph: post-blank 0
paragraph: post-blank 0
```


Question ---> So why is the item "baz" gaining a post-blank instead of the 
inner plain-list (bar baz) keeping that post-blank?

I would expect it to instead be:
```
plain-list: post-blank 0
item: post-blank 1
paragraph: post-blank 0
here -> plain-list: post-blank 1
item: post-blank 0
paragraph: post-blank 0
not here -> item: post-blank 0
paragraph: post-blank 0
item: post-blank 0
paragraph: post-blank 0
paragraph: post-blank 0
```

I re-did both test cases using greater blocks and lesser blocks instead of 
paragraphs to make sure it wasn't that historical exception at the end of your 
email, and the post-blank behavior was exactly the same.


-- 
Tom Alexander

Re: Clarification on blank lines following list items

[Ihor Radchenko]

"Tom Alexander"  writes:

> I am noticing the list items have some very context-sensitive specific 
> behavior regarding ownership of the trailing blank lines. I was hoping to get 
> some clarification on this (namely, are my observations correct, am I 
> stumbling across a bug, or have I not dug deep enough to suss out the real 
> rules?). The org-mode documentation states:
>
>> With the exception of list items and footnote definitions blank lines belong 
>> to the preceding element with the narrowest possible scope
>
> but it does not state who ends up owning those blank lines.

I can see how this explanation steered you into wrong line of thoughts.
It should better be explained from the widest scope to the narrowest
scope, not the opposite.

Greater Org elements are generally represented by contents where child
elements are located + markup defining the greater element itself +
optional trailing blank lines.

For example, drawers are

:NAME:

...
:END:


Naturally, blank lines are the attribute of such drawer - they belong to
it and are recorded as :post-blank property.

The above works for many greater elements. However, it becomes a bit
tricky when a greater element does not have any "end" delimiter:


- item
  Some text
  Or even
  :drawer:
  with text
  :end:

Not an item.


Now, assigning contents vs. blank lines is not so obvious. We can either
include these blank lines into contents or keep them separate within
:post-blank property.

Then, there are two kinds of greater elements that can end with blank
lines without separator:

1. Elements where blank lines do not affect parsing (headlines)

2. Elements where trailing blank lines are syntactically meaningful and
   by themselves serve as a marker of element ending.
   
   - footnote-definition ends when Org see two consecutive blank lines
 or a heading or another footnote-definition.
 
   - plain-list also uses two consecutive blank lines as delimiter.

   
In the second case, Org makes the plain-list/footnote-definition element
"own" the blank lines (set :post-blank) instead of putting these blank
lines inside contents. If we did otherwise, changes in contents could
make the parent plain-list/footnote-definition invalid - if the double
blank delimiter is edited away.

-

Further, there is a special case with greater elements without contents:

* Heading with no contents


* Another heading

The first heading does not have contents, yet we want to record the fact
that it has multiple blank lines before the next element - :post-blank
here is set, unlike heading with contents.

-

Finally, :post-blank in items is special.

Consider:

- item 1

- item 2

- item 3

We do not treat blank between items as parts of their paragraphs
historically. Also, it makes sense for such short items.
(there are actually some reasons why we might want to alter this
historical convention, but for now it is how it is)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Clarification on blank lines following list items

[Tom Alexander]

I am noticing the list items have some very context-sensitive specific behavior 
regarding ownership of the trailing blank lines. I was hoping to get some 
clarification on this (namely, are my observations correct, am I stumbling 
across a bug, or have I not dug deep enough to suss out the real rules?). The 
org-mode documentation states:

> With the exception of list items and footnote definitions blank lines belong 
> to the preceding element with the narrowest possible scope

but it does not state who ends up owning those blank lines.

In a previous email the incredibly helpful Ihor Radchenko expanded on this 
further with:

> Also, in addition to list items, footnote-definitions do not extend their 
> contents to the trailing blank lines.

which, I would interpret as the list items do not own their trailing blank 
lines but rather the list owns them. But that is not the behavior I am seeing. 
If I had to summarize the behavior I am seeing into words it would be:

> List items own their trailing blank lines unless they are both the final list 
> item and not the final element of a non-final list item.

Below I have how I reached this conclusion, but before diving into the weeds I 
want to point two things out:

1. I have hastily thrown together a tool to help rapidly visualize the 
ownership of nodes in org-mode's AST. Before this tool, I was manually running 
org-element-parse-buffer and using M-g c to jump around to the various indices 
to see where nodes started/ended. With this tool, I can paste in my org-mode 
source and get a tree showing the contents of each node and I can click on the 
nodes to highlight the relevant characters in the org-mode source. It is 
available at https://github.com/tomalexander/org_mode_ast_investigation_tool .

2. I have flattened my analysis for plain-text consumption over email below, 
but if you'd prefer the original org-mode version of this investigation it is 
available at 
https://github.com/tomalexander/org_mode_ast_investigation_tool/blob/cba1d1e988be230f3104f5f63dfaeaaf5cd0d280/notes/plain_list_ownership_notes.org
 .

And now, here is how I reached that conclusion:

*** Test case 1
```
1. foo

   1. bar

   2. baz

2. lorem

ipsum
```
| Plain List *Item*  | Owns trailing blank lines |
|+---|
| foo (includes bar baz) | Yes   |
| bar| Yes   |
| baz| Yes   |
| lorem  | No|

In this test case, we see that the only list item that doesn't own its trailing 
blank lines is "lorem", the final list item of the outer-most list.


*** Test case 2
We add "cat" as a paragraph at the end of foo which makes "baz" lose its 
trailing blank lines
```
1. foo

   1. bar

   2. baz

   cat

2. lorem

ipsum
```
| Plain List *Item* | Owns trailing blank lines |
|---+---|
| foo -> cat (includes bar baz) | Yes   |
| bar   | Yes   |
| baz   | No|
| lorem | No|

In isolation, this implies that the final plain list item does not own its 
trailing blank lines, which conflicts with "baz" from test 1.

New theory: List items own their trailing blank lines unless they are both the 
final list item and not the final element of a list item.

Adding why to the table:
| Plain List *Item* | Owns trailing blank lines | Why   
|
|---+---+---|
| foo -> cat (includes bar baz) | Yes   | Not the final 
list item   |
| bar   | Yes   | Not the final 
list item   |
| baz   | No| Final item of 
bar->baz and not the final element of "foo" |
| lorem | No| Final item of 
foo->lorem and not contained in a list item |


*** Test case 3
So if that theory is true, taking the entire (foo -> lorem) list from test 1 
and nesting it inside a list should coerce "lorem" to own its trailing blank 
lines since it would then be a final list item (of foo -> lorem) and the final 
element of the new list.
```
1. cat
   1. foo

  1. bar

  2. baz

   2. lorem

ipsum
```
| Plain List *Item*   | Owns trailing blank lines |
|-+---|
| cat (includes foo -> lorem) | No|
| foo (includes bar baz)  | Yes   |
| bar | Yes   |
| baz | Yes