Fwd: [PATCH] ox-publish.el: Speed up org-publish-cache-file-needs-publishing

[Gustav Wikström]

FYI. It seems we fixed the same issue half a year apart by chance..!

Maybe if we had a forge or smth this would have been solved already by Emily's 
patch beginning 2021. One can always wish, right? ;-)

Best
Gustav


From: Bastien 
Sent: Monday, September 27, 2021 10:48 PM
To: Gustav Wikström
Subject: Re: [PATCH] ox-publish.el: Speed up 
org-publish-cache-file-needs-publishing

Gustav Wikström  writes:

> Aha, hmm... I don't remember commiting any other persons code. I
> honestly think it's just me fixing the same thing thing that
> apparently already had a patch without me knowing it!

I'm glad to read this :) - thanks, then!

--
 Bastien

FYI Speedup of publish command in master

[Gustav Wikström]

Hi,

In a similar vein as the speedups of scanning for IDs that was made a while 
back, there now is a speedup for org publish as well. Soon to be pushed to 
code.orgmode.org (we're still using that, aren't we?) with commit aa0fa8c75.

The speedup is seen when scanning through files to decide if an existing cache 
can be used or not.

A note on the topic, fwiw. Loading an Org mode file using find-file is very 
slow. So when there is a reason for looking through the content of Org mode 
files, or any file for that matter, and there is no reason for triggering 
hooks, consider instead inserting the content of the file in a delay-mode-hooks 
macro within a temporary buffer (using the with-temp-buffer macro).

Example:
*Instead of*:
(find-file path-to-file)
;; do stuff
(...)
;; clean up state
(kill-buffer ...)

*Consider*:
(with-temp-buffer
  (delay-mode-hooks
(insert-file-contents path-to-file)
;; do stuff
(...)))

Best
Gustav

Re: Concerns about community contributor support

[Gustav Wikström]

Hi Tim,

Another data point from me. I agree with your concerns, although they are 
difficult to solve! Since we're talking about voluntary work and non-paid work. 
And maintenance can take a lot of time and effort.

This is not the first time this topic comes up on the list. Both you and me 
took part of the discussion after the 9.4.2 release, that hinted at similar 
topics [1]. And since Bastien has been in the process of handing over the 
maintenance role to someone else for quite some time now, it's not strange that 
the issue will continue to resurface until that process is done.

What I take away from this thread is mainly one thing: Another hand raised, 
eager to take on community work! Since you mentioned that you're interested. 
The more the merrier! Open source is tough though. It's very much a 
meritocracy. But by doing stuff that is considered to be good, and good stuff 
will come back to you.

Org mode isn't "finished". And I for one hope the community can continue to 
surprise with new, nice functionality. Even though some are perfectly happy 
where it is right now. In my world this is dual property of stability and 
extensibility is enabled by refactoring into a stable, small(er), less 
entangled and functional core while also encouraging extension packages/modes. 
Like org-roam and the likes.

A suggestion from me, fwiw, and if you're serious about getting more involved 
in the community, is to see if Bastien has some time to discuss this 
maintenance transition, and to see if there is anywhere where you can fit into 
that picture. But start small. ;)

I'm very interested in hearing more on how the thoughts are going in that 
department - and if there are others who have similar thoughts as you in terms 
of maybe putting more time into the project. But maybe don't see how help out, 
and with what.

Personally I'm also still hoping for "Org mode maintainer/advocate/whatever" to 
be something that someone out there can be occupied full time with. The value 
of the ideas and the software surely is there to warrant it. The question is 
only how to make it interesting enough for people to help out with funding!

[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2020-12/msg00511.html

Best,
Gustav

From: Emacs-orgmode  on behalf of 
Timothy 
Sent: Friday, April 16, 2021 20:43
To: org-mode-email
Subject: Concerns about community contributor support

Dear all,

Over the last few months I have felt an increasing level of concern over
the lack of response to patches. This email is rather long, but please,
bear with me. The goal is to start a discussion on the problems this
creates, and consider short and long-term solutions.

When both community and maintainer response to new patches is lacking,
many first-time contributors are actively dissuaded from contributing
again. Furthermore, each patch represents a considerable time investment
--- particularly if it's from an individual who is new to the mailing
list / patch workflow. Org-mode is not "done" and still requires the
support of long-term contributors to keep improving, anything that
discourages them from contributing back to the community needs to be
carefully understood and resolved if we want to continue harmoniously.

Take for example Jay Bosamiya's patch from September last year [1]. It
appears to be his first submission to this mailing list, and yet there
has been absolutely no response to it. There are currently 24 other
patches listed on the updates.orgmode.org which have seen no response
from this community, some of which are from first-time contributors.
There are 36 other patches with at least two replies, but yet to be
resolved. Bastien's updates.orgmode.org is fantastic in helping prevent
contributions slip through the cracks, but it is also highlighting the
lack of community response to a significant number of patches.

This mailing list was my first experience with an email+patch based
contribution workflow. Thankfully, I received prompt and friendly
feedback and was guided through the adjustments needed so it could be
merged in a timely manner. Should my patch have been similarly ignored,
I would have been quite disheartened; it is not an overstatement to say
I would likely have written off this mailing list and not tried again.

Simply put, this is not good enough. This does a disservice to those
that have dedicated time and effort to try and better this project only
to be ignored. Not rejected, not even acknowledged, nothing.

It is imperative that this community improves our response to
contributions for the long-term health of this project. Do not take me
to be a doomsayer; I have faith that Org is going to keep on improving
regardless. However, failing to welcome and encourage contributors has a
deleterious effect on the health of the project.

I do not blame the maintainers in the slightest. As Bastien brought up
in a recent worg discussion, as time goes on we find ourselves 

Re: Concerns about community contributor support

[Gustav Wikström]

You didn't ask me, but since I'm currently here and reading the list I might 
just give 2c to the topic.

My understanding is that a BNF-grammar is virtually impossible for Org. The org 
language is ambiguous and writing a context free grammar for it hence is not 
possible. For reference, see [1]. It deals with the topic, but with markdown 
instead of Org as the language. Both languages face the same issue.

[1]: https://roopc.net/posts/2014/markdown-cfg/ 

Kindly,
Gustav


From: Emacs-orgmode  on behalf of 
David Masterson 
Sent: Monday, April 19, 2021 23:43
To: Tim Cross
Cc: emacs-orgmode@gnu.org
Subject: Re: Concerns about community contributor support

Tim Cross  writes:

> I suspect the best model for moving forward is for new features and
> enhancements to be initially implemented as add on contribution packages
> rather than as extensions/enhancement to the core 'org-mode' package.
> Such packages, if found to be popular or useful to a large number of
> org-mode users could then be considered for inclusion as part of the
> main org-mode package. The nature of elisp makes this type of model very
> suitable as you can modify almost any aspect of org-mode via add on
> packages in a consistent and stable manner and avoid impacting the core
> functionality until the extension/enhancement has matured
> sufficiently.

What is the current status of having a BNF (or...?) parser for Org
files?  In particular, the parser rules that could then be adopted by
tools that use Org files on other systems (iPhone, Android, ...)?  Given
the availability of parser generators (bison...), this would be good for
breaking into smartphones where Emacs doesn't run.

--
David Masterson

Re: straight.el and org info pages?

[Gustav Wikström]

Hmm, I've had problems getting straight.el to install org-mode itself. That was 
on windows though. And that issue was fixed recently 
(https://github.com/raxod502/straight.el/issues/707)

I do recall some discussions on building docs long ago and that it wasn't 
possible due to some design issues. Not sure if that problem remains, or if I 
recall correctly even.

/G

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Greg Minshall 
Sent: Wednesday, March 24, 2021 3:56:36 AM
To: Gustav Wikström 
Cc: emacs-orgmode@gnu.org 
Subject: straight.el and org info pages?

Gustav,

> Straight.el is worth looking into for this. Has served me well for
> similar use cases.

have you (or anyone else?) had problems getting straight.el to build and
install the info pages for Org mode?

cheers, Greg

Re: trivial software engineering'ish question: switching org's

[Gustav Wikström]

Straight.el is worth looking into for this. Has served me well for similar use 
cases.

BR
Gustav

Get Outlook for iOS

From: Emacs-orgmode  on behalf of 
Tim Cross 
Sent: Sunday, March 21, 2021 10:35:00 PM
To: emacs-orgmode@gnu.org 
Subject: Re: trivial software engineering'ish question: switching org's


Greg Minshall  writes:

> hi.  i occasionally want to switch from the org package to a git
> version, then back again.  and, i want to avoid the dread "mixed
> installation".
>
> i'm wondering is there a way people do this other than simply
> installing/deleting the package version?
>

As I understand it, the critical part is when Emacs compiles the org
files to get the *.elc versions. Provided you do not have any org
functionality loaded during that compilation process, everything should
be OK. The 'mixed' versions problem arises because you go to compile a
different version and Emacs includes definitions already loaded from
another version, generating *.elc files with mixed versions.

Once org is compiled, the critical part is having the org version you
want show up first in the load-path, so the problem becomes one of just
managing the load-path entries appropriately. You could just ensure the
version you want is higher in the load-path or you could go the
'paranoid' route and have code which removes the version you don't want
from the load-path.

In the past, what I've done is have the git version of org in a specific
directory which I build with a separate process from the command line
using the make recipes in the repository - essentially just configuring
and running make. I then have some code in my init.el file which sets
that version at the start of my load-path when I want to run it and
which I comment out when I just want to run the version installed by
package.el. I also use the use-package macro to load my org
configuration and have two different blocks for that - one loading the
git repo version and one loading the org-plus-contrib  version I
normally use. I just comment out the one I don't want to use. I probably
could write some elisp to automate this, but to be honest, I switch
between org versions so rarely, commenting/uncommenting parts of my
init.el file is easy enough.

I don't do any of that at the moment as I've not needed to run from the
git repo since I switched to spacemacs and the spacemacs setup already
has the necessary workflow to ensure new versions of org are compiled in
a clean environment. Have not yet thought about how I will need to add
git based org when using spacemacs. Suspect All I'll need to do is
adjust the load-path as part of the init to reference the git sources
before any org functionality is loaded.

--
Tim Cross

Storing ID-links before first heading uses title as description

[Gustav Wikström]

Just committed to master (be2966abb5) 

Storing links to files using ~org-store-link~ (==) when
~org-id-link-to-org-use-id~ is not nil will now store the title as
description of the link, if available.  If no title exists it falls
back to the filename as before.

Best
Gustav

Re: Release Org 9.4.2

[Gustav Wikström]

Yes, that looks like a good option in my books.

Look for example at how logseq has it set up. 
(https://opencollective.com/logseq)

Best
Gustav

From: TEC 
Sent: Tuesday, December 29, 2020 10:42:50 AM
To: Gustav Wikström 
Cc: Eric S Fraga ; emacs-orgmode@gnu.org 

Subject: Re: Release Org 9.4.2


Gustav Wikström  writes:

> It is my believe that Org would benefit from a counter that would "tic
> up" even if no one collects the money. Like it or not, money has an
> effect. Better to allocate it to open source work than anything else! Or?

I've never used it, but could something like https://opencollective.com
be good for this?

Just a thought,

Timothy.

Re: New startup options

[Gustav Wikström]

Hi,

Added.

/Gustav


From: Emacs-orgmode  on behalf of 
Colin Baxter 
Sent: Monday, December 21, 2020 18:20
To: emacs-orgmode@gnu.org
Subject: New startup options


In ORG-NEWS there is the line

*** New startup options #+startup: showlevels

Not being a cs grad, the line is opaque to me. For example for 3
levels, does it mean

1. #+startup: show<3>levels
2. #+startup: show3levels
3. #+startup: show 3 levels

etc.

An example in ORG-NEWS would be useful - for me at least.


Best wishes,

Colin Baxter.

Re: Release Org 9.4.2

[Gustav Wikström]

> From: Eric S Fraga 
> Sent: Wednesday, December 16, 2020 15:50
> To: Gustav Wikström
> Cc: emacs-orgmode@gnu.org
> Subject: Re: Release Org 9.4.2
> 
> On Wednesday, 16 Dec 2020 at 14:11, Gustav Wikström wrote:
> > I for one wouldn't feel sorry if we (the world) could collect
> > resources to make working with Org mode a financially viable way of
> > life for someone.
> 
> I think that's already possible, or least it was with the old web site:
> there is/was a patreon link if I remember correctly.

True, which is good. But it's personal. To Bastien who already have
signaled that he's signing of. I frankly don't know where to put my
financial contributions right now. I've offered other contributors to
sign up for a similar patreon to recieve some contributions but they
have declined, due to what I understand as a pressure for delivery
(even though I've clarified that at least I have no expectations, just
want to contribute for the already done valuable work).

It is my believe that Org would benefit from a counter that would "tic
up" even if no one collects the money. Like it or not, money has an
effect. Better to allocate it to open source work than anything else! Or?

Anyhow, maybe the next maintainer(s?) who come after Bastien sees this
and realizes that there at the least is around 5-20$/m laying in wait
from this particular source.

Regards
Gustav

Re: Release Org 9.4.2

[Gustav Wikström]

> From: Eric S Fraga 
> Sent: Wednesday, December 16, 2020 14:49
> To: Gustav Wikström
> Cc: emacs-orgmode@gnu.org
> Subject: Re: Release Org 9.4.2
> 
> ...
> 
> Why cool?  What's cool is that so many contribute, in a wide range of
> ways, without some financial recompense!

I mean, yeah - it's already cool! No arguing there. Just saying that
"already cool" doesn't have to be "cool enough"! It's not binary. I
for one wouldn't feel sorry if we (the world) could collect resources
to make working with Org mode a financially viable way of life for
someone. Even though relying on contributions is an insecure way of
getting income, we all need an influx of resources (unless you've
already accumulated a wealth somehow). Having some who could wake up
in the morning, open the computer and do daytime work for Org would
make at least me smile. Ofc, with money comes other problems etc etc.

Kind regards
Gustav

Re: Release Org 9.4.2

[Gustav Wikström]

> From: Emacs-orgmode  on behalf 
> of TEC 
> Sent: Wednesday, December 16, 2020 08:14
> To: Bastien
> Cc: emacs-orgmode@gnu.org; Pankaj Jangid
> Subject: Re: Release Org 9.4.2
> 
> ...
> 
> I actually have a few thoughts on this.  I'm afraid that I don't think
> Org/Emacs are doing a good job of being accessible to younger
> individuals who have never used a ML / sent patches before (I should
> know, I'm one such individual, and the lack of familiarity was a
> significant deterrent). Whether a ML is a more efficient way of doing
> things or not ultimately doesn't matter in this regard, because it's
> simply not something I or many others are used to.
> 
> Just to be clear, I'm not advocating for getting rid of the ML and
> jumping on GitHub etc. :P

But to be fair, the collaboration features of GitHub surely would be a
BIG net positive if the goal is to attract contributions and gain a
bigger mindshare. That together with an open collective funding model
of some sort. Because let's be fair. This is a hobby project and it
would be cool if it could be something else at least for someone!

Gustav

Sv: New startup options, showlevels

[Gustav Wikström]

Hi,

I believe we still can change the name if that’s preferred. I'm not overly 
attached to the name I committed, although I fail to see the ugliness in it. 
And it's only in master since a few days. No biggie to change.

In my opinion it's important to see the names in their context. One context is 
the other available startup options.

Principles I tried to follow when choosing the names (Alt 0 below):
1) Keep the name as close to the intent as possible. It should be possible to 
understand it based on name alone, if possible. (I.e. avoid ambiguity)
2) Let the pattern for the name match with the pattern already existing for 
names in its surrounding context.
3) Keep it short, but prefer principle 1 and 2 to short.

The following options have been presented, where the first one is already in 
master:

Already in master (Alt 0): 
 `overview'Top-level headlines only.  
 `content' All headlines. 
 `show2levels' Headline levels 1-2.   
 `show3levels' Headline levels 1-3.   
 `show4levels' Headline levels 1-4.   
 `show5levels' Headline levels 1-5.   
 `showall' No folding on any entry.   
 `showeverything'  Show even drawer contents. 

Alt 1:
 `overview'Top-level headlines only.  
 `content' All headlines. 
 `show:2'  Headline levels 1-2.   
 `show:3'  Headline levels 1-3.   
 `show:4'  Headline levels 1-4.   
 `show:5'  Headline levels 1-5.   
 `showall' No folding on any entry.   
 `showeverything'  Show even drawer contents. 

Alt 2:
 `overview'Top-level headlines only.  
 `content' All headlines. 
 `levels:2'Headline levels 1-2.   
 `levels:3'Headline levels 1-3.   
 `levels:4'Headline levels 1-4.   
 `levels:5'Headline levels 1-5.   
 `showall' No folding on any entry.   
 `showeverything'  Show even drawer contents. 

Alt 3:
 `overview'Top-level headlines only.  
 `content' All headlines. 
 `content:2'   Headline levels 1-2.   
 `content:3'   Headline levels 1-3.   
 `content:4'   Headline levels 1-4.   
 `content:5'   Headline levels 1-5.   
 `showall' No folding on any entry.   
 `showeverything'  Show even drawer contents.

To me no option is perfect. There are incongruencies for all. Maybe a fourth 
option?

Alt 4:
`overview'Top-level headlines only.  
 `content' All headlines. 
 `2levels' Headline levels 1-2.   
 `3levels' Headline levels 1-3.   
 `4levels' Headline levels 1-4.   
 `5levels' Headline levels 1-5.   
 `showall' No folding on any entry.   
 `showeverything'  Show even drawer contents.

Since show in showall and showeverything seems to symbolize the unfolding of 
things.

That would be an improvement based on all three principles above, in that it 
removes the ambiguity of "show" (and how it relates to unfolding), makes the 
names shorter, and stays in line with the naming convention (i.e. no ':'). Not 
sure what the syntax says about names starting with numerals though.

Your call. Personally I'd prefer Alt 4 or the existing one.

/Gustav

> -Ursprungligt meddelande-
> Från: Eric S Fraga 
> Skickat: den 13 december 2020 10:49
> Till: Bastien 
> Kopia: Gustav Wikström ; emacs-orgmode@gnu.org
> Ämne: Re: New startup options, showlevels
> 
> On Saturday, 12 Dec 2020 at 18:54, Bastien wrote:
> > In this case, I think we could come up with better option names than
> > "show2levels", even if I don't have a better suggestion right now.
> 
> I agree.  I had started responding to Gustav when the original post
> appeared but then aborted my response.  I wonder whether something like
> levels:N or show:N or content:N is possible in a startup setting, akin
> to H:N in options?
> 
> I do have (org-content N) often in my file local variables in any case.
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-160-g7c8dce

Re: New startup options, showlevels

[Gustav Wikström]

Hi Bastien,

Thanks for the feedback.

Regarding naming, there may ofc be other possibilities. I considered spelling 
the level out but discarded that option due to its verbosity. I find this quite 
elegant actually. One might argue that it's not general enough, with 2 to 5 
hard-coded. But thinking of the use-case a bit more, I struggled to find a 
reason for more dynamics.

I don't object to the mailing list RFC method of doing things, but in this case 
it felt like a too small contribution for that trouble. Had there been a more 
streamlined pull-request workflow than the list I'm sure I'd have taken another 
decision. In this case the option was to just not do it due to time and energy 
constraints.

Nitpick noted, I'll try to care better with ending sentences moving forward!

Best
Gustav

From: Bastien 
Sent: Saturday, December 12, 2020 6:54:21 PM
To: Gustav Wikström 
Cc: emacs-orgmode@gnu.org 
Subject: Re: New startup options, showlevels

Hi Gustav,

Gustav Wikström  writes:

> Prompted by a question on StackOverflow,
> https://stackoverflow.com/questions/56536184/set-initial-visiblity-to-a-certain-level-in-org-mode,
> a few new options are added to the startup setting.

thanks -- in the future, I suggest discussing such additions on this
list before committing them.  In this case, I think we could come up
with better option names than "show2levels", even if I don't have a
better suggestion right now.

> Patch is applied to master as this is non-critical and it is
> communicated here and now for full transparency. See commit hash
> a71ac14e4,
> https://code.orgmode.org/bzg/org-mode/commit/a71ac14e46bb820abdbd2e6651c58179c50eb2fa

Mandatory nitpick: the log should contain proper sentences, ending
with a dot.  I'm mentioning this because your other commit has the
same small error.

> Hope these new options will be usable for some of you!

It sure is, thanks for taking care of this,

--
 Bastien

New startup options, showlevels

[Gustav Wikström]

Prompted by a question on StackOverflow, 
https://stackoverflow.com/questions/56536184/set-initial-visiblity-to-a-certain-level-in-org-mode,
 a few new options are added to the startup setting.

Patch is applied to master as this is non-critical and it is communicated here 
and now for full transparency. See commit hash a71ac14e4, 
https://code.orgmode.org/bzg/org-mode/commit/a71ac14e46bb820abdbd2e6651c58179c50eb2fa

Hope these new options will be usable for some of you!

Kind Regards
Gustav

Minor fix in org-shifttab on master

[Gustav Wikström]

Thought to mention that commit 9a154910ed is pushed to master,

It fixes an issue that drawers were opened when cycling global visibility using 
org-shifttab with a numeric argument (to only show outline levels up to that 
level).

/Gustav

Miscellaneous fix, make `org-goto-first-child' behave intuitively before first heading

[Gustav Wikström]

Hi again,

While fixing other issues in the code I found thing worth improving. Thought to 
just briefly mention it here as well, for full transparency. 
`org-goto-first-child' now understand what a child is when before first 
heading. This is inline with how other outline functions behave.

Since this change, and the other changes I've made, are relatively small I 
haven't taken them through any review process on the list. To save us all of 
some bureaucracy.
 
--- Commit dad436c60
Make `org-goto-first-child' behave intuitively before first heading

* lisp/org.el (org-goto-first-child): Make it understand outline
level 0 as well.  The function now behaves intuitively also before
first heading.

Regards
Gustav

Re: Default fold state of property drawers?

[Gustav Wikström]

Hi Kyle,

And thanks for a second pair of eyes on this!

I've pushed a patch to the repo that should fix it.

--- commit 8d7a9b4ce
Hide drawers before first headline properly when cycling visibility

* lisp/org.el (org--hide-drawers): New internal function consolidating
  logic from two places currently.

  (org-cycle-hide-drawers): Hide drawers before first headline at
  appropriate times.  Refactor to use new internal function
  `org--hide-drawers'.

  (org-hide-drawer-all): Refactor to use new internal function
  `org--hide-drawers'.

Kind regards
Gustav


From: Kyle Meyer 
Sent: Sunday, November 1, 2020 19:21
To: Gustav Wikström
Cc: emacs-orgmode@gnu.org
Subject: Re: Default fold state of property drawers?

Gustav Wikström writes:

> But maybe my issue rather lies in how the visibility toggling with
> S-TAB functions. The file property drawer is open in OVERVIEW and
> CONTENT but hidden in SHOW ALL. My intuition says that the
> file-property drawer should be closed for all toggle-states. Thoughts?

I agree.  While I've never used file-level property drawers, my
expectation would be for them to behave in the same way a heading's
would.

FWIW when you introduced file-level property drawers in 1bdff9f73 (Org
document property-drawers, 2019-05-26), it looks like cycling wasn't
considered and S-TAB would just ignore these drawers, preserving their
current state.

The cycling behavior you described above appears to come with 1aa095ccf
(Fix drawer invisibility, 2020-06-06).  I think that was in response to
<https://orgmode.org/list/87a71thwp2@fastmail.fm/T/#u>.  While I
haven't taken a close look, my guess is that the effect on file-level
property drawers was unintended, and that improvements here would be
welcomed.

Speedup of org-id-update-id-locations

[Gustav Wikström]

Hi, A patch is applied to master (commit 19d2f79a0) in order to speed up the 
rebuilding of ID locations.

Performance testing on my own setup shows a huge reduction in time to rebuild 
the cache. It goes from 168 seconds to 3 seconds. This performance increase is 
not to be expected for all operating systems. But for Windows (where Emacs has 
notorious problems working with the filesystem) this certainly helps!

The rest of this mail is the content of the commit message. 

Kind regards
Gustav

--- Commit 19d2f79a0 ---
Speedup of org-id-update-id-locations

Since this is about performance, a benchmark before this change, on a
set of 519 files with total size of 1500 kb gives the following result:

  519 files scanned, 504 files contains IDs, and 911 IDs found.
  (168.243948 38 2.0539493)

After the change the following result:

  519 files scanned, 504 files contains IDs, and 911 IDs found.
  (3.034806 3 0.164457622)

Benchmark done on a a Windows machine with no files previously loaded
into Emacs.

* lisp/org-id.el (org-id-update-id-locations): This function has
gotten a bit of back and forth changes in terms of performance.  One
year ago in 9865e6bd8 and then six months ago in 37a5020bb.
Unfortunately the latest speedup actually was a speed-down.  Speed is
not good again.

Re: Thoughts on the standardization of Org

[Gustav Wikström]

Hi,

I agree with your sentiment Asa. It would indeed be good to "standardize" Org. 
It's worth spending a few words here reasoning about what this standardization 
would mean. The text below are not specifically to you, Asa. But to the list. 
As food for thought on this topic. FWIW.

It's easy to be hesitant to standardization, thinking it will slow down and 
limit development. Personally I don't think Org mode is at risk of that. The 
issues come first with coordination between multiple parties. Especially if the 
visions, goals and perspectives of the parties differ. For Org mode this 
coordination should not be an issue, since no one as of now could dispute that 
Emacs Org mode implementation is the standard implementation. Few would also 
dispute which party has the leading role in defining the standard. (I.e. this 
community, with the maintainers as the "signing" bodies, define the standard. 
And it can be done either in the manual or in worg).

Issues with a standard hindering evolution of Emacs Org mode can be dealt with 
in the same way as the evolution of Org mode itself is handled. By versioning. 
Right now the Org mode version effectively declares what the DOM and syntax is. 
If we can extract that information from the source code. And assign a version 
to the DOM and syntax so they can be managed separate from the Emacs 
implementation, it's quite easy to evolve them as well. Although I think the 
syntax will evolve much less than the tool itself, since much of the changes 
aren't about changing the syntax but rather about changing the way Emacs 
augments and works with that syntax!

What it really boils down to, I think, is that there is a benefit of a clear 
document describing what an org mode document can consist of (DOM in your 
terminology I suppose?) and what the textual representation of that is (the 
syntax). Put a version number on that/those things that can be incremented as 
the community see fit. And it's done. Standard is defined.  No third party 
should need to sign it. It would be the "Emacs Org mode" community 
standardization of the Org mode object model and textual syntax and document 
format. And that in itself should be more than enough to get the ".org" file 
extension globally approved. And help parser developers and other tool 
developers to support the format. AND help further develop Emacs 
implementation. Discussions regarding composing these documents are already 
started in the MIME-type threads. In my humble opinion there is not much else 
needed to get this "standardization" done.

Nicolas has started the journey in an excellent way with the Org element 
documentation and source code library in my opinion. Hats off to him! Anyone 
willing of following in his footstep gets another hat off. I'm sure it will be 
of great benefit to all Org mode users out there!

Kind regards
Gustav

> -Ursprungligt meddelande-
> Från: Emacs-orgmode  För Asa
> Zeren
> Skickat: den 1 november 2020 01:22
> Till: emacs-orgmode@gnu.org
> Ämne: Thoughts on the standardization of Org
> 
> Hi,
> 
> Even though I am new to the org-mode community, I would like to share
> some thoughts on the specification of org-mode, especially since I
> have seen some recent discussion of it in relation to registering org
> as a MIME type.
> 
> First, I would like to repeat the importance of developing standards
> for org-mode. If we want to expand the influence of org, tooling must
> expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
> convince the entire world to use Emacs and (b) org-mode should be
> integrated into tooling unrelated to text editing, and is outside of
> the Emacs-Lisp environment. Without additional org implementations,
> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.
> 
> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.
> 
> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
>  - Org Syntax
>  - Org Standard Environments
> 
> Before we get to that, a brief note on /how/ I think that org should
> be specified. I think that org should be specified in terms of an
> /environment/ that defines the properties, etc. that can be used in a
> document. For instance, the org standard would say something to the
> effect of "An environment may specify block bounding keywords that may
> be used like #+\n...#+. and the environment would specify
> "begin_src and end_src are a pair of block bounding keyword that
> indicates a source code block." This is for two reasons. First, this
> allows for development of org tool features independent of the
> standard. Second, this separates the individual features of org mode
> from the 

Re: Default fold state of property drawers?

[Gustav Wikström]

Hmm, maybe.. Didn't consider that one.

But maybe my issue rather lies in how the visibility toggling with S-TAB 
functions. The file property drawer is open in OVERVIEW and CONTENT but hidden 
in SHOW ALL. My intuition says that the file-property drawer should be closed 
for all toggle-states. Thoughts?

/Gustav


From: Kyle Meyer 
Sent: Thursday, October 29, 2020 03:45
To: Gustav Wikström
Cc: emacs-orgmode@gnu.org
Subject: Re: Default fold state of property drawers?

Gustav Wikström writes:

> Hi,
>
> This may be a stupid question but I didn't find anything conclusive in
> the manual or online after a (albeit fairly quick) scan.
>
> Is there currently a way to declare that property drawers (and the
> file level property drawer in particular) should be hidden when
> opening a new Org file, no matter the startup keyword setting? I.e is
> it possible to control the open/closed state of the property drawer
> separate from the outline folding?

I may be missing something too, but I'm not aware of a way to do this
that's distinct from setting org-startup-folded and friends.  But
doesn't org-startup-folded's showall/showeverything distinction cover
your usecase?

Default fold state of property drawers?

[Gustav Wikström]

Hi,

This may be a stupid question but I didn't find anything conclusive in the 
manual or online after a (albeit fairly quick) scan.

Is there currently a way to declare that property drawers (and the file level 
property drawer in particular) should be hidden when opening a new Org file, no 
matter the startup keyword setting? I.e is it possible to control the 
open/closed state of the property drawer separate from the outline folding?

Thanks and best regards
Gustav

Re: Any reason not to generate my own custom ID value (NOT CUSTOM_ID)?

[Gustav Wikström]

Hi TRS-80,

Your approach should work just fine. So fine, in fact, that it's already kind 
of built in! Configure org-id-method and set it to 'ts and you'll get 
timestamps as ID instead of uuid.

I do believe the manual lacks a description for this. Not entirely sure though, 
and can't check atm. But the configuration is there none the less, and 
supported.

Kind regards
Gustav

From: Emacs-orgmode  on behalf of 
TRS-80 
Sent: Thursday, September 10, 2020 10:02:45 PM
To: emacs-orgmode@gnu.org 
Subject: Any reason not to generate my own custom ID value (NOT CUSTOM_ID)?

First, I want to express my sincere and heart-felt gratitude to Carsten
(and other contributors) for making and sharing this wonderful piece of
software.  I have come to refer to it as "one of the gateway drugs to
Emacs" (the other being Magit, IMHO).  It was certainly one of (if not
/the/) main reason(s) why I started using Emacs initially.

I could in fact gush all day, however people are busy, so, on to the
main issue...  :)

It seems to me that there is nothing really stopping me from inserting
whatever value I like for value of "ID" Property.  Based on brief
experimentation, org-store-link and org-insert-link seem to happily
accept whatever value is already there (which I entered manually, for
testing purposes).

However I seem to recall reading some warning somewhere about this,
although I cannot seem to find it right now.

What I would like to do, is generate my own ID values in a more human
readable format, something "ISO-like" for example "2020-09-10-1433" (as
opposed to the default "uuid" method).  These sort of ID are still
"Unique" (well, within my own system, anyway) as long as I am not
generating them more often than once per minute[0].  And they have the
advantages of being shorter, human readable, and meaningful.

Even when org-id-link-to-org-use-id and org-id-track-globally are both
set to "t", org-id seems happy to insert my "ISO-like" ID right into the
hash table and org-id-locations-file.

I do need the "across files" functionality.  My understanding is that
this is main difference between ID and CUSTOM_ID (the latter being local
only to the file).  Unless I am misunderstanding?

So, what am I missing here?  Any reason(s) /not/ to use my own custom ID
value?

In addition to the general case, one particular area I am unsure about
(as I have yet to get it working) is how this all works out with HTML
export, as that is something I also wanted to get working at some point.

I tried studying some of the related sources (as well as mailing list
archive), but could not seem to reach a conclusive answer.  I was hoping
that some more knowledgeable people could confirm
whether this is a really bad idea, or not.  Any feedback would be
greatly appreciated!

Cheers!

TRS-80

[0] It could easily be extended to second (or further) resolution, if
needed.  For me, minute resolution will be fine.

Re: Shower thought: submit an IETF RFC to register Org as a MIME type

[Gustav Wikström]

That would be very nice indeed.

/Gustav

From: Emacs-orgmode  on behalf of 
TEC 
Sent: Friday, September 4, 2020 4:44:50 PM
To: org-mode-email 
Subject: Shower thought: submit an IETF RFC to register Org as a MIME type


Hi everyone,

Prompted by the fact that Markdown is registered as a MIME type
(RFC7763) and perusing the MIME registration procedure (RFC6838),
I wonder if it may be possible to register Org as a MIME type?

There are a few parts of RFC6838 in particular which give me hope,
such
as:
> [§4.9] universal support and implementation of a media type are
> NOT a
> requirement for registration.

I'm guessing the main barrier wold be a the lack of a published
specification --- I'm guessing a complete version of
https://orgmode.org/worg/dev/org-syntax.html published under the
main
site (i.e. https://orgmode.org/standard.html) would be required.

Looking for other uses of the .org extension, there doesn't seem
to be
much. The main result is from "Lotus Organiser", which seems to be
a
(discontinued) PIM from IBM which used .org as its file type in
the 1992
release. Other than that it seems that Yamaha and SoundBlaster
have used
it as an extension for organ audio samples.

If it does seem possible to have text/org formally added as a mime
type,
I would love to push this.

Please let me know what you think.

All the best,

Timothy.

[ANN] Org-collection - (experimental) global minor mode

[Gustav Wikström]

Hi there,

Just wanted to mention that there’s an embryo at 
https://github.com/Whil-/org-collection for de-globalizing your org mode setup 
into collections. For those interested in that.

This relates to: 
https://lists.gnu.org/archive/html/emacs-orgmode/2019-12/msg00215.html

It’s hasn’t turned out exactly like I envisioned it. Not yet anyways. But it 
has allowed me to define multiple sets of Org mode configurations that I can 
work with in parallel. I.e. multiple agenda definitions, TODO-keyword 
definitions, etc. etc.

Documentation is sparse but code is fortunately quite concise and hopefully 
readable. I’m not sure where exactly to take this next, but thought to share it 
with the list anyhow.

Caveat: I’ve used hooks that are quite new and only available in Emacs 27.1. 
Luckily that version was “officially” released a few days ago.

Final remark: I’m writing embryo above, but this is currently working in my 
local setup(s). So maybe it’s a bit more than an embryo.

Kind regards, and take care
Gustav

RE: Website revamp?

[Gustav Wikström]

Hi TEC,

Just wanted to add one datapoint here. I think this effort your putting down is 
commendable and deserves many +1's.

Reading about something ofc happen on devices other than where the software in 
the end is installed. Thus, I agree fully with what you write below!

Your draft is already a big improvement and I, for one, hope it can go into 
"release" as soon as possible. Even if it's not "perfect". Because nothing is, 
especially not regarding design, where facts are few and opinions are many.

Thank you.

/Gustav 

> -Original Message-
> From: Emacs-orgmode  On Behalf
> Of TEC
> Sent: den 3 augusti 2020 07:11
> To: Colin Baxter 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: Website revamp?
> 
> 
> Colin Baxter  writes:
> 
> >> TEC   writes:
> >> - The site is now more mobile friendly, the navbar now has a
> >
> > Why? How many users are installing org-mode on their 'phones - smart
> > or otherwise?
> 
> Zero, I expect :P
> 
> However, I don't think that's the question we should be asking.
> Consider someone tweeting about org-mode, the question then is:
>   how many people use twitter on their phones?
> 
> Likewise, if someone wants to share a org-mode with a friend, and sends them
> a link - whether it be by email, sms, whatsapp, messenger, or telegram:
> what's the chance that they'll take a peek oh their phones?
> 
> Of course we don't exactly have any hard answers to these questions, but I'd
> wager the answer is enough to warrant a little bit of effort to make the
> mobile experience not-garbage. To see how orgmode.org currently appears, see
> https://i.imgur.com/XPFfBaB.png
> 
> IMO that's downright off-putting --- and I say that intending no criticism
> to Carsten or Bastien, in 2012 mobile design simple wasn't a priority.
> 
> Moreover, we can improve the mobile experience with no compromise to the
> desktop
> --- so why wouldn't we? :D
> 
> I hope this fleshes out my motivation for improving the mobile experience :)
> 
> Timothy.

RE: FWD: Org-Babel Support for Powershell

[Gustav Wikström]

Hi,

Org babel support for Powershell is a great idea!

Powershell is very ...*ahem*... powerful, and it would likely benefit many 
users if it was usable and executable from within Org mode!

> -Original Message-
> From: Emacs-orgmode  On Behalf
> Of Russell Adams
> Sent: den 6 juni 2020 12:43
> To: emacs-orgmode@gnu.org
> Subject: Re: FWD: Org-Babel Support for Powershell
> 
> On Fri, Jun 05, 2020 at 11:32:45PM +0200, stanislaw_lem--- via General
> discussions about Org-mode. wrote:
> > With Powershell 7 we got now a cross-platform scripting language for Windows
> > and Linux OS.
> 
> Can you provide an example script you would run, and the example output?
> Basically a minimum working example, except the part where Babel would run it
> for you?

I know the question wasn't for me. But since I'm answering to your mail I can 
just as well provide an example anyhow:

#+begin_src powershell
  2+2
#+end_src

#+RESULTS:
: 4

> I think Org-Babel working with many languages is a noble goal, but I expect
> there would be difficulty getting Powershell support added to the core
> distribution. It might be limited to a separate package as it is very specific
> to a Windows only non-free language from a vendor with a long and sordid
> history
> of anti-OSS practices.

There should be zero issues in including it into core. Powershell is 
MIT-licensed and works across all platforms.

> My opinion is I would never consider using Powershell on Linux, despite your
> positive impression.

And I say the opposite. I would certainly use it!

/of topic
It's safe to say the free software movement has gained a strong footing in the 
software world. If it's "enemies" from the past are starting to walk in the 
same direction it would serve the free software advocates well to embrace and 
further encourage those steps. It would in my opinion be signs of maturity and 
true, good intentions. Judge not from where one comes, but for where one is 
heading.

Kind regards
Gustav

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 14 maj 2020 23:21
> To: Gustav Wikström 
> Cc: Matthew Lundin ; Org Mode List 
> Subject: Re: [Bug] org-store-link should not insert a document level ID
> property
> 
> [...]
> 
> Regenerating ".org-id-locations" is not magical. It looks into
> a predefined set of files. If the renaming moves the file out of this
> set, there is no way it can associate again the ID to that file.

The same argument could be made for ID on a heading, if that heading is refiled 
into a location outside of that predefined set of files.

> IOW, I wouldn't trust an ID more than a file name.

But I would. Not entirely unlikely for files to be renamed within my folders. 
Using ID's helps for that. Be it to a file or a heading. But of course, YMMW.

Kind regards
Gustav

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi Matt,

> -Original Message-
> From: Matthew Lundin 
> Sent: den 7 maj 2020 23:41
> To: Gustav Wikström ; Org Mode List 
> Subject: RE: [Bug] org-store-link should not insert a document level ID
> property
> 
> [...]
>
> What I was thinking of in terms of configuration is being able to
> preserve path-based links (instead of IDs) if creating a link above the
> first headline. This is the behavior that existed in the past when
> org-id-link-to-org-use-id was set to t or
> 'create-if-interactive-and-no-custom-id.
> 
> I've attached a patch that implements this. However, I also know that we
> are trying to avoid configuration/feature creep in Org, so I'm
> submitting this here for comments. Is this a configuration other people
> want?

This is a very specific customization. If it was to be implemented I would 
probably rename it a bit for its purpose to be more clear. My personal opinion 
of course. Name could be something in the line of 
"org-id-inhibit-id-before-first-heading" and make it clear in the docstring 
that the customization really only takes effect if org-id-link-to-org-use-id is 
not nil. That way one won't wonder as much about the interplay between the 
functions.

What say you? :)

Kind regards
Gustav

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 13 maj 2020 18:45
> To: Matthew Lundin 
> Cc: Gustav Wikström ; Org Mode List 
> Subject: Re: [Bug] org-store-link should not insert a document level ID
> property
> 
> Matthew Lundin  writes:
> 
> > Nicolas Goaziou  writes:
> >>
> >> AFAIK, ID is associated to a file name, and possibly a location in it.
> >> In this case, the ID is strictly equivalent to the file name, so why
> >> bother?
> >
> > I'm not sure I understand the question. Are you asking: Why bother
> > generating IDs at the top level of a file (which was the change Gustav
> > introduced)? Or why bother turning off that behavior? I can't address
> > the former question but I will address the latter.
> 
> Sorry for not being clear. This was the first question. I don't
> understand why we are generating an ID for the whole file.

I see some purposes. They maybe aren't for everyone, but they surely are there. 
1) Attachments for the whole file without setting the DIR property.
2) Link targets, making the file name irrelevant for the links.

> > The main reason is that I find these IDs redundant and visually
> > distracting. I can see how file IDs would be useful if one is
> > constantly renaming files (or perhaps writing custom functions that
> > convert files to entries and vice versa).
> 
> IIUC, currently, renaming the file breaks the association between the ID
> and the file name. IOW, the ID is useless if you rename the file.

As long as org-id-track-globally isn't set to nil I don't think what you write 
is true. And even if renaming the file would break the link it's just 
momentarily since regenerating the .org-id-locations file should make the link 
work again.

Regards
Gustav

RE: wip-cite status question and feedback

[Gustav Wikström]

Hi,

I'm curious. So take this for what it is; I.e. curiosity. What /exactly/ is 
meant with a citation here? Is it a new general concept in Org mode, or is it 
something more narrow, as an extension for some specific third party software? 
Would I be able to use it without that third party software? What would the 
content of a citation be? Is it a link to some source plus annotations and 
formatting? Is it only the link? Is it also the formatting? Is it something 
else entirely? I'm wondering since Org mode has existing facilities for much of 
this already. But maybe not packaged together for citations just yet? Is there 
any purpose of thinking of citations as a wrapper for already existing 
functionality? Or could the links-syntax be extended with more properties and 
auxiliary functionality to fulfill the need for citations?

Maybe these questions have been discussed already. Sorry in that case, it was a 
long thread and I admit to not having read everything. I'm simply wondering 
since there may be reason for reusing and extending existing functionality 
before creating something new. Hence the question of what exactly a citation 
consists of above. Because Org mode can already provide links to external 
content. And, to me at least, a citation is simply that: a link. A link which 
is the key to the information that is to be marked as the source of the 
citation. Auxiliary functionality on top of that could provide facilities to 
collect all such links into bibliographies based on link-backend and properties 
of that link. Configurations (emacs config or org mode properties, 
[info:org#Property Syntax]) could instruct on formatting.

But as I said in the beginning. Take this as curiosity. And take it as 
questions for someone not very much into the academic world at this time, not 
having to bother with typesetting, formatting and latex related issues.

Kind regards
Gustav

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of Nicolas Goaziou
> Sent: den 8 april 2020 11:32
> To: Bruce D'Arcus 
> Cc: Joost Kremers ; emacs-orgmode@gnu.org;
> András Simonyi ; John Kitchin
> 
> Subject: Re: wip-cite status question and feedback
> 
> Hello,
> 
> "Bruce D'Arcus"  writes:
> 
> > Note that in CSL processors, the locators are meaningful key-values,
> > basically; not plain text strings.
> 
> OK, but it is enough for Org to feed a CSL processor with, e.g.,
> 
>   key-> "@doe99"
>   prefix -> "see "
>   suffix -> ", pp. 33-35"
> 
> Then CSL processor does its job to extract whatever information it
> needs. Am I right?
> 
> > While I of course can't speak for John, I would hope and expect org
> > ref to support the new syntax.
> >
> > I would hope and expect the same of other packages (like ebib) that
> > use their own custom syntax.
> 
> I hope so, too. But it would help if developers could chime in and tell
> what API Org should provide. This is particularly important since Org
> will only provide the API, without any specialized implementation. More
> on this at the end of the message.
> 
> > 1. just have a super simple citation export formatter, with (per
> >above) room to configure an external one
> 
> Sure. Org should provide default export for citations in LaTeX, ASCII,
> HTML, Texinfo and ODT. Suggestions are more than welcome.
> 
> > 2. while I don't know its status, include citeproc-org in org and
> > emacs
> 
> I think citeproc-el should ship with Emacs, but I cannot speak for
> Emacs; it would be nice to ask Emacs developers about it.
> 
> > I'd say if citeproc-org is far along and free of substantial bugs, 2
> > might make sense. I am, of course, biased, but CSL's been widely
> > deployed in the wild for more than a decade, and there are thousands
> > of available styles.
> >
> > Otherwise, and more likely, 1 would be the best path; users get basic
> > output formatting for citations, but then can plug in more elaborate
> > tools (citeproc-org, citeproc-hs*, etc.) if they need them.
> 
> Option 2 makes a lot of sense if citeproc-el is integrated with Emacs.
> Until them, I agree that option 1 is the way to go at the moment.
> 
> > WDY about that?
> 
> Sounds like a plan. Let me summarize it a bit :
> 
> 1. [ ] Finalize Org citation syntax. It is mostly good, but we need to
>decide about the following points:
> 
>- [ ] Should it include both "(cite):" and "cite:", i.e., does it
>  make sense to provide a (very limited) style specification piece
>  wise?
> 
>- [ ] Should it include /global/ prefix and suffix?
> 
>- [ ] Should we keep the short specification, i.e., "[@key]"?
> 
>- [ ] What kind of markup do we allow in a citation? At the moment
>  the exhaustive list is: bold, code, entity, italic, latex-
> fragment,
>  strike-through, subscript, superscript, underline, verbatim and
>  line breaks.
> 
> 2. [ ] Decide about API Org should provide for it to be useful. Here
> are
>some low hanging fruits:
> 
>- [ ] List all ".bib" files 

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

Just wanted to say that the patch is applied on master. It's solving the 
problem. Regarding user-configuration that can be added later if needed. It's 
not wrong right now per se.

Cheers!
Gustav

> -Original Message-
> From: Gustav Wikström
> Sent: den 5 april 2020 17:50
> To: Matt Lundin ; Org Mode List 
> Subject: RE: [Bug] org-store-link should not insert a document level ID
> property
> 
> Hi again,
> 
> Patch is attached. It's not applied yet as it doesn't include anything about
> user-configuration yet. @Matt Lundin, care to elaborate what you had in mind
> in terms of that?
> 
> With this patch a link before first headline is stored with the filename (no
> path) as description. Following the link does what you'd expect. Tests ran
> fine with the patch applied.
> 
> Regards
> Gustav
> 
> > Hi,
> >
> > > -Original Message-
> > > From: Matt Lundin 
> > > Sent: den 5 april 2020 00:13
> > > To: Org Mode List 
> > > Cc: Gustav Wikström 
> > > Subject: [Bug] org-store-link should not insert a document level ID
> property
> > >
> > > The introduction of document-level property drawers (commit
> > > 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> > > inconsistencies in the behavior of org-id and org-store-link.
> > >
> > > If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> > > calling org-store-link above the first headline in an org file will
> > > insert a PROPERTY drawer and an ID at top of the file, so that the file
> > > (call it "~/test.org") looks like this:
> > >
> > > --8<---cut here---start->8---
> > > :PROPERTIES:
> > > :ID:   1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> > > :END:
> > >
> > > * First headline
> > > --8<---cut here---end--->8---
> > >
> > > However, the link that Org actually stores is "[[file:~/test.org]]", so
> > > the ID is irrelevant.
> > >
> > > In addition, a link to a document-level ID does not work. Following
> > > "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> > >
> > > user-error: Before first headline at position 14 in buffer test.org
> > >
> > > So either:
> > >
> > > 1. org-id and org-store-link/org-open-link should support document level
> > >ids in a user-configurable way, or
> > > 2. org-id-get-create should detect whether it is above the first heading
> > >and should not create an id
> > >
> > > Option #2 would obviously be the easier fix.
> > >
> > > Gustav: were IDs within the scope of your initial thinking about
> > > document level properties? Or are these largely irrelevant?
> >
> > Yes, they were in scope and ID should mean something also before first
> > headline. I'll have a look to see what can be done. Thanks for finding
> this!
> >
> > >
> > > Best,
> > > Matt

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi again,

Patch is attached. It's not applied yet as it doesn't include anything about 
user-configuration yet. @Matt Lundin, care to elaborate what you had in mind in 
terms of that?

With this patch a link before first headline is stored with the filename (no 
path) as description. Following the link does what you'd expect. Tests ran fine 
with the patch applied.

Regards
Gustav

> Hi,
> 
> > -Original Message-
> > From: Matt Lundin 
> > Sent: den 5 april 2020 00:13
> > To: Org Mode List 
> > Cc: Gustav Wikström 
> > Subject: [Bug] org-store-link should not insert a document level ID property
> >
> > The introduction of document-level property drawers (commit
> > 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> > inconsistencies in the behavior of org-id and org-store-link.
> >
> > If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> > calling org-store-link above the first headline in an org file will
> > insert a PROPERTY drawer and an ID at top of the file, so that the file
> > (call it "~/test.org") looks like this:
> >
> > --8<---cut here---start->8---
> > :PROPERTIES:
> > :ID:   1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> > :END:
> >
> > * First headline
> > --8<---cut here---end--->8---
> >
> > However, the link that Org actually stores is "[[file:~/test.org]]", so
> > the ID is irrelevant.
> >
> > In addition, a link to a document-level ID does not work. Following
> > "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> >
> > user-error: Before first headline at position 14 in buffer test.org
> >
> > So either:
> >
> > 1. org-id and org-store-link/org-open-link should support document level
> >ids in a user-configurable way, or
> > 2. org-id-get-create should detect whether it is above the first heading
> >and should not create an id
> >
> > Option #2 would obviously be the easier fix.
> >
> > Gustav: were IDs within the scope of your initial thinking about
> > document level properties? Or are these largely irrelevant?
> 
> Yes, they were in scope and ID should mean something also before first
> headline. I'll have a look to see what can be done. Thanks for finding this!
> 
> >
> > Best,
> > Matt


0001-Allow-storing-and-following-ID-links-before-first-he.patch
Description: 0001-Allow-storing-and-following-ID-links-before-first-he.patch

RE: [Bug] org-store-link should not insert a document level ID property

[Gustav Wikström]

Hi,

> -Original Message-
> From: Matt Lundin 
> Sent: den 5 april 2020 00:13
> To: Org Mode List 
> Cc: Gustav Wikström 
> Subject: [Bug] org-store-link should not insert a document level ID property
> 
> The introduction of document-level property drawers (commit
> 1bdff9f73dc1e7ff625a90e3e61350bdea99f29c from May 2019) introduced
> inconsistencies in the behavior of org-id and org-store-link.
> 
> If org-id-link-to-org-use-id is set to t or 'create-if-interactive,
> calling org-store-link above the first headline in an org file will
> insert a PROPERTY drawer and an ID at top of the file, so that the file
> (call it "~/test.org") looks like this:
> 
> --8<---cut here---start->8---
> :PROPERTIES:
> :ID:   1f43e860-9e7b-4c8f-82b9-6ed3352e589f
> :END:
> 
> * First headline
> --8<---cut here---end--->8---
> 
> However, the link that Org actually stores is "[[file:~/test.org]]", so
> the ID is irrelevant.
> 
> In addition, a link to a document-level ID does not work. Following
> "[[id:1f43e860-9e7b-4c8f-82b9-6ed3352e589f]]" results in this error:
> 
> user-error: Before first headline at position 14 in buffer test.org
> 
> So either:
> 
> 1. org-id and org-store-link/org-open-link should support document level
>ids in a user-configurable way, or
> 2. org-id-get-create should detect whether it is above the first heading
>and should not create an id
> 
> Option #2 would obviously be the easier fix.
> 
> Gustav: were IDs within the scope of your initial thinking about
> document level properties? Or are these largely irrelevant?

Yes, they were in scope and ID should mean something also before first 
headline. I'll have a look to see what can be done. Thanks for finding this!

> 
> Best,
> Matt

RE: [ANN] public-inbox archive of the mailing list

[Gustav Wikström]

Hi,

This is useful, thanks!

> -Original Message-
> From: Emacs-orgmode  On Behalf
> Of Kyle Meyer
> Sent: den 3 april 2020 06:56
> To: emacs-orgmode@gnu.org
> Subject: [ANN] public-inbox archive of the mailing list
> 
> I've set up a public-inbox [^1] archive of the mailing list at
> 
> https://yhetil.org/orgmode/
> 
> It should be fairly complete, going back to the earliest non-test email
> listed in the lists.gnu.org archive [^2].  If you notice that a message
> is missing, please first check whether you can find it with a
> +message-id: search against the gnu.org archive.  If you can't find
> it there, it probably didn't make it to the list; if you can, please let
> me know.
> 
> New messages should hit the archive within 15 minutes or so.
> 
> I should say more about why I think public-inbox is great [^3], but I
> doubt I could do it justice while keeping this email short.  Anyway, if
> you'd like to have a look, perhaps start here:
> 
>   (eww "https://yhetil.org/orgmode/_/text/help/;)
>   (eww "https://yhetil.org/orgmode/871uc8mtpy@poukram.net/;)
> 
> 
> [^1]: https://public-inbox.org/README
> [^2]: https://lists.gnu.org/archive/html/emacs-orgmode/
> [^3]: I'm not alone :>
>   https://public-
> inbox.org/meta/20190912135606.GE16671@pure.paranoia.local/
>   https://public-
> inbox.org/git/nycvar.qro.7.76.6.1808271552580...@tvgsbejvaqbjf.bet/

RE: File Scoped Properties?

[Gustav Wikström]

Hi Tim,

First you must make sure to allow property inheritance. That can be done by 
setting org-attach-use-inheritance to t. Then add the DIR property to a 
property block for the file, or using a property keyword. I.e. either this:

#+begin_src org
  :PROPERTIES:
  :DIR:  ~/.foo/data
  :END:
#+end_src

#+begin_src org
  ,#+PROPERTY DIR ~/.foo/data
#+end_src

Running org-attach with option s will give you a prompt that will set DIR for 
you. If you invoke that before first headline it should result in a DIR 
property that applies for the whole buffer (when org-attach-use-inheritance is 
set to t that is). Note that property blocks before first headline is a new 
feature for version 9.4 and currently only exist in the master branch.

Regards
Gustav

From: Emacs-orgmode  On Behalf Of 
Tim Visher
Sent: den 5 mars 2020 16:28
To: Emacs Org Mode mailing list 
Subject: File Scoped Properties?

Hello,

I'm trying to get org-attach to use a different data directory for a particular 
file.

My understanding is that this is controlled by `org-attach-id-dir` by default 
but can be overridden at the file or entry level by use of the `DIR` property. 
I can successfully override it at the entry level by adding `DIR` to the 
entries properties but I can't figure out how to set it for all entries in the 
file.

I tried:

```
#+DIR: ~/.foo/data
```

as the first line of the file.

I _am_ able to get it to work by adding a file local variable like

```
# Local Variables:
# org-attach-id-dir: "~/.foo/data"
# End:
```

but then whenever I open the file it tells me it's possibly not safe to set 
that.

Any ideas?

--

In Christ,

Timmy V.

https://blog.twonegatives.com
https://five.sentenc.es

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hello,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 26 februari 2020 16:23
> To: Gustav Wikström 
> Cc: Bastien ; emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > Expanding attachment-links in the buffer makes the link type no longer
> > be attachment.
> 
> What makes you think that? The link type is preserved during the
> substitution, according to the (concat "attachment" ...).

What I meant is that the content of the link, even though mentioned to be 
"attachment" cannot be reasoned about and worked with in the same way as an 
unexpanded attachment link. Thus, it's not the same type of link any longer. 
It's effectively a "file" link when expanded. But this discussion doesn't carry 
much meaning any longer I suppose, given the agreement to apply the patch.

> > I would prefer if we explicitly set the link type to
> > files here instead. Storing intermediate state in an attachment link
> > types makes less sense imo.
> >
> > One issue with the current way it's done is that images are treated
> > differently between attachment links and file links. For HTML exports,
> > file links are wrapped in a div with class figure where expanded
> > attachment links are not.
> 
> It would be nice to investigate why this happens.

Surely custom code for file links. Haven't dug much deeper since the issue is 
solved by the attached patch.

> > Letting org-attach-expand-links do the full
> > transform to file links would solve that issue. That also
> > means :export is not needed for org-link-set-parameters.
> >
> > Patch attached if you agree to this.
> 
> Of course! This is what I suggested, like, 6 weeks ago (although, at
> this time, I was thinking about having the substitution in "ox.el").

Well yes... I had other ideas at that point and would rather see attachment 
links as something of its own. But we're not in that place now, and the 
decision was to treat link types that can be expanded into file links just like 
that. You've kindly fixed the code so Org mode can do that now, something I'm 
sure I couldn't have done myself anyhow. So let's do it as you suggested from 
the beginning. I'll apply the patch then, unless already taken care of by you 
or Bastien! :)

Kindly
Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi Bastien and Nicolas,

> -Original Message-
> From: Bastien 
> Sent: den 22 februari 2020 14:32
> To: Nicolas Goaziou 
> Cc: Gustav Wikström ; emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> [...]
>
> Nicolas Goaziou  writes:
> 
> > I'd like to hear from Gustav, too, since this affects the changes he
> > introduced. If you are in a hurry, we can also apply it. Up to you.
> 
> I just applied it - Gustav let us know if it fits your needs and if it
> does not introduce regressions.

It does work. Thanks. One issue though, with org-attach-expand-links:

+(defun org-attach-expand-links (_)
+  "Expand links in current buffer.
+It is meant to be added to `org-export-before-parsing-hook'."
+  (save-excursion
+(while (re-search-forward "attachment:" nil t)
+  (let ((link (org-element-context)))
+   (when (and (eq 'link (org-element-type link))
+  (string-equal "attachment"
+(org-element-property :type link)))
+ (let* ((description (and (org-element-property :contents-begin link)
+  (buffer-substring-no-properties
+   (org-element-property :contents-begin link)
+   (org-element-property :contents-end link
+(file (org-element-property :path link))
+(new-link (org-link-make-string
+   (concat "attachment:" (org-attach-expand file))
+   description)))
+   (goto-char (org-element-property :end link))
+   (skip-chars-backward " \t")
+   (delete-region (org-element-property :begin link) (point))

Expanding attachment-links in the buffer makes the link type no longer be 
attachment. I would prefer if we explicitly set the link type to files here 
instead. Storing intermediate state in an attachment link types makes less 
sense imo.

One issue with the current way it's done is that images are treated differently 
between attachment links and file links. For HTML exports, file links are 
wrapped in a div with class figure where expanded attachment links are not. 
Letting org-attach-expand-links do the full transform to file links would solve 
that issue. That also means :export is not needed for org-link-set-parameters.

Patch attached if you agree to this.

> >> Do you want to do this for 9.4 or can it be done later on?
> >
> > This can be done later on. I don't have time to work on this. Of
> > course, if someone wants to do it, that would be great, too.
> 
> Okay, thanks.  Noted for Org > 9.4.
> 
> Best,
> 
> --
>  Bastien


org-attach.patch
Description: org-attach.patch

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Btw, the change 20d293b4a was done after a filed bug report. Maybe should be 
mentioned as well. The change was not something I just felt like doing just for 
the fun of it.

Regards
G

> -Original Message-
> From: Kyle Meyer 
> Sent: den 14 februari 2020 03:42
> To: Gustav Wikström ; Nicolas Goaziou
> 
> Cc: Bastien Guerry ; emacs-orgmode@gnu.org
> Subject: RE: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> >> Gustav Wikström  writes:
> >>
> >> > Ah, you mean the reference on line 3216. No, I don't think it can
> >> > be removed. And I honestly don't think it should be either. It's
> >> > there to let attachment links mirror the peculiarities of file
> >> > links. It's needed for feature compatibility. I don't see the
> issue with that.
> >> > It's a core link type and it needs the information. That
> particular
> >> > logic doesn't affect the parse tree. It's there only to give
> >> > attachment links the same properties as file links.
> >>
> >> I disagree. This is not a core link type. The issue is that the
> >> parser should self-contained. Please use a different way to obtain
> >> the information; we already discussed it and suggested multiple
> solutions.
> >
> > Maybe time for Bastien to step in. Because I can't remove the
> > reference to attachment in org-element.el without breaking it's
> > functionality. Which, btw, was broken before adding the reference in
> > org-element.el. The thing that started this discussion in the first
> > place. We're in a better place now.
> 
> It seems unfair to say you can't remove it because it would break
> functionality.  You committed 20d293b4a (Give link parser knowledge of
> attachment link expanded path, 2020-01-18) without posting it to the
> list [0] and giving Nicolas a chance to comment, which you've agreed
> was too hasty [1].  Misjudging the situation is of course okay, but
> please don't use that as a reason to keep a change that would not have
> landed if you had submitted it for discussion.
> 
> [0]: https://lists.gnu.org/archive/html/emacs-orgmode/2020-
> 01/msg00155.html
> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2020-
> 01/msg00162.html

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: Kyle Meyer 
> Sent: den 14 februari 2020 03:42
> To: Gustav Wikström ; Nicolas Goaziou
> 
> Cc: Bastien Guerry ; emacs-orgmode@gnu.org
> Subject: RE: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> >> Gustav Wikström  writes:
> >>
> >> > Ah, you mean the reference on line 3216. No, I don't think it can
> >> > be removed. And I honestly don't think it should be either. It's
> >> > there to let attachment links mirror the peculiarities of file
> >> > links. It's needed for feature compatibility. I don't see the
> issue with that.
> >> > It's a core link type and it needs the information. That
> particular
> >> > logic doesn't affect the parse tree. It's there only to give
> >> > attachment links the same properties as file links.
> >>
> >> I disagree. This is not a core link type. The issue is that the
> >> parser should self-contained. Please use a different way to obtain
> >> the information; we already discussed it and suggested multiple
> solutions.
> >
> > Maybe time for Bastien to step in. Because I can't remove the
> > reference to attachment in org-element.el without breaking it's
> > functionality. Which, btw, was broken before adding the reference in
> > org-element.el. The thing that started this discussion in the first
> > place. We're in a better place now.
> 
> It seems unfair to say you can't remove it because it would break
> functionality.  You committed 20d293b4a (Give link parser knowledge of
> attachment link expanded path, 2020-01-18) without posting it to the
> list [0] and giving Nicolas a chance to comment, which you've agreed
> was too hasty [1].  Misjudging the situation is of course okay, but
> please don't use that as a reason to keep a change that would not have
> landed if you had submitted it for discussion.
> 
> [0]: https://lists.gnu.org/archive/html/emacs-orgmode/2020-
> 01/msg00155.html
> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2020-
> 01/msg00162.html

Just a short note before leaving for work. I'm not trying to strong-arm 
anything here. I'm just saying I don't think it's possible to remove the 
reference without breaking functionality. The changes discussed in [0] and [1] 
are in my opinion removed already. The reference to attachment we're talking 
about here is something else. This string reference is not a change in the 
design of org-element.el for links. The other was. And that's rolled back.

I'm sorry if this is perceived as disingenuous and being done with a bad 
agenda. It's not. It's a discussion about design. Yes, we disagree about some 
things. Nothing strange there. Let's talk it out and try to find the middle 
ground. Which is what's going on. I'll respect the direction given by the 
better Org moders out there. But in this case I'm simply stating that the 
principles that needs to apply according to Nicolas will either break 
attachment link functionality (it was broken before my change in 20d293b4a) or 
require quite some thinking and/or code duplication in org-attach.el to get it 
right. And I don't have the capacity for that redesign. Rolling back the change 
to a broken state is ofc. still possible, but that is more of a regression in 
my mind than the string-reference to attachment link types in org-element.el.

Which is why I think it's a decision for the maintainer of the project to step 
in and say if it's fine to have broken attachment links, or if it's fine to let 
attachment be used the way it is on line 3216 in org-element.el, while there is 
hard-coded logic for file links the way it is on that line today.

Regards
Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 14 februari 2020 01:16
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org; Bastien Guerry 
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> > What makes attachment links not be a core link type?
> 
> 1. Org Attach is not loaded by default
> 2. Implementing Org syntax doesn't require to implement attachments
> 
> Attachment links are in the same category as Docview links, for
> example.
> There is no "docview" occurrence in "org-element.el". We have been
> there already.
> 
> > As I see it we have two categories here: Core = those inside Org
> mode.
> > Not core = those in external libraries. No other distinction needs to
> > be made. If a link type is inside Org mode, let it then use the
> > special properties that Org mode provides.
> 
> Sure thing, but you use the wrong ones!

I meant that Core = inside the Org mode repository of source code files. Both 
Docview and Attachment is, both are mentioned in the manual as part of the 
system. Both should be able to use :search-option inside the Org element parser.

> > Attachment links need to be treated just as file links in most
> > regards. Since file-links have special logic which attachment links
> > also needs. Thus a reference to attachment in org-element.el. Nothing
> > strange here, nothing breaking and the parser is still self-
> contained.
> 
> Attachement links should live only in "org-attach.el": like 99% of
> other links do live in their own library. They do not require a special
> treatment.
> 
> Again, the parser is not self-contained if it references org-attach.el.

It doesn't referencen org-attach.el any longer. That reference is removed. What 
remains is a reference to the string "attachment". You may argue that even that 
is to much. And I can only agree with that vision if you also state that 
org-element shouldn't reference the string "file" either. Anything else is a 
subjective design that puts a sentiment on "more" or "less" importance of link 
types and how to regard if they're "important" enough to be mentioned. I don't 
think you want that in org-element any more than I do.

> > Maybe time for Bastien to step in. Because I can't remove the
> > reference to attachment in org-element.el without breaking it's
> > functionality. Which, btw, was broken before adding the reference in
> > org-element.el. The thing that started this discussion in the first
> > place. We're in a better place now.
> 
> We're not. The way it is implemented is not correct.

Functionally we are - we have working attachment links that are treated in the 
same way as file links through out Org mode. Since there is special treatment 
for file links in Org, attachment links needs the same special treatment to get 
the same behavior. You might see it as a regression in the design of 
org-element.el. To which I've already given arguments about why it's not really 
- and different ways it can be solved - by removing special file-related 
properties from org-element.el. The added string-reference to attachment links 
is there because special string-references to file links are there. If special 
file link references are removed, then clearly attachment links will/should be 
removed as well.

> For opening link, attachment links should use the :follow link
> parameter. The "following" function can extract the search option, if
> any, and the file name, then call `org-open-file' appropriately. You're
> doing it backwards and modify `org-open-file' instead. Additional links
> are not supposed to modify "ol.el".

Sure, that's a principle I can agree to. And have already mentioned that I will 
try to refactor that part by giving the :follow function the same abilities as 
org-open-file currently has. But that won't be completed quite yet.

> For exporting link, it should use the :export link parameter. The
> "exporting" function can extract the search option, if any, and the
> file name. It can then re-create a file link object, and call `org-
> export-data' on it. You're also doing it backwards here and prefer to
> modify each export back-end instead.

Which I've argued about already, saying it will be difficult without having 
file already being treated the same way. I'm not opposed to doing it that way 
in the long run. But let it be done for file links first, then attachment links 
can follow. I don't have the capacity to dig into this on my own.

> > Attachment links works as intended. It would be a pity to have to
> > change that because of your argument that attachment links aren't
> > "core" enough t

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 13 februari 2020 22:38
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org; Bastien Guerry 
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> > Ah, you mean the reference on line 3216. No, I don't think it can be
> > removed. And I honestly don't think it should be either. It's there to
> > let attachment links mirror the peculiarities of file links. It's
> > needed for feature compatibility. I don't see the issue with that.
> > It's a core link type and it needs the information. That particular
> > logic doesn't affect the parse tree. It's there only to give
> > attachment links the same properties as file links.
> 
> I disagree. This is not a core link type. The issue is that the parser
> should self-contained. Please use a different way to obtain the
> information; we already discussed it and suggested multiple solutions.

What makes attachment links not be a core link type? As I see it we have two 
categories here: Core = those inside Org mode. Not core = those in external 
libraries. No other distinction needs to be made. If a link type is inside Org 
mode, let it then use the special properties that Org mode provides. Attachment 
links need to be treated just as file links in most regards. Since file-links 
have special logic which attachment links also needs. Thus a reference to 
attachment in org-element.el. Nothing strange here, nothing breaking and the 
parser is still self-contained.

Maybe time for Bastien to step in. Because I can't remove the reference to 
attachment in org-element.el without breaking it's functionality. Which, btw, 
was broken before adding the reference in org-element.el. The thing that 
started this discussion in the first place. We're in a better place now. 
Attachment links works as intended. It would be a pity to have to change that 
because of your argument that attachment links aren't "core" enough to be 
mentioned in org-element.el.

Regards
Gustav

RE: [Help] Preparing for the release of Org 9.4 tomorrow

[Gustav Wikström]

Just for reference, this is answered in the parallel thread about attachment 
links.

/G

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 13 februari 2020 22:32
> To: Gustav Wikström 
> Cc: Bastien ; emacs-orgmode@gnu.org
> Subject: Re: [Help] Preparing for the release of Org 9.4 tomorrow
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > This should be fixed since commit a24c8c481f though, no?
> >
> > https://code.orgmode.org/bzg/org-
> mode/commit/a24c8c481f63113215e66a70382d93cce82c9c7e
> 
> AFAICT, there still is an "attachment" mark in "org-element.el", at
> line 3216. Would that be an omission?
> 
> Regards,
> 
> --
> Nicolas Goaziou

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Continuation from previous answer,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 13 februari 2020 21:41
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org; Bastien Guerry 
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> [...]
> 
> > Given the above paragraph, the :path and :search-option property
> > doesn't make sense in the parser. :raw-link is enough. Less ambiguous
> > names for :path and :search-option would be :file-path
> > and :file-search-option. But that's sub-typing. We've already
> > concluded that that should not belong to the parser.
> 
> I don't have much time, I apologize if I'm not clear.
> 
> I disagree with you conclusion. Sub-typing is necessary in the parser.
> The `link' object is complex, so it needs categories or types. There are
> plain links, angle links, square links. In this last category, there are
> internal links, which include coderefs, fuzzy links, custom id, file
> links, and, "links with an explicit scheme part". For each of these
> categories, it is fine to have specific properties, like search-options.

Sure, that's not too far away from what I suggested in one of my first mails on 
the subject (except I took it a few steps too far maybe). Saying there are 
properties for categories of sub-types is fine but then this needs to be made 
much more explicit than today. 

> Note that this is sub-typing per syntax, not by meaning. This is what
> should not move, unless absolutely necessary. For example, attachment
> links belong to "square links with an explicit 'attachment' scheme
> part". That is all the parser needs to know.

That, and the way to extract the :search-options and :application from the 
link, as is done for file-links. Since attachment links would fit into your 
category of links that needs specific properties, like search-options.

> Now, it is true that [[file:...]] links are put in the same category as
> [[~/...]] links, for practical reasons, i.e., you wouldn't want them to
> differ in meaning. But this is a breach in the categories above.

Perfectly valid in my opinion. [[~/...]] is shorthand for [[file:~/...]] and, 
as long as documented as such, I see no breach. Only one such shorthand is 
possible anyways so not much ambiguity there.

> We might ignore the :search-option part in file links, but it's very
> easy to get, and, more importantly, we must extract the filename, so
> further parsing is necessary.

As you might have figured, I don't hold a very strong opinion on which design 
decision to take. I just want the choice to be based on principles and to be 
without ambiguities. If you say there are sub-types of links that require the 
:search-option, then fine, let it be so. But then we have to make it perfectly 
clear which particular types this applies to. And also make it clear that those 
special properties are only available for built-in types. I.e. to use them one 
have to get the new link-type into Org mode and into org-element.el.

> > I agree that option 1 is suboptimal. :search-option isn't obvious as
> > a property for all link types. Since option 3 is fairly trivial,
> > option 2 isn't necessary either. For attachment links to reuse
> > the :search-option semantics, without the hard-coding we're currently
> > doing, I see one option where attachment link higher order functions
> > could reuse file link higher order functions. Those file link higher
> > order functions don't exist yet as far as I know.
> 
> Higher order functions for what? `org-open-file' is such a higher order
> function for opening file links. There is no higher order function for
> exporting because each exporter handles file links differently. A higher
> order function for parsing doesn't mean much, since the consumer is not
> known yet.

Yes, each exporter handles file links differently. And I'm saying each exporter 
shouldn't handle file links at all. They should delegate that to the file 
exporter higher order function. In the same way that other link types are 
supposed to be dealt with. Principles.

> At this point, the best thing to do is to clarify what you need.
> I probably do not understand it.

I'm trying :) But it's not that /I need/ anything. It's rather the issue of how 
to solve the conundrum we're in. Where you say "attachment" links should work 
differently and not be hardcoded next to file links. And I'm saying it won't 
work unless we refactor how file-links are handled. And how we should have a 
principled approach to these kinds of things.

> > It might be seducing but I'm not sold. I'd rather have an
> > attachment-link exporter explicitly reuse functionality for exporting
> > file links than automatic translation. For that to be possible, there
> > first is a 

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

Partial answer for now;

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 13 februari 2020 21:41
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org; Bastien Guerry 
> Subject: Re: attachment: link type export to HTML invalid attach dir
>
> [...]
>
> There is an "attachement" reference left in "org-element.el" at the
> moment. Can it be removed safely?

Ah, you mean the reference on line 3216. No, I don't think it can be removed. 
And I honestly don't think it should be either. It's there to let attachment 
links mirror the peculiarities of file links. It's needed for feature 
compatibility. I don't see the issue with that. It's a core link type and it 
needs the information. That particular logic doesn't affect the parse tree. 
It's there only to give attachment links the same properties as file links.

> I'm Cc'ing Bastien because that particular point should be solved
> before any release, IMO.
> 
> Regards,
> 
> --
> Nicolas Goaziou

Re: [Help] Preparing for the release of Org 9.4 tomorrow

[Gustav Wikström]

Hi,

This should be fixed since commit a24c8c481f though, no?

https://code.orgmode.org/bzg/org-mode/commit/a24c8c481f63113215e66a70382d93cce82c9c7e

Regards Gustav

Get Outlook for Android


From: Emacs-orgmode  on behalf of 
Nicolas Goaziou 
Sent: Thursday, February 13, 2020 11:46:31 AM
To: Bastien 
Cc: emacs-orgmode@gnu.org 
Subject: Re: [Help] Preparing for the release of Org 9.4 tomorrow

Hello,

Bastien  writes:

> I would like to release Org 9.4 tomorrow, during the "I love Free
> Software Day 2020" campaign: https://fsfe.org/campaigns/ilovefs/
>
> Can you all (re)submit blocking bugs and issues?

Please don't as long as "org-element.el" retain references to attachment
links.

Regards,

--
Nicolas Goaziou

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 7 februari 2020 15:28
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> > Hmm, maybe that is so.. Except raw-path is called path (not really an
> > issue though) and there is another property called raw-link. Not sure
> > how to interpret the use of both path and raw-link. And there are
> > things happening between parsing the actual buffer link and storing
> > the raw-link and path parameters.
> 
> There is some normalization involved, indeed. The intent of :raw-link is
> to provide the link almost as it was in the buffer, without any parsing.
> I agree there should not be any `org-link-unescape' and
> `org-link-expand-abbrev' involved there. Something to fix at some point.
> 
> :path, on the contrary is parsed. It is the part of the link between the
> type and the search options, i.e., [[type:path::search]], [[type:path]],
> or [[path]].

Unless search-option applies as a general construct for all link types I don't 
think it should be in the parser. Given the discussion we've had about design 
of the parser from before. I.e. if the parser isn't supposed to know anything 
about the specific types themselves, all properties of the parsed elements have 
to make sense for all types of links. So either search-option remains but is 
parsed in exactly the same way for all link types, or it's not there at all. 
And if it's not available in the parser, the file link interpreters (that still 
might need to be constructed) gets to parse the :search-option from the 
raw-link as it wishes itself. 

Given the above paragraph, the :path and :search-option property doesn't make 
sense in the parser. :raw-link is enough. Less ambiguous names for :path and 
:search-option would be :file-path and :file-search-option. But that's 
sub-typing. We've already concluded that that should not belong to the parser.

> > In the end, what made me consider the sub-typing I've been writing
> > about could very well just have been the ambiguities regarding what's
> > what, and the lack of treatment for parts that arguably could be seen
> > as additional parameters to the link-path. For example the
> > "::extra_content" suffix in file links, that by the parser currently
> > is just a part of the path itself.
> 
> In [[file:name::extra_content]] :path is "name", and :search-option is
> "extra_content". As you noted, :search-option is not valid in non-file
> links, so it belongs to the path.
> 
> It seems there is some friction about this search option part. IIUC, you
> want attachment links to support this link-specific syntax. This is
> indeed not possible. As I commented earlier, letting libraries decide
> what the parser should do is not an option. There are a few options,
> though:
> 
> 1. Allow every link with an explicit type (i.e., not internal links) to
>have a search option, so you can write [[docview:filename::12]] or
>[[attachment:filename::*Section 2]]. IMO this is a waste, because
>most links do not need this, and it could become confusing
>[[https:www.orgmode.org::#sec2]].
> 
> 2. Provide a function in "ol.el" to do the parsing for you, so that
>every new link library doesn't have to re-invent the wheel. E.g.,
>(org-link-extract-search-options "foo::bar") => (:path
>"foo" :search-options "bar").
> 
> 3. Keep that way, i.e., any link library requiring to read the search
>part can do a dumb `match-string` and, in two or three lines of code,
>extract it. IOW, since this is so trivial, why bother?
> 
> WDYT?

I agree that option 1 is suboptimal. :search-option isn't obvious as a property 
for all link types. Since option 3 is fairly trivial, option 2 isn't necessary 
either. For attachment links to reuse the :search-option semantics, without the 
hard-coding we're currently doing, I see one option where attachment link 
higher order functions could reuse file link higher order functions. Those file 
link higher order functions don't exist yet as far as I know.

> > Maybe clearer documentation in the function of what each part is
> > /supposed/ to be, and the design principle that is applied, i.e. that
> > the path is the raw path with options included can help future me and
> > others who might want to understand. Thoughts on that?
> 
> There is some information in the manual, and the Org Syntax document.
> 
> > Hmm, don't really know if a link description should be regarded as
> > content. I for one hadn't considered it until now when you mentioned
> > it! But it preserves space in the parse tree I suppose.
> 
> Thi

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi again,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 5 februari 2020 17:54
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> > That was kind of what I was trying to do, by allowing subtyping, so
> > that the parser would be more knowledgeable about particular types of
> > links, by allowing extended attributes. Maybe not implemented in the
> > most extensible way though I'll admit.
> 
> Just to be clear, I firmly believe Element library should be as
> low-level as possible. Ideally (we're not there yet), it should not
> depend on any other part of Org. It should be the step (largely) above
> "re-search-forward".

Ok, fair enough. I guess where I was going was a bit further than only parsing 
the characters, into also interpreting them in meaningful ways. Especially true 
for attachment links that would need to look up other parts of the tree to 
fetch extended information. I'll leave that trail of thought. 

> > My suggestion to the link parser would be to have the following properties 
> > as the catch-all properties for links:
> > - type :: Which type of link protocol applies. E.g. file, http, ftp, 
> > attachment, duckduckgo (ex. for a custom link abbreviation to search ddg)
> > - raw-path :: The path to use for the particular protocol. Would contain 
> > everything in the link following "protocol:"
> > - format :: Which format the link has. Plain, bracket or angular bracket
> 
> Barring custom link abbreviations, this is already the case.

Hmm, maybe that is so.. Except raw-path is called path (not really an issue 
though) and there is another property called raw-link. Not sure how to 
interpret the use of both path and raw-link. And there are things happening 
between parsing the actual buffer link and storing the raw-link and path 
parameters. 

In the end, what made me consider the sub-typing I've been writing about could 
very well just have been the ambiguities regarding what's what, and the lack of 
treatment for parts that arguably could be seen as additional parameters to the 
link-path. For example the "::extra_content" suffix in file links, that by the 
parser currently is just a part of the path itself. Maybe clearer documentation 
in the function of what each part is /supposed/ to be, and the design principle 
that is applied, i.e. that the path is the raw path with options included can 
help future me and others who might want to understand. Thoughts on that? 

> > - description :: The description part of the link 
> > ([[protocol:path][description]])
> 
> Description is not meaningful here. This is parsed content.
>
> > - begin, end, post-blank :: The default properties used for all elements. 
> > Not sure if contents-begin and contents-end are a part here as well.
> 
> They are when the link has contents, i.e., a description.

Hmm, don't really know if a link description should be regarded as content. I 
for one hadn't considered it until now when you mentioned it! But it preserves 
space in the parse tree I suppose. If the docstring of the link parser would 
make it clear what each property is supposed to contain, in this case 
:contents-begin & :contents-end, I'm sure I would have been less confused about 
this, and wouldn't have had any objections.

> > As you've stated previously, from a performance perspective maybe it
> > will be best to not expand the specific properties and instead let the
> > expansion of those be handled in code by the link type owner (e.g.
> > org-attach.el for attachment links).
> 
> More than performance, this is a design issue.
> 
> > Ah yes. Just briefly looking at the docview code. Docview doesn't seem
> > to use the parser when extracting information about that "go to page"
> > information from the link. Which means docview links doesn't really
> > care how the link information is encoded in the parser anyways.
> 
> Indeed. However, higher level functions, e.g., org-open-link, do care
> about it and ensure that "ol-docview.el" really processes a link.
> 
> > Maybe if docview were allowed to encode custom docview information
> > into the parsed link in the parser it and others could reused that
> > encoded information later? Docview would be able to add
> > a ":go-to-page" option, for example. Just a thought.
> 
> I'm quite sure this is a wrong way to go.

Ok, got it. You're saying the link interpreter for docview (in this case) have 
to be able to parse the link one step further, to be able to extract this 
":go-to-page" information, before being able to operate on it. Indeed a design 
decision. Maybe the best one, who am I to say

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi again,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 20 januari 2020 02:25
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> > Ok, noted. To my defense I was thinking more in the terms of subtyping
> > then hardcoding. Because we have multiple link types which try to
> > share an interface. But the interface isn't perfect for all different
> > kinds of links.
> 
> Then, I suggest to improve the interface.

That was kind of what I was trying to do, by allowing subtyping, so that the 
parser would be more knowledgeable about particular types of links, by allowing 
extended attributes. Maybe not implemented in the most extensible way though 
I'll admit.

My suggestion to the link parser would be to have the following properties as 
the catch-all properties for links:
- type :: Which type of link protocol applies. E.g. file, http, ftp, 
attachment, duckduckgo (ex. for a custom link abbreviation to search ddg)
- raw-path :: The path to use for the particular protocol. Would contain 
everything in the link following "protocol:"
- format :: Which format the link has. Plain, bracket or angular bracket
- description :: The description part of the link 
([[protocol:path][description]])
- begin, end, post-blank :: The default properties used for all elements. Not 
sure if contents-begin and contents-end are a part here as well.

In addition to that each protocol would be able to declare their own properties 
using a :parse function. All would probably implement a path property. Some 
would implement an options property, and some maybe would maybe declare their 
own properties.

As you've stated previously, from a performance perspective maybe it will be 
best to not expand the specific properties and instead let the expansion of 
those be handled in code by the link type owner (e.g. org-attach.el for 
attachment links). But not letting specific protocol parsers expand their own 
properties into a link makes it more difficult to implement support outside of 
emacs for the org specification, which in my opinion includes also a certain 
set of link protocols which do have their own specifications.

> > So it doesn't seem too farfetched that some of those link types would
> > benefit from additional custom properties, specific for those types.
> > =application= and =search-option= for example isn't universally
> > applicable to all link types.
> 
> As a side note, :application is on its way out, i.e., "file+emacs" stuff
> is in "org-compat.el".
> 
> Also, IIRC, "docview" links have "go to page" option, and they don't
> touch the parser.

Ah yes. Just briefly looking at the docview code. Docview doesn't seem to use 
the parser when extracting information about that "go to page" information from 
the link. Which means docview links doesn't really care how the link 
information is encoded in the parser anyways. Maybe if docview were allowed to 
encode custom docview information into the parsed link in the parser it and 
others could reused that encoded information later? Docview would be able to 
add a ":go-to-page" option, for example. Just a thought.

> > What I'm trying to argue for here is: Maybe it's not that crazy after
> > all to allow links to have additional properties in the parser based
> > on its type? (While certainly still trying to avoid it if possible!)
> 
> If new link types may not function correctly without touching the
> parser, how do you create new link types out of Org's core? This is not
> modular at all.

Most link types probably won't need custom link properties anyways. But could 
maybe let the parser know stuff by use of higher order functions? Or push for 
being important enough to be included into Org and allowed to be known by the 
parser.

> > (Off topic) I'm sorry to hear that you think I'm intentionally making
> > things fuzzy.
> 
> Not at all! My wording is terrible. When I wrote
> 
> Also, you sometimes seem to blur, on purpose, the difference between
> "attachment" and "file" links.
> 
> I meant something like:
> 
>It seems to me that your intent is to have "attachment" links
>transparently become "file" links to the user.
> 
> Hence my suggestion to use link abbreviations. There's nothing negative
> in it.

Ok, got it. Thanks for explaining. I'll admit it hasn't been totally clear to 
me what the best way forward is. After refreshing my understanding on links the 
conclusion I came to was that link abbreviations in its current form was not a 
fitting concept for attachment links. Attachments are pretty much similar to 
file links though, so even if 

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi Nicolas,

Thanks for your comments!

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 19 januari 2020 22:12
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > Functionally it's in a good (and maintainable) state right now, in my
> > opinion. But I do understand that the contextual attribute added to
> > the parser may require some discussion.
> 
> This is not my main gripe here, although I'm not convinced that extra
> attribute is warranted. My concern is that you hard-code "attachment"
> type in the parser. I don't think this is the correct way to handle the
> situation.

Ok, noted. To my defense I was thinking more in the terms of subtyping then 
hardcoding. Because we have multiple link types which try to share an 
interface. But the interface isn't perfect for all different kinds of links. So 
it doesn't seem too farfetched that some of those link types would benefit from 
additional custom properties, specific for those types. =application= and 
=search-option= for example isn't universally applicable to all link types. And 
link abbreviations are completely hidden from the parser output, since even the 
raw-link is expanded. So maybe, in the quest for completion, link abbreviations 
should have custom properties as well? And allowing some kind of sub-typing 
might make the parsed output more easy to use, because the interpreter can 
infer based on type (which already exist as a link property) what extra 
properties to use.

What I'm trying to argue for here is: Maybe it's not that crazy after all to 
allow links to have additional properties in the parser based on its type? 
(While certainly still trying to avoid it if possible!)

> Some link types specifically handle files, e.g., "docview", yet do not
> need a special treatment in the parser. Note that "file" links have a good
> reason to be treated specially there, besides their obvious importance, as
> their type can be omitted. E.g., [[~/file.org]]. OTOH, I see no strong
> reason to handle "attachments" in here, since they behave like any other
> link type.
> 
> Worse, the parser is more or less the definition of the Org syntax.
> Therefore, including "attachment" in the parser is a signal meaning that
> in order to fully implement Org syntax, e.g., in another language, one
> need to support attachment links. Attachment links are undoubtedly useful,
> but they are not core, at all. So, I feel uneasy about leaking that type
> of link in the Element library.
> 
> Also, you sometimes seem to blur, on purpose, the difference between
> "attachment" and "file" links. If there should be no difference of
> treatment between them, as I already suggested, you may want to consider
> "attachment" as some functional link abbreviation. Then the "attachment"
> type doesn't really exist, much like the "bugzilla" link type from the
> manual.

(Off topic) I'm sorry to hear that you think I'm intentionally making things 
fuzzy. If it seems so, then please take it as a lack of clarity of thought or 
communication rather than intentional bad will. (/Off topic)

> In any case, we need a proper definition, a proper category too, for
> "attachment" links. Meanwhile, modifying the parser is just grasping at
> straws.
> 
> > I argue that the attachment folder is a part of the attachment link,
> > even though the information is found at a different location in the
> > document (i.e. as a property to nodes in the document hierarchy).
> > Parsing an attachment link would then be incomplete if that
> > information is discarded.
> 
> I argue that the buffer position of the attachment link and the path as
> written in the link are enough to fully expand the attachment file name.

One can indeed use the buffer position to derive the part of the path that 
comes from the subtree. But leaving it at that puts more requirements on the 
user of the parsed link in order to use it. Which takes us back to my first 
implementation of how the exporters would deal with the parsed link information 
(d70db54dbc).

> If I'm wrong, which could be, I probably didn't invest enough time in the
> Attach changes, then having the expanded form in :path and the initial
> form in :raw-path is enough.
> 
> > One option to adding an attribute could be to modify existing
> > properties by adding the attachment folder to, for example, the path
> > property. But that means to remove information about what was written
> > as path in the original link.
> 
> There is :raw-path for that purpose.

There is no :raw-path available in the properti

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi,

> -Original Message-
> From: stardiviner 
> Sent: den 19 januari 2020 05:28
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> [...]
> 
> From the source code, it seems indeed still original code logic.
> 
> Then I checked out my Emacs init git log, confirmed it is ~'attached~
> option value at first time. So this is my mistake.
> 
> Because this new ~attachment:~ link type has potential issues like on
> exporting.

The issue with attachment links in the exporters should be fixed since the 
report of the export issues you sent a short while back. Any external exporter 
you use may ofc still need to be updated for attachment links.

> So I decided to still use old ~file:~ link type. So I thought should set
> ~org-attach-store-link-p~ to ~t~. It indeed use ~file:~ link type instead
> of ~attachment:~ now. But the store link behavior affected.
> 
> So the problem is how can I use both ~file:~ link type for attachments and
> use this new attached store link?

Understood, and I see no issue with complementing org-attach-store-link-p with 
another option. One that stores a file link to the attached files location. 
Pushed a change for that just now. 

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi!

org-attach-store-link-p with option t is supposed to store a link to the 
original location (i.e. the location the file was/is in before it was attached 
to the node. That was the default before I started working with attachments I 
believe. Haven't ever used that feature myself but the patch you provide would 
change the functionality which I don't think is correct. It would also not 
match the documentation any longer.

See the documentation for the customization parameter:

#+begin_src emacs-lisp
  (defcustom org-attach-store-link-p nil
"Non-nil means store a link to a file when attaching it."
:group 'org-attach
:version "24.1"
:type '(choice
(const :tag "Don't store link" nil)
(const :tag "Link to origin location" t)
(const :tag "Link to the attach-dir location" attached)))
#+end_src

Regards
Gustav

> -Original Message-
> From: stardiviner 
> Sent: den 18 januari 2020 15:56
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> 
> stardiviner  writes:
> 
> I finally figured out why the link always failed. Because it use wrong
> variable which is old filename path. I attached a patch.
> 
> > Gustav Wikström  writes:
> >
> >> Hi,
> >>
> >>> -Original Message-
> >>> From: stardiviner 
> >>> Sent: den 15 januari 2020 07:21
> >>> To: Gustav Wikström 
> >>> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> >>> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach
> >>> options
> >>>
> >>> [...]
> >>>
> >>> >> I found when I set option ~(setq org-attach-store-link-p t)~.
> >>> >> Then attach a file, store file link with =[C-c C-l]=. The stored
> >>> >> link. I open this link got error "No such file: ". I tested
> >>> >> this with minimal Emacs config. confirmed this problem.
> >>> >>
> >>> >
> >>> > I cannot reproduce this. In my try with a minimal Emacs (emacs -q)
> >>> > and
> >>> with only that single customization it works for me. I'm testing it
> >>> in linux. A wild guess.. Could it be that you used the move
> >>> operation instead of the copy operation when attaching the file?
> >>> >
> >>> > Regards
> >>> > Gustav
> >>>
> >>> Did you reproduce this issue with =emacs -q= ? That is a built-in
> >>> Org Mode version which does not contains the latest version =org-
> attach.el=.
> >>>
> >>> Here is my minimal Emacs config:
> >>>
> >>> [...]
> >>>
> >>> ;;==
> >>> ==
> >>> ==
> >>> ;;; Here is org-attach.el customization
> >>>
> >>> (require 'org-attach)
> >>>
> >>> ;; store link auto with `org-store-link' using `file:' link type or
> >>> `attachment:' link type.
> >>> (setq org-attach-store-link-p 'attached) (setq
> >>> org-attach-dir-relative t) (setq org-attach-preferred-new-method
> >>> 'ask) #+end_src
> >>>
> >>> #+begin_src sh :eval no
> >>> emacs -q -l '~/.config/emacs/minimal-init.el'
> >>> #+end_src
> >>
> >
> >> Hmm, in the first mail you said that you set org-attach-store-link-p
> >> to t, but in your config it says 'attached.
> >
> > Sorry about this.
> >
> >> I've tried with a minimal config as well (using emacs -q because I
> >> build the newest org mode version into the emacs
> >> folder) and can only reproduce your issue when using the attached
> >> option for org-attach-store-link-p and then inserting that link with
> >> C-c C-l /in another heading/. Pasting the link in another heading is
> >> expected to break since the attachment link is context dependent (i.e.
> requires an attachment folder).
> >> Makes sense? If I'm still misunderstanding your use-case, would you
> >> care to describe the steps to reproduce it more in detail?
> >
> > After updated commit, don't know why, but all links worked again. I'm
> > not good at expressing thanks, but you got all my thanks on this. :)
> >
> >> Regards
> >> Gustav
> >>
> 
> --
> [ stardiviner ]
>I try to make every word tell the meaning what I want to express.
> 
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
> 

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 18 januari 2020 12:34
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > Ok, so change pushed...
> 
> I'm sorry, but this is going too fast. We're discussing core design here
> (the parser), and I couldn't even answer your proposal. Let's at least
> reach an agreement on the change to make.

Yes, agreed, it was a bit fast. The final changes was done to not stop "in the 
middle" so to say, with something both you and me think is suboptimal. 
Functionally it's in a good (and maintainable) state right now, in my opinion. 
But I do understand that the contextual attribute added to the parser may 
require some discussion. If the decision is to not allow contextual attributes 
in the parser I'm prepared to revert and change again. No stress though.

Just to add a note about the trail of my thoughts regarding this... And why I 
thought the contextual attribute was a good option here:

I argue that the attachment folder is a part of the attachment link, even 
though the information is found at a different location in the document (i.e. 
as a property to nodes in the document hierarchy). Parsing an attachment link 
would then be incomplete if that information is discarded. One option to adding 
an attribute could be to modify existing properties by adding the attachment 
folder to, for example, the path property. But that means to remove information 
about what was written as path in the original link. So I argue to keep path as 
the original path. But that means extra information is needed to also make it 
work in the filesystem. If we would translate an attachment link to a file link 
in ox.el that means we remove the option for exporters to decide for themselves 
what to do with the link. And I think the exporter should have that option. :) 
Right now the ASCII exporter for example outputs attachment links as 
attachment:expanded_path instead of file:expanded_path. Since the link type 
actually is attachment. And for a solemnly textual export the exported 
information should be kept as close to source as possible. So either we 
explicitly and always say attachment-links *are* file-links in disguise (i.e. 
even change type in the parser), or we don't say that, and then don't say that 
all the way to the edge of the system. And let the uses of the link type decide 
themselves what to do. Which is what I propose. But with the addition of a bit 
of extra information contextual for the attachment links. 
 
Whoops long paragraph, sorry... I'm just trying to explain my current way of 
looking at this.

Regards
Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Ok, so change pushed...

Where org-element-link-parser now sets one extra property only for attachment 
links. The exporters and org-link-open use this additional information when 
exporting and opening attachment links. Feature parity with file links should 
now be complete. Note that exporters outside of the Org mode repo need to be 
aware of the attachment link type if the path expansion is to be correct. They 
aren't translated in between into file-links. Not doing that translation is the 
proper think in my opinion. No magic hiding of the attachment link type. Who 
knows - maybe some exporters in the future need the link as is, without 
expansion!?. Making an exporter in the wild aware of attachment links can be 
done using org-element-property and the new property :attachment-path created 
by the parser only for attachment links.

Regards
Gustav

> -Original Message-
> From: Gustav Wikström
> Sent: den 17 januari 2020 19:36
> To: Nicolas Goaziou 
> Cc: emacs-orgmode@gnu.org
> Subject: RE: attachment: link type export to HTML invalid attach dir
> 
> Hi again,
> 
> > -Original Message-
> > From: Gustav Wikström
> > Sent: den 17 januari 2020 15:30
> > To: 'Nicolas Goaziou' 
> > Cc: emacs-orgmode@gnu.org
> > Subject: RE: attachment: link type export to HTML invalid attach dir
> >
> > Hi,
> >
> > [...]
> >
> > >
> > > It is true that Element library expands link abbreviations right
> > > before parsing a link, and an attachment is similar to a local link
> > abbreviation.
> > > This is not great because some information is lost in the
> > > process: interpreting the parse tree will not bring the abbreviation
> > > back, only its expanded form. Actually, `org-link-expand-abbrev' is
> > > called so that the parser knows what is the true type of the link,
> > > since abbreviations could expand to anything. OTOH, attachments can
> > > only expand to a "file" link, so the motivation for expansion
> > > doesn't
> > hold.
> >
> > Hmm, interesting... And are we sure the destructive behavior is
> > something we want to maintain? I for one would vote for the parsers
> > ability to provide information that can reconstruct the source... Is
> > it really worth the space saving in the parse tree to do that
> > destruction? I feel inclined here to add a :link and :raw-path
> > property to the output from the link parser for example. That would
> > allow those links that expand to be stored in the parse tree in both
> expanded and non-expanded form.
> >
> > Reasons against?
> 
> Hmm, I'm actually kind of going full circle here, back to think that the
> logic currently implemented is in its right place... Either that, or to
> just decorate the link in the parse-tree with some auxiliary information
> that can be specific for the type of link. For attachment links that
> auxiliary information would be attachment-path-prefix (or something
> shorter but possibly less clear). For abbreviated links possibly the
> auxiliary information would be it's unexpanded form. But I'm not sure of
> the necessity or need for that, except from allowing interpreters to
> reconstruct the original link.
> 
> >
> > [...]
> >
> 
> Regards
> Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi again,

> -Original Message-
> From: Gustav Wikström
> Sent: den 17 januari 2020 15:30
> To: 'Nicolas Goaziou' 
> Cc: emacs-orgmode@gnu.org
> Subject: RE: attachment: link type export to HTML invalid attach dir
> 
> Hi,
> 
> [...]
>
> >
> > It is true that Element library expands link abbreviations right
> > before parsing a link, and an attachment is similar to a local link
> abbreviation.
> > This is not great because some information is lost in the
> > process: interpreting the parse tree will not bring the abbreviation
> > back, only its expanded form. Actually, `org-link-expand-abbrev' is
> > called so that the parser knows what is the true type of the link,
> > since abbreviations could expand to anything. OTOH, attachments can
> > only expand to a "file" link, so the motivation for expansion doesn't
> hold.
> 
> Hmm, interesting... And are we sure the destructive behavior is something
> we want to maintain? I for one would vote for the parsers ability to
> provide information that can reconstruct the source... Is it really worth
> the space saving in the parse tree to do that destruction? I feel inclined
> here to add a :link and :raw-path property to the output from the link
> parser for example. That would allow those links that expand to be stored
> in the parse tree in both expanded and non-expanded form.
> 
> Reasons against?

Hmm, I'm actually kind of going full circle here, back to think that the logic 
currently implemented is in its right place... Either that, or to just decorate 
the link in the parse-tree with some auxiliary information that can be specific 
for the type of link. For attachment links that auxiliary information would be 
attachment-path-prefix (or something shorter but possibly less clear). For 
abbreviated links possibly the auxiliary information would be it's unexpanded 
form. But I'm not sure of the necessity or need for that, except from allowing 
interpreters to reconstruct the original link.

> 
> [...]
> 

Regards
Gustav

RE: Missing `org-attach-set-inherit' function

[Gustav Wikström]

Hi,

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of stardiviner
> Sent: den 17 januari 2020 08:39
> To: emacs-orgmode@gnu.org
> Subject: Missing `org-attach-set-inherit' function
> 
> 
> I found the function ~org-attach-set-inherit~ is missing. I noticed it in
> the Info document.
> 
> #+begin_quote
>  ‘i’ (‘org-attach-set-inherit’)
>   Set the ‘ATTACH_DIR_INHERIT’ property, so that children use
>   the same directory for attachments as the parent does.
> #+end_quote

I can't find that reference in the newest version of the documentation. Should 
be removed since commit ae9cd437.

Are you sure the documentation and your Org mode version is the same?

> 
> --
> [ stardiviner ]
>I try to make every word tell the meaning what I want to express.
> 
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
> 

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 17 januari 2020 01:40
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Gustav Wikström  writes:
> 
> > Hardcoding the translation of attachment-links into file-links in an
> > in-between layer (ox.el - that is somewhat complicated as well) is not
> > transparent and I think best to avoid. Even if an attachment link is
> > /very/ similar to a file link it may be best still to treat them for
> > what they are. If some export back-ends out in the wild don't work
> > with attachment-links today then so be it. But let's at least make it
> > easy to fix! So I'll try to remove the hard coding of org-attach
> > invocation and instead make the attachment-path when parsed by
> > org-element return a path that is an actual file-system path out of
> > the box. I'll see what I figure out in terms of code I suppose...!
> >
> > What do you say?
> 
> It is true that Element library expands link abbreviations right before
> parsing a link, and an attachment is similar to a local link abbreviation.
> This is not great because some information is lost in the
> process: interpreting the parse tree will not bring the abbreviation back,
> only its expanded form. Actually, `org-link-expand-abbrev' is called so
> that the parser knows what is the true type of the link, since
> abbreviations could expand to anything. OTOH, attachments can only expand
> to a "file" link, so the motivation for expansion doesn't hold.

Hmm, interesting... And are we sure the destructive behavior is something we 
want to maintain? I for one would vote for the parsers ability to provide 
information that can reconstruct the source... Is it really worth the space 
saving in the parse tree to do that destruction? I feel inclined here to add a 
:link and :raw-path property to the output from the link parser for example. 
That would allow those links that expand to be stored in the parse tree in both 
expanded and non-expanded form. 

Reasons against?  

> Besides, there's no good reason to store redundant, or even remote,
> information in the parse tree. The parser needs to be as low level as
> possible.

Yes, this is a reason against... But big enough? I humbly question that a bit. 
Also, would it be feasible to store the original "raw" form only if it differs 
from the expanded form? It may complicate the interpreters work though... They 
would have to have a bit more insight into when the original form will be 
needed. Not sure of the better design decision here. 

> Moreover, the issue encountered here is specific to export, so it makes
> sense to solve at the export level, i.e., in "ox.el". This is not trivial
> though, as it needs to be done very early, much link footnotes.
> Indeed, since the meaning of the link is position dependent, including
> parts of a document, or executing arbitrary code can mess it up. IOW
> expansion should happen in `org-export--prepare-file-contents', i.e.,
> within every included piece of text, and in `org-export-as', somewhere
> after `org-export-expand-include-keyword' call, but before Babel code
> execution.

It's actually not only specific to exporting. Opening attachment links show a 
similar issue (org-link-open). Because the current way to open files and 
attachments differ. And it shouldn't have to! That as well would be solved if 
the link parser would take care of the path expansion. One example of the 
difference today is that attachment links cannot abide to any prefix argument. 
As a user I would expect the prefix arguments that work for files to also work 
for attachments. Something I think we can fix here, by making the parser do a 
little bit more work. I'd be happy to apply a patch. 

Kind regards
Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi again,

> -Original Message-
> From: Gustav Wikström
> Sent: den 16 januari 2020 22:42
> To: Nicolas Goaziou 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: RE: attachment: link type export to HTML invalid attach dir
> 
> Hi Nicolas,
> 
> > -Original Message-
> > From: Nicolas Goaziou 
> > Sent: den 16 januari 2020 14:18
> > To: Gustav Wikström 
> > Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> > Subject: Re: attachment: link type export to HTML invalid attach dir
> >
> > Hello,
> >
> > Gustav Wikström  writes:
> >
> > > Ah yes. Found the culprit for this issue. Hopefully the last one.
> > > The exporter doesn't actually move the point in the buffer during
> > > the export. So org-attach-expand tried to expand from the first
> > > character in the buffer. This should be fixed from a few minutes ago.
> >
> > I'm not sure hard-coding attachment links in exporters in the best way
> > forward. For example, exporters in the wild may not cope with them
> > before a long time, if ever. There is some code duplication, too.
> 
> Yes indeed, duplicated functionality for all export backends as it stands.
> 
> >
> > If attachments links are similar to file links from an export point of
> > view, then I suggest to add a phase in ox.el to expand the former into
> > the latter, before even using export back-ends. This way, there is no
> > change required in the exporters, shipped in or not.
> 
> Yeah, I do think attachment links should be treated as file links when
> exported. And I like this suggestion, although that means I probably have
> to dig into the ox.el code. Not an easy task. I suspect you'd guide me to
> adding logic inside org-export-as for this. I'll have a look starting from
> there. But wouldn't mind some further insights here!

After thinking a while I'm leaning towards thinking this should be handled 
already in the element link parser and interpreter. Need a bit more metadata 
for that though, to be able to deconstruct and reconstruct the link properly 
while still providing the correct paths.

Hardcoding the translation of attachment-links into file-links in an in-between 
layer (ox.el - that is somewhat complicated as well) is not transparent and I 
think best to avoid. Even if an attachment link is /very/ similar to a file 
link it may be best still to treat them for what they are. If some export 
back-ends out in the wild don't work with attachment-links today then so be it. 
But let's at least make it easy to fix! So I'll try to remove the hard coding 
of org-attach invocation and instead make the attachment-path when parsed by 
org-element return a path that is an actual file-system path out of the box. 
I'll see what I figure out in terms of code I suppose...! 

What do you say? 

> 
> Regards
> Gustav
> 
> >
> > Regards,
> >
> > --
> > Nicolas Goaziou

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi Nicolas,

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 16 januari 2020 14:18
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > Ah yes. Found the culprit for this issue. Hopefully the last one. The
> > exporter doesn't actually move the point in the buffer during the
> > export. So org-attach-expand tried to expand from the first character
> > in the buffer. This should be fixed from a few minutes ago.
> 
> I'm not sure hard-coding attachment links in exporters in the best way
> forward. For example, exporters in the wild may not cope with them before
> a long time, if ever. There is some code duplication, too.

Yes indeed, duplicated functionality for all export backends as it stands.

> 
> If attachments links are similar to file links from an export point of
> view, then I suggest to add a phase in ox.el to expand the former into the
> latter, before even using export back-ends. This way, there is no change
> required in the exporters, shipped in or not.

Yeah, I do think attachment links should be treated as file links when 
exported. And I like this suggestion, although that means I probably have to 
dig into the ox.el code. Not an easy task. I suspect you'd guide me to adding 
logic inside org-export-as for this. I'll have a look starting from there. But 
wouldn't mind some further insights here!

Regards
Gustav

> 
> Regards,
> 
> --
> Nicolas Goaziou

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi,

> -Original Message-
> From: stardiviner 
> Sent: den 15 januari 2020 07:21
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> [...]
> 
> >> I found when I set option ~(setq org-attach-store-link-p t)~. Then
> >> attach a file, store file link with =[C-c C-l]=. The stored link. I
> >> open this link got error "No such file: ". I tested this with
> >> minimal Emacs config. confirmed this problem.
> >>
> >
> > I cannot reproduce this. In my try with a minimal Emacs (emacs -q) and
> with only that single customization it works for me. I'm testing it in
> linux. A wild guess.. Could it be that you used the move operation instead
> of the copy operation when attaching the file?
> >
> > Regards
> > Gustav
> 
> Did you reproduce this issue with =emacs -q= ? That is a built-in Org Mode
> version which does not contains the latest version =org-attach.el=.
> 
> Here is my minimal Emacs config:
>
> [...]
>
> ;;
> ==
> ;;; Here is org-attach.el customization
> 
> (require 'org-attach)
> 
> ;; store link auto with `org-store-link' using `file:' link type or
> `attachment:' link type.
> (setq org-attach-store-link-p 'attached) (setq org-attach-dir-relative t)
> (setq org-attach-preferred-new-method 'ask) #+end_src
> 
> #+begin_src sh :eval no
> emacs -q -l '~/.config/emacs/minimal-init.el'
> #+end_src

Hmm, in the first mail you said that you set org-attach-store-link-p to t, but 
in your config it says 'attached. I've tried with a minimal config as well 
(using emacs -q because I build the newest org mode version into the emacs 
folder) and can only reproduce your issue when using the attached option for 
org-attach-store-link-p and then inserting that link with C-c C-l /in another 
heading/. Pasting the link in another heading is expected to break since the 
attachment link is context dependent (i.e. requires an attachment folder). 
Makes sense? If I'm still misunderstanding your use-case, would you care to 
describe the steps to reproduce it more in detail?

Regards 
Gustav
 

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi,

> -Original Message-
> From: stardiviner 
> Sent: den 15 januari 2020 06:53
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
>
> [...]
>
> I checked out the latest commit "0ac6a9e1f", The ~~ tag is solved.
> But the ~(org-attach-dir)~ still not work. The exported ~~
> inline image still does not contains the attach directory.

Ah yes. Found the culprit for this issue. Hopefully the last one. The exporter 
doesn't actually move the point in the buffer during the export. So 
org-attach-expand tried to expand from the first character in the buffer. This 
should be fixed from a few minutes ago.

Crossing fingers for this to work properly now!

Regards
Gustav
 

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi,

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of stardiviner
> Sent: den 13 januari 2020 14:42
> To: emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> 
> I found when I set option ~(setq org-attach-store-link-p t)~. Then attach
> a file, store file link with =[C-c C-l]=. The stored link. I open this
> link got error "No such file: ". I tested this with minimal Emacs
> config. confirmed this problem.
> 

I cannot reproduce this. In my try with a minimal Emacs (emacs -q) and with 
only that single customization it works for me. I'm testing it in linux. A wild 
guess.. Could it be that you used the move operation instead of the copy 
operation when attaching the file?

Regards
Gustav

RE: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi again,

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of stardiviner
> Sent: den 14 januari 2020 06:04
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: attachment: link type export to HTML invalid attach dir
> 
> [...]
>
> Many thanks for really quick patching.
> 
> I tested out the new patch, still does not work.
> 
> #+begin_src org
> ,** Strings are not Lists, but Anyway…
>:PROPERTIES:
>:ID:   2fd354f3-ac7a-499d-9fe4-a76626bbdb38
>:END:
> 
> In Calva Paredit, strings are treated in much the same way as lists are.
> Here’s an example showing Slurp and Barf, Forward/Backward List, and Grow
> Selection.
> 
> [[attachment:string-as-list.gif]]
> 
> #+end_src
> 
> The upper org content is exported as this (HTML page):
> 
> #+begin_example
> Strings are not Lists, but Anyway…
> 
> In Calva Paredit, strings are treated in much the same way as lists are.
> Here’s an example showing Slurp and Barf, Forward/Backward List, and Grow
> Selection.
> 
> file:///home/stardiviner/Org/Wiki/Computer
> Technology/Programming/Emacs/Data/Emacs Packages/string-as-list.gif
> #+end_example
> 
> You can see:
> 
> 1. the link still does not contains the attach directory from ~(org-
> attach-dir)~.
> 2. image links are not exported as inline image displayed with ~ src="...">~.

Ah, you're right. I missed a few things. Should be fixed now in master. Care to 
verify from your end again?

Thanks
Gustav

Re: attachment: link type export to HTML invalid attach dir

[Gustav Wikström]

Hi!

Thanks for reporting this!

The code is updated on the master branch to make the exporters aware of how to 
deal with attachment links. Commit d70db54db for the curious. Basically, 
attachment links are expanded into file-links by the exporters now, before 
further processing into links in the respective markup language.

Regards
Gustav


From: Emacs-orgmode  on behalf of 
stardiviner 
Sent: Monday, January 13, 2020 13:24
To: emacs-orgmode@gnu.org
Subject: Re: attachment: link type export to HTML invalid attach dir


When I export with =[C-c C-e h o]=, I found attached inline image links are not
displayed in HTML page.

Here is Org content:

#+begin_src org
,** Strings are not Lists, but Anyway…
   :PROPERTIES:
   :ID:   2fd354f3-ac7a-499d-9fe4-a76626bbdb38
   :END:

In Calva Paredit, strings are treated in much the same way as lists are. Here’s
an example showing Slurp and Barf, Forward/Backward List, and Grow Selection.

[[attachment:string-as-list.gif]]

#+end_src

The email attachment contains the screenshot of HTML page.

I used Edebug on functions, track down to the error function ~(org-attach-dir)~,
it returns ~nil~ when in exporting to HTML, but when I evaluate 
~(org-attach-dir)~
interactively under the headline, it works correctly.

--
[ stardiviner ]
   I try to make every word tell the meaning what I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

RE: [Idea] Org Collections

[Gustav Wikström]

Hi,

> -Original Message-
> From: Roland Everaert 
> Sent: den 23 december 2019 14:32
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: [Idea] Org Collections

> Have you had a look at org-brain. I don't use is much, but there are some
> overlapping functionnality to merge, maybe.

Yeah, I'm a Org brain user. In my mind the collections and the active
collection would ideally be what the Org brain would operate on. 

> >> Did you think about the specific UI of aspects management?
> >> Proposal of UI I particularly like:
> >> - Mu4E
> >> - forge/magit
> >
> > Not really.. Except I agree with you on magit. The other I haven't used.
> 
> Mu4E is a major mode for managing e-mails using the mu index. it provides
> a main view with bookmarks and entries to perform searches and composing
> message, among othe thing, but what I find more useful are the header
> view, which displays a multi-columns list of e-mails with associated meta-
> data and a message view allowing to view the content of an e-mail. The
> header view allows for bulk actions while the message view act, obiously,
> on the current message and permit replying and transfering the current e-
> mail.

Ah ok, thanks for the description! I think a UI for collections still
is quite far away. This is a hobby project after all and it may take
quite some time before anything useful is created. But it's good with
some pointers to inspirational material!

Regards,
Gustav

RE: [Idea] Org Collections

[Gustav Wikström]

Hi!

> -Original Message-
> From: Emacs-orgmode  On Behalf
> Of Roland Everaert
> Sent: den 16 december 2019 12:26
> To: emacs-orgmode@gnu.org
> Subject: Re: [Idea] Org Collections
> 
> +1 for this idea.
> 
> You speak about one document used by multiple collections, how do you
> plan to manage that from a file system point of view?

The idea was to let the user define the scope of each collection herself. 
Similar to how an agenda is defined today (Maybe in the same way even?). Most 
simple configuration would be to let a collection be one folder. But in the end 
it would be up to the imagination of and usefulness for the user. Having one 
document in multiple collections wouldn't be any issue because the collections 
are only pointing to locations of files in the filesystem. And if creating 
overlap between collections sounds dumb then it's a simple choice by the user 
to not do it!

> How will be organized a collection, still from the FS point of view?

Maybe the comments above answer that as well?

> As some are delving into the abyss of sementic, I propose aspects
> instead of collections or contexts. Ultimately we are trying to manage
> various aspects of our life, by splitting those aspects into files or
> diretories and what not. So, if it is the intent of your idea, the term
> aspect seems more appropriate than collection or context IMHO.

Many words could work. Context, Project, Group, Aspect, Areas, etc. I first 
thought of the name "project" to match the Projectile package. But I think 
collection is a better concept here. It lets the user think not of how it 
should be used but rather of what it consists. Which is a collection of files 
(and settings). That collection can ofc. be used for project, as aspects, or be 
seen as contexts or areas. So in my mind collection is the broader, more 
applicable term. It has less subjective meaning attached to how this 
functionality could be used. It IS a collection but can be USED as aspects, for 
projects, etc. What do you say? 

> 
> Did you think about the specific UI of aspects management?
> Proposal of UI I particularly like:
> - Mu4E
> - forge/magit

Not really.. Except I agree with you on magit. The other I haven't used.

> 
> How to keep track of all those aspects?

My first thought was to define them in a simple list.

> 
> I will surely have more to say, but, as of know I am at work.
> 
> Regards,
> 
> Roland.

Thanks for your comments!

Regards
Gustav

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of stardiviner
> Sent: den 15 december 2019 15:26
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> ...
> 
> I figured out why on my side inline image does not work.
> 
> When have link with description, the inline image does not work. When has
> no link description, inline image works.
> 
> Here is an example:
> 
> #+begin_src org
> # link work for inline image
> [[attachment:screenshot.png]]
> 
> # inline image does not work
> [[attachment:screenshot.png][description]]
> #+end_src
> 
> I tested by EMC, found this is caused by some option. I will figure it
> out. Thanks!

FWIW, I see the same behavior as you do when I add descriptions. The same seems 
to 
apply for file-links. I haven't looked it up to see if that's intended though. 
Just 
wanted to share how it works for me, if you're scratching you're head over this!

Actually... Not wanting to push the "Send" button without looking into the 
manual, I 
found the following in the "Images" section:

"An image is a link to an image file(1) that does not have a 
description part"

See [[info:org#Images]]

Regards
Gustav

RE: [Idea] Org Collections

[Gustav Wikström]

Hi!

Hmm, not sure if this would classify as something hierarchical. I’d classify it 
as something set-based. It would allow “many to many” relations between files 
and collections, with definitions of the relation being declared per 
collection. I.e. a collection is a set of files. I haven’t thought of adding 
hierarchical relations between collections. I also haven’t thought of adding 
arbitrary named relations between collections (i.e. creating a graph). So this, 
in my mind, is orthogonal to hierarchies and graphs. I haven’t thought of what 
relations between collections should mean. And haven’t thought that there 
should be a syntax for that either.

But please explain more on what you think, if I misunderstood you!

/Gustav

From: John Sturdy 
Sent: den 15 december 2019 13:14
To: Gustav Wikström 
Cc: emacs-orgmode@gnu.org
Subject: Re: [Idea] Org Collections

That seems hierarchical, which is ok (as in org-mode itself) but how about 
implementing a more general graph mechanism, which could be used to do this but 
more flexibly?

On Sat, 14 Dec 2019, 21:04 Gustav Wikström, 
mailto:gus...@whil.se>> wrote:
Hi list and all honored readers!

I have an idea. One that I've mentioned before in side notes. And I want to 
emphasize that this still only is an idea. But I want to present this idea to 
you. As a way to gather feedback and get input. Maybe the idea is stupid!? 
Maybe you think it's already solved? Or maybe it's not, and lots of you 
resonate with it as well. In any case, please let me know what you think on the 
piece below!



 ORG MODE: COLLECTIONS/PROJECTS

    Gustav Wikström


…

RE: [Idea] Org Collections

[Gustav Wikström]

Hi Adam,

> -Original Message-
> From: Emacs-orgmode  On
> Behalf Of Adam Porter
> Sent: den 15 december 2019 12:01
> To: emacs-orgmode@gnu.org
> Subject: Re: [Idea] Org Collections
> 
> How does this idea compare with Akira Komamura's org-starter package?
> 
> https://github.com/akirak/org-starter

Haven't looked much at that project before. Interesting, but not quite
what I have in mind here. That project seems like a transpose of Org
mode configurations into something that is file centric. If I play
with the thought of Org collections already being available then I
think org-starter could use that concept to, in a file centric way,
declare if a file belongs to one or multiple collections instead of
(or as a complement to...) belonging to the (default) Org agenda.

Thanks for the pointer
Gustav

RE: [Idea] Org Collections

[Gustav Wikström]

Yes, aware at least. I’m sure there is a big overlap in what I’m trying to 
achieve and what projectile tries to achieve. Not sure if an Org collection 
should be a Projectile add-on, or if it should be a facility inside Org mode 
that projectile could attach to. I think the second option would be better. But 
not sure yet how to realize it. Thinking in progress.

Thanks
Gustav

From: Emacs-orgmode  On Behalf Of 
tbanelwebmin
Sent: den 15 december 2019 10:08
To: emacs-orgmode@gnu.org
Subject: Re: [Idea] Org Collections


Interesting idea!

Is everyone aware of Emacs Projectile?
https://github.com/bbatsov/projectile

Not exactly the Org Collections you talks about, Gustav, but somehow related.

Projectile manages collections of files that belong together. They may be 
anything (Org Mode, Python, C++, HTML, LibreOffice, anything). It is most often 
zero-configuration: if a directory contains a .git or .bzr sub-directory, it is 
a Projectile project. A Maven project is a Projectile project, and so on. If 
the directory is not of a known type, just adding and empty .projectile 
sub-directory makes it a Projectile project.

Quite handy.

Could your Org Collections idea be a Projectile add-on?

As an example, there is such an add-on: org-projectile
https://github.com/IvanMalison/org-projectilehttps://github.com/IvanMalison/org-projectile<https://github.com/IvanMalison/org-projectilehttps:/github.com/IvanMalison/org-projectile>
"org-projectile provides functions for the creation of org-mode TODOs that are 
associated with projectile projects."

Regards
Le 14/12/2019 à 18:32, Gustav Wikström a écrit :

Hi list and all honored readers!



I have an idea. One that I've mentioned before in side notes. And I want to 
emphasize that this still only is an idea. But I want to present this idea to 
you. As a way to gather feedback and get input. Maybe the idea is stupid!? 
Maybe you think it's already solved? Or maybe it's not, and lots of you 
resonate with it as well. In any case, please let me know what you think on the 
piece below!







 ORG MODE: COLLECTIONS/PROJECTS



    Gustav Wikström

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hmm, inline images work for attachment links on my side on Org mode master.

I considered keeping the file link option but ruled against it in the end. I 
could remove some complexity from the code in the end due to it.

Don't know why attachments aren’t expanded as images for you...

/G 

> -Original Message-
> From: stardiviner 
> Sent: den 15 december 2019 10:29
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> 
> Gustav Wikström  writes:
> 
> > FYI, pushed to master. Commit 26ace9004
> >
> 
> After update, I found the new attachment link type can't display inline
> image now. Maybe should keep *both* original ~file:~ link type and new
> ~attachment:~ link type in stored link? WDYT, Gustav?
> 
> --
> [ stardiviner ]
>I try to make every word tell the meaning what I want to express.
> 
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
> 

[Idea] Org Collections

[Gustav Wikström]

Hi list and all honored readers!

I have an idea. One that I've mentioned before in side notes. And I want to 
emphasize that this still only is an idea. But I want to present this idea to 
you. As a way to gather feedback and get input. Maybe the idea is stupid!? 
Maybe you think it's already solved? Or maybe it's not, and lots of you 
resonate with it as well. In any case, please let me know what you think on the 
piece below!



 ORG MODE: COLLECTIONS/PROJECTS

Gustav Wikström



Table of Contents
_

1. Motivation
2. Idea
3. Benefit
.. 1. For the user
.. 2. For the developer
4. Example use cases
.. 1. Separate actions from reference
.. 2. Work / Personal separation
.. 3. Separated book library
.. 4. More?
5. Risks and challenges
.. 1. Which configuration to use?
.. 2. Should project config allow local variables?
. 1. How to initialize the local variables?
.. 3. Conflict with other customizations
.. 4. Files that belong to multiple collections
.. 5. Dynamic lists of files and folders for a collection?
6. Alternatives
7. References


1 Motivation


  Org mode is more than a major mode for emacs buffers. That has been
  clear for quite some time. Org mode can operate on sets of files.
  Consolidate TODO's and calendar information into custom views. Publish
  sets of files. To do this Org mode assumes that the user has a
  configuration for each of those features. Each feature is responsible
  for maintaining its own context. And almost all of that context has
  to be set globally. So even though Org mode has commands and features
  that operate on sets of files and folders it has not yet developed
  that in a congruent, extensible and composable way. Thus, for the
  sanity of our users and developers I think it's time to ... introduce
  another concept! One that hopefully can simplify things both for users
  and developers.


2 Idea
==

  I propose to introduce `Collection' as a concept in the realm of Org
  mode. [1]

  An Org mode collection is defined as the combination of:
  1. A short name and description
  2. A collection of Org mode documents
  3. A collection of files and/or folders called attachments and
 attachment-locations for the project
  4. A collection of configurations for the given project

  Globally available collections are defined in a list,
  `org-collections'. Org mode should include a safe parameter that can
  be set as a folder customization to the local active project,
  `org-collections-active'. The default should be to the first entry in
  `org-collections' unless customized. This local parameter would be
  used to instruct Emacs and Org mode on which collection is active.
  Only one collection at a time can be active.

  Org agenda should use `org-collections-active' as default for the
  collection of Org mode documents to operate on. Org agenda should get
  a new command to switch between active projects.

  I'm thinking that there could be a special Emacs major mode for the
  collection as well, called "Org collections mode". Not sure exactly
  what to display and how to represent the project there... But
  certainly some kind of list of included documents and attachments.
  When in that mode there should possibly be single key
  keyboard-shortcuts to the most important features that operate on the
  collection. And switch between them.


3 Benefit
=

3.1 For the user


  A user would gain mainly two benefits as I can see right now:
  1. The ability to clearly define (multiple) collections of files that
 belong together across org mode, with unique configurations.
  2. Less global configuration state to manage and worry about!

  The second point might not look like much but is sooo important! Most
  programmers know that global state should be avoided. Putting things
  in a context most of the time makes things better. And if we can
  configure Org mode connected to a context it makes it much more useful
  for those who use Org mode for multiple purposes.

  The first point is equally important in my opinion. Today one must
  configure Org mode per feature. If you want to configure publishing
  you do that globally. If you want to configure the agenda, you have to
  do that globally as well. If you want to define a location for
  attachments, do it globally! What about custom TODO-keywords? Do it
  globally! Track ID-locations? Define a location globally!

  All above adds cognitive load to the user and makes it difficult to
  maintain the configuration as the use of Org mode grows (as it should
  ;) ). You have to define the context for each and every feature for it
  to know what to operate on. I claim that both the human psyche and the
  system itself will have a much more easy time if it could configure
  these features together, in a giv

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

FYI, pushed to master. Commit 26ace9004

/Gustav

> -Original Message-
> From: Emacs-orgmode  On Behalf
> Of Gustav Wikström
> Sent: den 13 december 2019 22:37
> To: numbch...@gmail.com
> Cc: emacs-orgmode@gnu.org
> Subject: RE: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> Hi,
> 
> > -Original Message-
> > From: stardiviner 
> > Sent: den 13 december 2019 14:39
> > To: Gustav Wikström 
> > Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> > Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> >
> > ...
> > > I tried to reproduce that using emacs -q just now but couldn't... Is there
> a
> > > customization that you've enabled?
> >
> > Aha, seems yes. I checked my config. I have option enabled:
> >
> > #+begin_src emacs-lisp
> > (setq org-attach-store-link-p 'attached)
> > #+end_src
> >
> > About this feature, WDYT?
> 
> Ah, yes of course. What you propose make sense. More sense than the current
> functionality!
> 
> If no one objects I'll change the link that is stored for the 'attached option
> to be an
> attachment link instead of a file link.
> 
> Regards
> Gustav

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi,

> -Original Message-
> From: stardiviner 
> Sent: den 13 december 2019 14:39
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> ...
> > I tried to reproduce that using emacs -q just now but couldn't... Is there a
> > customization that you've enabled?
> 
> Aha, seems yes. I checked my config. I have option enabled:
> 
> #+begin_src emacs-lisp
> (setq org-attach-store-link-p 'attached)
> #+end_src
> 
> About this feature, WDYT?
 
Ah, yes of course. What you propose make sense. More sense than the current 
functionality!

If no one objects I'll change the link that is stored for the 'attached option 
to be an
attachment link instead of a file link.

Regards
Gustav 

RE: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi,

> -Original Message-
> From: stardiviner 
> Sent: den 12 december 2019 10:53
> To: Gustav Wikström 
> Cc: numbch...@gmail.com; emacs-orgmode@gnu.org
> Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options
> 
> ...
> 
> For example, I press =[Ctrl-c Ctrl-a]= to attach a file. Then I press =[Ctrl-c
> Ctrl-l]= (~org-insert-link~) to insert link which will show a list of
> completions
> which are all link types prefix like ~attachment:~, and
> =file:data/a2//attachFile.png=. I mean the second link. it is already in
> completion list, but ~attchment:~ does not have this support.

Hmm, I'm not sure I follow. Is it in the same suggestion list for link type 
prefixes 
that you also get the file-link to the newly attached file? I tried to 
reproduce that 
using emacs -q just now but couldn't... Is there a customization that you've 
enabled?

Regards
Gustav 

Re: [O] FW: [RFC] Link-type for attachments, more attach options

[Gustav Wikström]

Hi stardiviner,

It is my belief that =[Ctrl-C Ctrl-l]= already is supported. You will only get 
suggestions for attachments if there are any attachments on the outline node 
you're in. Or in any of its parents if inheritance is configured.

Regards Gustav

From: Emacs-orgmode  on behalf of 
stardiviner 
Sent: Thursday, December 12, 2019 6:21:30 AM
To: emacs-orgmode@gnu.org 
Subject: Re: [O] FW: [RFC] Link-type for attachments, more attach options


Hi Gustav,

I suggest to add support for =[Ctrl-C Ctrl-l]= like ~file:~ link type. which 
will
auto in completion list. It will be convenient for user.

--
[ stardiviner ]
   I try to make every word tell the meaning what I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: Release 9.3

[Gustav Wikström]

Hi Bastien,

> I've released Org 9.3.
>
> ...
>
> As usual, this is the work of Nicolas acting as a maintainer
> and of many other contributors!

Glad to hear! Thanks a lot to everyone involved! 

Regards
Gustav

Re: [RFC] Document level property drawer

[Gustav Wikström]

Hi,

> One issue for me is the positioning of the level 0 property drawer.
> Having the requirement for that drawer starting in the very first
> line is too strong for me. I guess one would at least like to have
> the option to add some configuration with the ‘-*-...-*-’ construct
> which currently only works in the first line.

Hmm, that should work right now. 0..n comment lines are supposed to be
allowed anyways. I can debug that a bit later to see if something has
gone amiss.

> Further I think one would also like to place #+: configuration lines
> there in particular the #+title: line. What about allowing lines
> starting with character # above the level 0 property drawer? And put
> a newly created level 0 property drawer below the first line in the
> file that does not start with #?

The first patch allowed both coments ("# "-lines) and keyword lines
(#+...:) as well. But I removed keyword lines for now to start with a
bit more strict definition, per request from Nicolas. I think the
parser will be happy if there is as little information abouve the
drawer as possible, since it will have to retrace itself from the
first line of the buffer every time it needs to verify that the drawer
actually is the "proper" property drawer. If that makes sense. So the
more restrictive we can allow us to be, the better the performance and
the easier it will be to understand where the drawer goes. And less
complexity.

Happy to get more feedback on that decision though!

Thanks! Gustav

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi Adam,

Adam Porter  writes:

> There are a lot of deprecation recommendations in your attached
> document:
> 
> > I propose to depricate property-keywords
> > I propose to depricate the Options-keyword
> > I propose to relabel these keywords as document keywords
> > I propose to depricate the #+CATEGORY syntax
> > I propose to depricate the #+FILETAGS syntax
> > I propose to depricate the #+COLUMNS syntax
> > I propose to depricate the #+ARCHIVE syntax
> > I propose to depricate the TODO-keywords
> > I propose to depricate the priorities-keyword
> > I propose to depricate the tags-keyword
> > I propose to depricate the link-keyword
> > I propose to depricate the constants-keyword
> > I propose to depricate the setupfile-keyword
> > I propose to depricate the Macro-keyword
> 
> The thoroughness of your investigation is admirable.

Thanks!

> However, I propose that we don't deprecate any of those.  Org has been
> around for over a decade now.  Such drastic changes would not serve
> users well.

I think you're right in the fact that Org mode will need to continue
to understand them. I'll say again that I wrote the document quite a
while ago. It's unedited and initially meant for my eyes only. So "to
deprecate" may be perceived to strong and I certainly didn't mean to
cut them out straight away. I don't think the diverse use of keywords
are good for the future of Org mode though, and I do think there is
value in trying to consolidate functionality and possibly promote
something that is more clear and easy to understand. Many of the
existing keywords have a corresponding property-syntax. For those I
think it would be good to start promoting using properties instead of
keywords (as I've written over and over again :) ). Other keywords
affect the Org mode configuration for the current buffer. They are
basically shortcuts to customization for the current buffer. Which led
me to proposing a settings drawer. It may be that it initially is
enough to just update the documentation with a chapter about keywords
and categorizing them in some groups based on intended purpose. I do
however still like the idea of collecting those keywords in a drawer
instead of having them spread out in the document.

> Note that I'm taking your use of the word "deprecate" to mean what
> it's expected to mean in this context: that the software developers
> recommend against using it, with the intention to eventually remove
> support for the feature.  We shouldn't be removing any such features
> from Org.
> 
> Not only would it not serve users well, but it would make the software
> much more complicated.  As it stands, finding, e.g. a #+CATEGORY:
> keyword and getting its value is as simple as:
> 
> (save-excursion
>   (goto-char (point-min))
>   (when (re-search-forward (rx bol "#+CATEGORY:" (1+ blank)
>(group (1+ nonl)))
>nil t)
> (match-string 1)))
> 
> Hiding those keywords in drawers means that either:
> 
> a) Eligible drawers must be located, and then the desired
> property must be searched for inside of them.
> 
> b) Possibly valid properties must be located, and each one must be
> confirmed to be inside an eligible drawer.
> 
> What benefit would this added complexity serve?  To put the keywords
> in one place in the document?  There are already multiple ways to
> achieve that.

I don't agree here. Keywords today break the outline and have no
positional requirements. Both a property and a settings drawer would
have a fixed position to make it easy to locate, both programmatically,
visually and by search.

> I can't emphasize enough how important stability and consistency is
> for Org and its file formats right now.  As I've said, there are new
> implementations in development, which have the potential to bring a
> lot of publicity and new users to Org.  For example, this one was
> mentioned on a Hacker News post a few days ago:

There is always a balance between stability, backwards compatibility
and progression. I agree that backwards compatibility and stability is
important. But I also argue that progression is important. Good if Org
mode is gaining traction! But that doesn't mean we can't improve it
further, for it to gain even more traction! And in the larger scheme
of things, Org mode still is tiny. So let's not oversell ourselves
here. It's almost like a catch 22. But the largest hinderance for Org
mode to grow further is probably its ties with Emacs, the very thing
that makes Org mode into the powerhouse it is!

> https://github.com/mickael-kerjean/filestash
> 
> In the same HN post were examples of implementations for Vim and
> VSCode.  Already, especially in the VSCode ones, there were apparent
> incompatibilities in their implementations of the Org file format.

I've been a promoter of separating the Org mode syntax from Emacs for
a long time. I think I've written about it on this list previously as
well. So I know what you mean. 

Re: [RFC] Document level property drawer

[Gustav Wikström]

Hi,

> One issue for me is the positioning of the level 0 property drawer.
> Having the requirement for that drawer starting in the very first
> line is too strong for me. I guess one would at least like to have
> the option to add some configuration with the ‘-*-...-*-’ construct
> which currently only works in the first line.

Hmm, that should work right now. 0..n comment lines are supposed to be
allowed anyways. I can debug that a bit later to see if something has
gone amiss.

> Further I think one would also like to place #+: configuration lines
> there in particular the #+title: line. What about allowing lines
> starting with character # above the level 0 property drawer? And put
> a newly created level 0 property drawer below the first line in the
> file that does not start with #?

The first patch allowed both coments ("# "-lines) and keyword lines
(#+...:) as well. But I removed keyword lines for now to start with a
bit more strict definition, per request from Nicolas. I think the
parser will be happy if there is as little information abouve the
drawer as possible, since it will have to retrace itself from the
first line of the buffer every time it needs to verify that the drawer
actually is the "proper" property drawer. If that makes sense. So the
more restrictive we can allow us to be, the better the performance and
the easier it will be to understand where the drawer goes. And less
complexity.

Happy to get more feedback on that decision though!

Thanks! Gustav

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi!

I'll start with the most important info at the top. I've applied the
patch! But before anyone comes screaming I'll just say it's applied on
a separate branch. After consultation with Nicolas Goaziou that was
seen as the most reasonable thing to do. The idea is that it's high
time to start wrapping up 9.3 for a release and (even though I
personally would like to have this in there) since this patch might
continue to generate discussion and possibly bugs and fixes it's more
safe to let this come after the 9.3 release.

Sooo, a separate branch is created in the Org mode repository named
"next". I'm not entirely sure how we're supposed to work with it. But
I've anyways pushed my (non-breaking) patch there.

Comments on what you've written below!

Marco Wahl writes:

> > @Marco Wahl; As I understand you've applied the patch and tried it
> > out. Have you found any issues yet? What do you think of the patch
> > after having used it for a while?
> 
> Indeed I applied your patch and have it applied still.  Please note that
> I did nothing fancy and in particular I did not try to break the patch.

Got it.

> The patch works good for me.

Nice to hear!

> Noteworthy observations AFAICT:
> 
> 1. I could not translate my personal "#+TODO: . N ~ | x c g >" into a
> respective :TODO: property.

Yes, that's true. The reason is that there is no TODO-property that
fits in property drawers right now. I.e. special properties such as
TODO, TAGS, priority, scheduling and deadlines that have special
syntax for the outline still have no defined meaning for outline level
0. I ofc. think that's an oversight ;) But I may also be a bit crazy. 

A conclusion to draw from that, that may be worth writing more about,
is that the property drawer for node level 0 will not be able to
replace all file-level keywords that exist today.  Only properties that
currently can also be defined in property drawers in the outline will
work in the property drawer on level 0.  Makes sense?

The idea I had for all the other keywords that apply for the whole
file was to create another drawer, what I called a settings drawer.
Because the TODO-keyword you refer to above really is a setting that
you're making for the current file, much the same as when you make
changes in global, folder local or file local variables using the
standard emacs framework.

I've attached an investigation I did of the world of Org mode
keywords. It was done quite a while back and some things in there are
subjective and may not represent my current picture of the "ideal".
Nonetheless, maybe an interesting read for the ... other crazy people
out there?

> 2. With org-ids turned on and point before the first heading, function
> org-store-link creates an org-id property at the document level.
> 
> Regarding number 1. I think a list of document-level properties which
> don't behave the same when used in the document property drawer would be
> nice.  Ideally this list is empty AFAICT.  Maybe I overlook something.
> Is this an issue?

Ahh, see the attachment ;) Maybe can give answers to some of your
thoughts.

> I think observation 2. is just a little surprise but turns out to be
> natural when the document level property drawer is enabled.

Glad to hear!

> 
> 
> Still +1 for the inclusion of the patch and HTH,
> -- 

Thanks
Gustav


Investigation.org
Description: Investigation.org

Re: [O] org-id fixups and minor changes

[Gustav Wikström]

Hi again,

FYI, I’ve applied a patch that will make it easier to change the default 
without breaking any existing functionality. I.e. we can avoid breakage for 
existing attachment-folders if we at some point in the future would choose to 
change default for ID’s to timestamp.

Commit 42b8db0d3

/G

From: Gustav Wikström
Sent: den 31 augusti 2019 21:17
To: Carsten Dominik 
Cc: emacs-orgmode@gnu.org
Subject: RE: [O] org-id fixups and minor changes

Hi Carsten,

Yeah – you’re right, I didn’t think that much about automated ID creation so I 
stopped at seconds. I agree that it would be more general with more precision 
but that will also add some more noise into each ID. Maybe that’s not 
significant. But I also wonder how common it will be to try to batch-add ID’s…? 
I have nothing against adding more precision though, if that’s requested. What 
do you think?

Regarding documentation I’ll try to give it some thought. Maybe I’ll find some 
time to describe this area better .

I wouldn’t mind changing the default from random to timestamp. But I’m not so 
sure about all the others? One thing that complicates things is the way 
attachment functionality parse the ID. If we use timestamps as the default ID 
it makes sense to change the default way org-attach parses the ID into folders 
as well. Something like “/MM/DD_and_the_rest”. But that will be a breaking 
changing. The existing folder-structure for attachments wouldn’t match the new 
any longer. Cleverness in code might solve the breakage though… If there is 
interest in changing the default I can try to solve the issue with the breaking 
change as well.

Regards
Gustav

From: Carsten Dominik mailto:domi...@uva.nl>>
Sent: den 10 augusti 2019 00:34
To: Gustav Wikström mailto:gus...@whil.se>>
Cc: emacs-orgmode@gnu.org<mailto:emacs-orgmode@gnu.org>
Subject: Re: [O] org-id fixups and minor changes

Hi Gustav,

I can see that it feels more natural to use timestamps.  I certainly see that 
relative file names are good for across-computer compatibility.

You system assumes, if I see that correctly (have not studied it yet), that not 
more that one ID will be created per second.  Or do you have something in place 
that will catch this, for example if someone uses the mapping API to assign IDs 
to a whole bunch of entries in a short time?

You are right that this is not documented in the manual, even though it is used 
for lings and for attachments.  Maybe it would be good to explain it in an 
appendix to the manual?  Would you like to draft such a section, since you have 
been looking into this problem?

Do you think the default setting for Org should be modified?

Carsten

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi again,

I'd like to take the next step with this patch. I'm hesitant to do it without 
wider support though, since only a few people have commented.

@Marco Wahl; As I understand you've applied the patch and tried it out. Have 
you found any issues yet? What do you think of the patch after having used it 
for a while?

Any other thoughts/comments/objections/praises?

Regards
Gustav

> -Original Message-
> From: Gustav Wikström
> Sent: den 29 september 2019 12:27
> To: emacs-orgmode@gnu.org
> Cc: Nicolas Goaziou 
> Subject: [RFC] Document level property drawer
> 
> Hi,
> 
> This patch introduces a document level property drawer.
> 
> This has been discussed previously in a larger context:
> - https://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg0.html
> - https://lists.gnu.org/archive/html/emacs-orgmode/2019-08/msg00339.html
> - https://lists.gnu.org/archive/html/emacs-orgmode/2019-09/msg00010.html
> 
> The patch is a somewhat modified version of what was included in the third
> link above.
> 
> The following will be true for document level property drawers:
> 1) In the same way that one can have a property drawer for a heading, one
>can have a property drawer for a whole document.
> 2) All existing commands that can work with property drawers will
>(shall) work also on property drawers before the first heading.
> 3) Properties defined in a property drawer will have precedence over
>properties defined as a property keyword, if the same property is
>defined using both conventions.
> 4) The position for the document level property drawer is:
>- At the first line in a file that is not a comment or a keyword.
> 
>  I.e. the following will work:
>  #+begin_src org
># -*- mode: org -*-
>,#+TITLE: Test
>:PROPERTIES:
>:CATEGORY: Test
>:END:
> 
>Preamble
> 
>,* Some heading
>Some content
>  #+end_src
> 
>  but not this:
>  #+begin_src org
>Some comment and/or empty line
> 
>:PROPERTIES:
>:CATEGORY: Test
>:END:
> 
>,* Some heading
>Some content
>  #+end_src
> 
> What do you say?
> 
> Regards
> Gustav Wikström

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi Matt,

Thanks for your comment! I can assure you that you need not worry
about the propsed patch here in terms of your workflow. This is in no
way a hasty, sloppy work. Care has been taken when developing it to
not break anything existing. I hear your concerns on the larger topic
of keywords though. But that's not something that is changing in this
patch.

Some more comments below.

> I'd like to just quickly chime in in support of Adam's caution on this
> issue. I can absolutely see advantages to document level properties, I
> have written many code fragments that rely on the use of keywords and
> expect org filensyntax to be consistent with what actually exists. I
> use these code fragments to hold together a somewhat fragile workflow
> that allows me to use org in a work environment that is not especially
> receptive to simple text documents. I have invested a lot of time in
> making those systems run and sometimes even I don't entirely remember
> what I did to make them possible.
> 
> It would really, really suck to have those systems break. It would
> take me a lot of time to track down the causes and change what I
> needed to. VMs that currently pull in Emacs andnorg and my code would
> stop working. Old versions of my files would no longer render
> properly. My efforts to make my courses and other writings effectively
> reproducible by others would be significantly set back. Etc. I think
> these are the kinds of difficulties Adam means to describe.

Again, caution is ofc taken. In no way does this patch change any of
your existing properties. If something, it proposes a way for you to
simplify your setup if you at any time in the future choose to do
that.

The patch that is applied is tested and no existing functionailty is
changed. Ofc, if you have a critical flow I'd advice you to use the
stable branch of Org mode and not rely on the master branch, since
that branch from time to time will get bugs in it.

Regards
Gustav

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi Adam,



> > In no way is this a major, breaking change.  No document you have

> > today will break by the introduction of this.  The only thing changing

> > is if you *actively* create a document level property drawer and

> > choose to enter a property there that you already have defined in the

> > same document, using a property keyword.

>

> There is a bigger picture which you seem to be ignoring.



No I'm not ignoring any bigger picture. Hence the RFC which I'm asking

for comments about. So far a couple of positive remarks and your

comments on the other end of the spectrum.



> Org is distributed with Emacs itself.  Emacs versions are long-lived.

> People use old Emacs and Org versions for years, because few people

> build and install Emacs themselves, so most users use the versions

> included with their OS or other distributions.



Yes.



> Your proposed change would introduce a major change in the way document

> properties are interpreted.  This means that the same document, when

> used on Emacs/Org versions before and after this change, would be

> interpreted differently.  That's breaking forward-compatibility between

> versions, ones which will be in the wild for years.  That decision

> should not be taken lightly.



No, not really. Here we disagree again. I already interpret the

documents as an outline. And it seems others agree with me on that as

well. Your interpretation may be different but I fail to see where the

documentation agrees with your sentiment. The fact that some commands

that works for the outline doesn't work before the first headline has

been a shortcoming for a long time. I don't think that shortcoming is

something to continue to strive for.



Forward compatibility should indeed be taken into account. I don't

think the argument to stop this change due to breaking forward

compatibility is that strong though. If I as a user choose to use new

features, I agree to also use new software. As a package author you

can continue to rely on Org element and assume the user understands

that relation. If your packages creates property keywords today you

can continue to do so, or consider upgrading your package to depend on

Org mode 9.3 and start using the new feature to more easily insert

properties into a document level property drawer. If you do that you

of course have to mark your dependency to Org mode 9.3 as well.



> > Keywords that previously had file-wide effects will continue to have

> > that. That's not removed. So you must have misunderstood something

> > here.

>

> I did not misunderstand; I am looking from a different perspective.  In

> your proposal, line-based keywords can be overridden by document-level

> property drawers, while they currently are applied to the entire file

> regardless of any drawers.  That is a major change.



Ok, so correct me if I'm wrong here. The thing you object to in my

patch is the fact that the properties in the document level drawer

have a higher precedence over property keywords? Since it means that

outline level 0 have higher precedence than file level keywords? I've

already argued why I think that's natural. This order seems fine to

me: (from lowest to highest rank)

1. File level property keyword

2. Node level 0 property drawer property (file level property drawer)

3. Node level 1 property drawer property

4. Node level 2 property drawer property

5. ...and so on...



This feels less unintuitive and will be more difficult to explain to

new users:

1. Node level 0 property drawer property (file level property drawer)

2. File level property keyword

3. Node level 1 property drawer property

4. Node level 2 property drawer property

5. ...and so on...



I can't agree on changing to the second ranking only based on your

concerns. If more people were to back you up I think it's fair to have

that discussion though!



> >> What you're proposing is actually a fundamental change to the way Org

> >> documents are interpreted.  Org documents are not currently an

> >> outline, just a series of elements which may include an outline.

> >> Text and elements before a first heading are not part of a node,

> >> they're just text and elements in the document.

> >

> > I don't agree here. What I'm proposing in this patch doesn't change

> > the fundament of an Org mode document. I'd rather say it enhances the

> > fundament! Since the outline to a large extent is the fundament! The

> > following quote is from the documentation - Chapter 2, Document

> > Structure:

> >

> >   #+begin_quote Org is an outliner.  Outlines allow a document to be

> >   organized in a hierarchical structure, which, least for me, is the

> >   best representation of notes and thoughts.  #+end_quote

>

> Org is a collection of Emacs code built on top of Outline Mode, which

> interprets plain-text files in a certain way.  Since the beginning, such

> files have been interpreted such that content before the first heading

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi again Adam,

> IIUC, your proposal would work like this:
> 
> #+BEGIN_SRC org
>   :PROPERTIES:
>   :CATEGORY: Gamma
>   :END:
>   
>   # Category here is "Gamma"
> 
>   ,* Node 1
> 
>   # Category here is "Gamma"
> 
>   ,* Node 2
>   :PROPERTIES:
>   :CATEGORY: Beta
>   :END:
> 
>   # Category here is "Beta"
> 
>   ,#+CATEGORY: Alpha
> #+END_SRC

You understand correctly. In that case precedence would kick in since
a category is defined using both a category keyword and a category
inside the document property drawer. The example above is mostly
theoretical since there is no use-case for having category set on
document level using both conventions. If the example above was a real
document I'd suggest removing either the category keyword or the
category property from the document property drawer.

> In Org, some keywords are special, like #+CATEGORY.  For many years,
> such keywords have had file-wide effects regardless of their placement
> in the file.  IIUC, your proposal would change that, and that would
> still be a major, breaking change.

This seems disingenuous. In no way is this a major, breaking change.
No document you have today will break by the introduction of this.
The only thing changing is if you *actively* create a document level
property drawer and choose to enter a property there that you already
have defined in the same document, using a property keyword.

Keywords that previously had file-wide effects will continue to have
that. That's not removed. So you must have missunderstood something
here.

I understand you dislike the preference of letting the property drawer
have a higher precedance than property keywords, if the same property
is defined in both ways. I've already argued why I think that is the
sane choice to make. But having that precedance doesn't break anything
since you cannot define a property drawer on document level today.

> > If you think of the document as an outline, something Org mode is all
> > about, it makes sense to also think of things before the first
> > headline as "node level 0". And with that way of conceptually thinking
> > of the document it makes perfect sense to have a property drawer fixed
> > at the top - in the same way as it is required for all other node
> > levels.
> 
> What you're proposing is actually a fundamental change to the way Org
> documents are interpreted.  Org documents are not currently an outline,
> just a series of elements which may include an outline.  Text and
> elements before a first heading are not part of a node, they're just
> text and elements in the document.

I don't agree here. What I'm proposing in this patch doesn't change
the fundament of an Org mode document. I'd rather say it enhances the
fundament! Since the outline to a large extent is the fundament! The
following quote is from the documentation - Chapter 2, Document
Structure:

  #+begin_quote
  Org is an outliner.  Outlines allow a document to be organized in a
  hierarchical structure, which, least for me, is the best representation
  of notes and thoughts.
  #+end_quote

Thus, saying Org documents are not currently an outline again feels
disingenuous and at this point I struggle to take your comments
seriously.

Regards
Gustav

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi Sebastian,

> From: Sebastian Miele
> Subject:  Re: [O] [RFC] Document level property drawer
> Date: Tue, 01 Oct 2019 12:38:12 +

> ...

> I would like to be able to make a clear distinction between properties
> that are visible by default and properties that are not. Maybe it would
> be possible to allow some #+.. syntax following headings for subtree
> properties that are visible by default. A requirement could be made that
> such property specifications always have to be followed by a property
> drawer, even if that is empty. Then everything #+.. that is before the
> property drawer would belong to the heading/subtree, and everything #+..
> that follows the drawer would be treated as it is until now.

That maps quite well to what I also had in mind initially. What I
called "Document property keywords" in [fn:1]:

  #+begin_quote
  I propose to allow properties to be defined also as document
  property keywords. All keywords in the top of a buffer, before any
  non-comment line, are document-level keywords. In effect, they are
  properties that apply in exactly the same way as properties defined
  in the property drawer. The only reason for using a document keyword
  instead of defining it inside the property drawer is to make it more
  visible. One example would be the title-keyword (#+TITLE: ...).
  #+end_quote

Although I didn't think of generalizing it to also work for the
outline nodes. Something that makes sense to do though, given the use
case you describe!

[fn:1] https://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg0.html

FWIW
Gustav

Re: [O] [RFC] Document level property drawer

[Gustav Wikström]

Hi Adam,

> How does it differ from what was previously proposed?

It differs by not introducing the document concept in Org element.
Removing that means there is no reason to wait for another major
version of Org mode.

> What exactly does "will (shall)" mean?

You can ignore the inside of the parentheses. It carries no important
meaning.

> > 3) Properties defined in a property drawer will have precedence over
> >properties defined as a property keyword, if the same property is
> >defined using both conventions.
> 
> That protocol seems unnatural and confusing to me:
> 
> - If precedence were to be defined by something other than file-order,
>   it seems to me that those defined with #+ keywords should have
>   precedence, because they are more visible, while those in drawers are
>   hidden.
> - However, it seems to me that the simplest, most natural protocol would
>   be for later declarations to override earlier ones.

I'd argue that precedence already works that way. One has to take
inheritance into account. With inheritance turned on, tell me which
value for Property1 is used for the nodes in the following example:

#+begin_src org
  ,* Node 1
  ,* Node 2
  :PROPERTIES:
  :Property1: Value1
  :END:

  ,#+PROPERTY: Property1 Value2
#+end_src

As you'll see line number already isn't the deciding factor. 

With two ways to define properties it makes sense to first think of
which syntax to promote as "more important" and then to think of
precedence rules for duplicates within each syntax.

Having the same syntax for node level 0 as for regular nodes makes the
property functionality easy to understand and congruent. Something I
think is worth promoting by saying that property blocks on file-level
has precedence over the keyword syntax.

> > 4) The position for the document level property drawer is:
> >- At the first line in a file that is not a comment or a keyword.
> >
> >  I.e. the following will work:
> >
> >  #+begin_src org
> ># -*- mode: org -*-
> >,#+TITLE: Test
> >:PROPERTIES:
> >:CATEGORY: Test
> >:END:
> >
> >Preamble 
> >
> >,* Some heading
> >Some content
> >  #+end_src
> >
> >
> >  but not this:
> >
> >  #+begin_src org
> >Some comment and/or empty line
> >
> >:PROPERTIES:
> >:CATEGORY: Test
> >:END:
> >
> >,* Some heading
> >Some content
> >  #+end_src
> 
> That feels unintuitive to me.  Document-level property keywords may
> appear anywhere in a file, so it seems inconsistent for document-level
> property drawers to be limited in this way, as if there were an implied
> headline at the top of the file.  If it were so, I would expect to see
> many mailing list posts by users asking why the properties defined in
> their document-level property drawers aren't working.  Given that there
> is no enforcement in Org's UI to keep such drawers in certain places, I
> think the implementation should be tolerant of users' preferences and
> mistakes (cf. Postel's Law).

If you think of the document as an outline, something Org mode is all
about, it makes sense to also think of things before the first
headline as "node level 0". And with that way of conceptually thinking
of the document it makes perfect sense to have a property drawer fixed
at the top - in the same way as it is required for all other node
levels.

Regarding the placement of drawers, if you apply my patch on your end
to test it out you'll see that the built in functionality to define
properties creates the drawer for you. That's easy to do since it's
positional rule is easy to derive by the system. Try for example
org-set-property (C-c C-x p) and you'll get the drawer defined for
you. In exactly the same way as it already works when you're inside a
heading today.

The lack of posts asking why properties defined on their outline nodes
doesn't work tells me that positional requirements for property
drawers already is well understood.

Kind regards
Gustav

[O] [RFC] Document level property drawer

[Gustav Wikström]

Hi,

This patch introduces a document level property drawer. 

This has been discussed previously in a larger context:
- https://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg0.html
- https://lists.gnu.org/archive/html/emacs-orgmode/2019-08/msg00339.html 
- https://lists.gnu.org/archive/html/emacs-orgmode/2019-09/msg00010.html

The patch is a somewhat modified version of what was included in the third 
link above.

The following will be true for document level property drawers:
1) In the same way that one can have a property drawer for a heading, one
   can have a property drawer for a whole document.
2) All existing commands that can work with property drawers will
   (shall) work also on property drawers before the first heading.
3) Properties defined in a property drawer will have precedence over
   properties defined as a property keyword, if the same property is
   defined using both conventions.
4) The position for the document level property drawer is:
   - At the first line in a file that is not a comment or a keyword.

 I.e. the following will work:
 #+begin_src org
   # -*- mode: org -*-
   ,#+TITLE: Test
   :PROPERTIES:
   :CATEGORY: Test
   :END:

   Preamble 

   ,* Some heading
   Some content
 #+end_src

 but not this:
 #+begin_src org
   Some comment and/or empty line

   :PROPERTIES:
   :CATEGORY: Test
   :END:

   ,* Some heading
   Some content
 #+end_src

What do you say?

Regards
Gustav Wikström


0001-Org-document-property-drawers.patch
Description: 0001-Org-document-property-drawers.patch

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

Hi again,

I'm separating the Org document concept from the document property drawer to 
not block the document property drawer by syntax changes.

New RFC coming in a moment. Now only for the document property drawer 
functionality.

/G 

> -Original Message-
> From: Nicolas Goaziou 
> Sent: den 6 september 2019 22:10
> To: Gustav Wikström 
> Cc: emacs-orgmode@gnu.org
> Subject: Re: [RFC] Org document concept + document property drawers
> 
> Hello,
> 
> Gustav Wikström  writes:
> 
> > I'm starting out slow by making this a non-breaking change. At least
> > I'm trying to. I.e. everything that worked before should continue to
> > work.
> 
> This still is a syntax change, which is not to be taken lightly. Since
> Org 9.3 is way overdue (I'm not blaming anyone for it, mind you),
> I suggest to wait, and discuss about it with users meanwhile.

I'll separate the patches and propose to go ahead with the document property 
drawer without the conceptual addition of Org documents in the parser.

> 
> >> > Patch 0001 introduces the document element into org-element.el, and
> >> > some restructuring related to that.
> >>
> >> This should be explained in comments, and, if it lands at some point,
> >> Worg pages about syntax and exporter should be updated, too.
> >
> > Which comments do you mean? I've tried to be rigorous in the patch
> > notes. But you mean in the mail itself?
> 
> In the "org-element.el" file. There are some comments there. You could
> add some info about the changes there.

Got it.

> 
> > Noted, except in this case I think it's warranted since
> > org-back-to-heading behaves exactly the same with the exception of
> > raising an error if before the first heading. Since both functions are
> > defined next to another I'm sure a later refactor will take care of
> > both since they're structurally identical and defined two lines apart.
> 
> Your call.

My call is to follow current pattern and make sure it's at least not adding any 
complexity for possible later refactor.

> 
> > I've just "pattern-matched" myself into that docstring. It matches the
> > existing definitions of org-at-drawer-p, org-at-comment-p,
> > org-at-block-p. Which are defined around this.
> 
> Then, these docstrings should be fixed too. Actually, "Non-nil if..." is
> for variables with a boolean value. For predicates, like these
> functions, it should be : "Return t if ...".
> 
> See (info "(elisp) Documentation Tips") for more information about the
> subject.

Fixed existing docstrings in separate patch already pushed to master.

> > Hmm... I see your point. Have to think a bit here - because I have the
> > feeling that defining the predicate using org-element-at-point will
> > incur quite a performance hit.
> 
> Of course it will be slower. But "almost correct" predicates are also
> bad, in other ways. Besides, it may be premature optimization at this
> point.

I've not made any changes to this just yet. It may be worth a revisit later 
though, because I do see your point. The issue is bigger than the one added 
predicate here though, and should most likely be looked at more holistically. I 
propose to take that in a separate change as refactoring isn't made more 
complicated by the patch I've suggested (although the amount of work will 
increase slightly).

BR
Gustav

Re: [O] git-annex-related org-attach tests failing on master

[Gustav Wikström]

Ahh, thanks for that!

/G

Get Outlook for Android<https://aka.ms/ghei36>


From: Kyle Meyer 
Sent: Sunday, September 8, 2019 5:46:21 PM
To: Gustav Wikström ; Org-mode 
Subject: RE: git-annex-related org-attach tests failing on master

Gustav Wikström  writes:

> Hi again Kyle,
>
> I've pushed a change to master that fixes the failed tests.
> Please verify on your end as well as.

It does.  I've pushed a minor fixup commit to silence the byte-compiler.

Thanks.

Re: [O] git-annex-related org-attach tests failing on master

[Gustav Wikström]

Hi again Kyle,

I've pushed a change to master that fixes the failed tests. 
Please verify on your end as well as. 

> -Original Message-
> From: Kyle Meyer 
> Sent: den 4 september 2019 05:32
> To: Gustav Wikström ; Org-mode 
> Subject: RE: git-annex-related org-attach tests failing on master
> 
> Gustav Wikström  writes:
> 
> > I've taken a look and have fixed the issues on my
> > local machine.
> 
> Thank you!
> 
> > I must say the whole git-annex thing feels misplaced in the
> > Org mode repository though. I guess, as long as git annex isn't installed
> > it's not really an issue. But it surely would fit better as a separate
> library.
> 
> I'm a heavy git-annex user, but I haven't done more than play with
> org-attach (a while back), so I don't have much of an opinion here, but
> your general assessment sounds fair.  That being said, the
> org-attach/git-annex functionality has been around since 2012, so
> provided it's not making other things more difficult, moving it outside
> of Org doesn't seem worth the effort or disruption of people's
> workflows.

No worries, I was just voicing my concern. Issue hopefully fixed now.

> Thanks again for looking into the failures.
> 
> --
> Kyle

Kind regards
Gustav

[O] Minor fix for org-attach-dir

[Gustav Wikström]

For full transparency: An issue was found in org-brain [1] due to a
change in functionality of org-attach-dir in commit ae9cd4370.

I've given it some thought and figured that particular change in
functionality wasn't really needed and not obviously correct either.
So org-attach-dir will now check the filesystem (again) to see if the
attachment folder from DIR or ID property exist or not. The check can
be skipped with a new optional flag.

See commit 4b7eda1a2 for the fix.

[1] https://github.com/Kungsgeten/org-brain/pull/203

Regards
Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

(Resending this mail due to formatting-issues. Sorry!)

> From: Adam Porter
> Subject:  Re: [O] [RFC] Org document concept + document property drawers
> Date: Sun, 01 Sep 2019 11:11:22 -0500
>
> > #+begin_src org
> >   :PROPERTIES:
> >   :DIR: ~/
> >   :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
> >   :CATEGORY: Test
> >   :END:
> >
> >   Section 1 before first headline.
> >
> >   ,* TODO Headline 1
> >   Section 1 in first headline.
> >
> >   ,** TODO Sub-headline 1    :Testtag:
> >   :PROPERTIES:
> >   :DIR:  _2018/1809 Spark/
> >   :CATEGORY: Test-cat
> >   :END:
> > #+end_src
> 
> If I may be honest, I don't feel very enthusiastic about this
> document-level property drawer.  I think it's because I'm accustomed to
> thinking of property drawers as not affecting the entire document, and
> expecting "#+KEYWORD:" for in-buffer settings.

You can think of property drawers in exactly the same way as before.
There is nothing magical about this drawer. Just think of a document
as a level-0 node. Things won't be inherited into level-1 and above
nodes unless explicitly set so by your configuration. In exactly the
same way as was previously done. To be clear - this property drawer is
not a substitute for all kinds of keywords - this drawer only aims at
substituting the "#+PROPERTY:" keyword. The end goal is alignment of
functionality, to make it more obvious both in code and for the user
what a property is and how it's defined.

> I do recognize the advantage of being able to collapse them to hide
> clutter.  However, as the manual explains, almost the same benefit can
> be achieved by putting them in an outline node at the bottom of the
> file, or in another file altogether:
> 
> In-buffer settings may appear anywhere in the file, either directly
> or indirectly through a file included using `#+SETUPFILE: filename'
> syntax.
> 
> I usually put a "Config" or "Footer" node at the bottom of the file,
> marked "COMMENT" and ":noexport:", containing such settings that I don't
> want cluttering the top of the file.

Configurations is something else than setting properties in my opinion. 
I've addressed configurations in my first mail (calling them "settings",
but "options" or "config" can be seen as synonyms if you like).
Keywords today is a source of confusion. They are the multi-purpose
swiss army knife for everything that didn't fit elsewhere. I'd like to
change that too - I've done a fairly rigorous investigation of the
multiple uses of keywords today. Simplifying how we use keywords will
aid both users and plug-in developers, I'm sure. But that, too, is
another topic! Not something that is a part of the patch I'm bringing
forward here anyways.

> > [1] Sidenote: We already define projects today when we declare that
> > multiple files together are seen as our "agendas" for example. Or when
> > we configure publishing. But we lack a common framework for what a
> > "project" is in our code.
> 
> As you said, that's another issue--however, if I may, I'll point out
> that that's your concept of what "project" means, and not all users
> think of a "project" in those terms.  For example, it's not at all what
> it means in "GTD" terms, which many Org users think in.

Yes, project can be a dangerous word. A word of many meanings. I'm not
very attached to it in this context but think it fits well even though
it is overloaded. I'm a GTD'er myself and have learnt to separate the
meaning of the word (any overloaded word really) depending on context.
"Collection" might be another fitting word for what I have in mind.
I'm not sure this thread is the place to share my thoughts about Org
mode projects/collections though. We can start a separate thread for 
that if you like!

Humbly
Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

> From: Adam Porter

> Gustav Wikström  writes:
> 
> > Previous tree: 
> > (org-data nil CONTENTS)
> >
> > With this patch:
> > (org-data nil (document (doc-props) CONTENTS))
> >
> > Yeah.. The structure changed a bit. But it's a fairly trivial change 
> > in my opinion. Everything else works as before AFAIK... But I might be 
> > overseeing something. Please enlighten me in that case!
> 
> Thanks, that's a helpful illustration of the difference.
> 
> I'm glad that it only changes the result of parsing entire buffers.
> That will limit the breakage.
> 
> However, there is code in the wild that does parse entire buffers that
> way.  So that will break some code.  At least the necessary changes
> would be minor.
> 
> However, if it does indeed precipitate a major Org version increment,
> then such code will likely need to maintain compatibility with
> older code for some time, because, like Emacs, Org versions tend to
> stick around for a while.
 
I'd like very much to avoid this patch be pushed into a major version
update. Which leads me to think of your first suggestion to maybe
provide some backwards-compatibility for the parser. Specifically for
the insertion of the document-node in the syntax-tree. I'm not sure
how to go about that in a nice way though.

> Another question: how does this patch affect org-export backends?  I've
> only a passing familiarity with them.  I'm guessing that some may break,
> some of which may live outside of the Org repo.  It might be a good idea
> to take a look at some of the ones on MELPA to see if they would be
> affected.

I'm relying heavily on the test suite of org-mode for this, but I've
only made a few lines of changes inside ox.el and as far as I can see
no breakage at all for the respective built in exporters. Which leads
me to believe that if some external exports are built on the ox.el
core they'll have the same no-breakage behavior. Either that, or we
have holes to patch in our test suite.

Thanks again Adam for your comments!

Regards
Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

> From: Adam Porter

> > #+begin_src org
> >   :PROPERTIES:
> >   :DIR: ~/
> >   :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
> >   :CATEGORY: Test
> >   :END:
> >
> >   Section 1 before first headline.
> >
> >   ,* TODO Headline 1
> >   Section 1 in first headline.
> >
> >   ,** TODO Sub-headline 1:Testtag:
> >   :PROPERTIES:
> >   :DIR:  _2018/1809 Spark/
> >   :CATEGORY: Test-cat
> >   :END:
> > #+end_src
>
> If I may be honest, I don't feel very enthusiastic about this
> document-level property drawer.  I think it's because I'm accustomed to
> thinking of property drawers as not affecting the entire document, and
> expecting "#+KEYWORD:" for in-buffer settings.

You can think of property drawers in exactly the same way as before.
There is nothing magical about this drawer. Just think of a document
as a level-0 node. Things won't be inherited into level-1 and above
nodes unless explicitly set so by your configuration. In exactly the
same way as was previously done. To be clear - this property drawer is
not a substitute for all kinds of keywords - this drawer only aims at
substituting the "#+PROPERTY:" keyword. The end goal is alignment of
functionality, to make it more obvious both in code and for the user
what a property is and how it's defined.

> I do recognize the advantage of being able to collapse them to hide
> clutter.  However, as the manual explains, almost the same benefit can
> be achieved by putting them in an outline node at the bottom of the
> file, or in another file altogether:
>
> In-buffer settings may appear anywhere in the file, either directly
> or indirectly through a file included using `#+SETUPFILE: filename'
> syntax.
>
> I usually put a "Config" or "Footer" node at the bottom of the file,
> marked "COMMENT" and ":noexport:", containing such settings that I don't
> want cluttering the top of the file.

Configurations is something else than setting properties in my opinion.
I've addressed configurations in my first mail (calling them "settings",
but "options" or "config" can be seen as synonyms if you like).
Keywords today is a source of confusion. They are the multi-purpose
swiss army knife for everything that didn't fit elsewhere. I'd like to
change that too - I've done a fairly rigorous investigation of the
multiple uses of keywords today. Simplifying how we use keywords will
aid both users and plug-in developers, I'm sure. But that, too, is
another topic! Not something that is a part of the patch I'm bringing
forward here anyways.

> > [1] Sidenote: We already define projects today when we declare that
> > multiple files together are seen as our "agendas" for example. Or when
> > we configure publishing. But we lack a common framework for what a
> > "project" is in our code.
>
> As you said, that's another issue--however, if I may, I'll point out
> that that's your concept of what "project" means, and not all users
> think of a "project" in those terms.  For example, it's not at all what
> it means in "GTD" terms, which many Org users think in.

Yes, project can be a dangerous word. A word of many meanings. I'm not
very attached to it in this context but think it fits well even though
it is overloaded. I'm a GTD'er myself and have learnt to separate the
meaning of the word (any overloaded word really) depending on context.
"Collection" might be another fitting word for what I have in mind.
I'm not sure this thread is the place to share my thoughts about Org
mode projects/collections though. We can start a separate thread for
that if you like!

Humbly
Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

Hi Adam,

> This is a very interesting idea, and I don't want to dismiss your work,
> but I am concerned about how much third-party code will likely break by
> changing the results returned by org-element for parsing an Org buffer.
> I haven't thoroughly studied all of the code in your patches, so I may
> be wrong, but I think the breakage could be extensive.  For example,
> simple operations like destructuring the results of org-element parsing
> functions may be broken.  Have you done any investigation into this
> issue?
> 
> Maybe there should be a transitional period in which the existing
> org-element parsing functions would work as before, and the new
> document-level elements would be returned by a new org-element
> document-level parsing function.  Frankly, if there is breakage,the
> transition would probably take a few years, because there is a lot of
> code out there that has worked for years and may not be updated or
> replaced for years.

I have not investigated much into that to be honest. I'd argue that
it's a fairly trivial change in terms of the parser though. Everything
will work as before except when you're after the whole buffer
syntax-tree. In that case one will have to dig one step deeper into
the tree to find the content.

Previous tree: 
(org-data nil CONTENTS)

With this patch:
(org-data nil (document (doc-props) CONTENTS))

Yeah.. The structure changed a bit. But it's a fairly trivial change 
in my opinion. Everything else works as before AFAIK... But I might be 
overseeing something. Please enlighten me in that case!

I've made specification-changes only at two locations in the
test-cases for org-element. (ref. patch nr.1) 

Best
Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

Hi again,

Nicolas requested a more thorough introduction to the patch so here it
comes.

To start with, this relates to the topic presented here:
- https://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg0.html 

The first patch deals with formalities. It introduces one new greater
element called "document". Parsers and everything around it are
modified to work with this new concept. No new functionality is
introduced. I'd call this patch an "enabler" since it allows us to
(hopefully) reason better about intended behaviors and such moving
forward, but doesn't really do anything. As an example I for one
wouldn't mind to take it even one step further by introducing
"project" as another concept[1]. But that is for another day.

The second patch introduces property-drawers on document level. No
existing code will stop working, i.e. property keywords and all other
keywords will behave just as today.

The first five lines in the following example will work just as
property drawers inside headings with this patch. All commands and
functions that work with "regular" property-drawers are updated to
work also with this document level drawer.

#+begin_src org
  :PROPERTIES:
  :DIR: ~/
  :ID: 730e0151-8e34-4dd9-b978-187c3c81e6b4
  :CATEGORY: Test
  :END:

  Section 1 before first headline.

  ,* TODO Headline 1
  Section 1 in first headline.

  ,** TODO Sub-headline 1  :Testtag:
  :PROPERTIES:
  :DIR:  _2018/1809 Spark/
  :CATEGORY: Test-cat
  :END:
#+end_src

I wouldn't be surprised if I've managed to hide a few bugs in the
code. But all tests, and a couple of new ones as well, signal an OK.

Thoughts? Comments?

Kind regards 
Gustav

[1] Sidenote: We already define projects today when we declare that
multiple files together are seen as our "agendas" for example. Or when
we configure publishing. But we lack a common framework for what a
"project" is in our code.

> -Original Message-
> From: Gustav Wikström
> Sent: den 31 augusti 2019 20:50
> To: emacs-orgmode@gnu.org
> Cc: Nicolas Goaziou ; Carsten Dominik
> 
> Subject: [RFC] Org document concept + document property drawers
> 
> Hi!
> 
> I'm continuing on my proposal to introduce a "document" element in
> org-mode and the idea of seeing everything before the first headline
> as the base level 0 outline for a file. I've attached two patches that
> I'd like some public review of before pushing to master.
> 
> Patch 0001 introduces the document element into org-element.el, and
> some restructuring related to that.
> 
> Patch 0002 makes it possible to use property drawers at the document
> level. I've hopefully covered all related commands to make this work.
> And I've added a bunch of tests to guard against future regressions.
> 
> Waiting for your comments!
> 
> Kind regards Gustav

Re: [O] [RFC] Org document concept + document property drawers

[Gustav Wikström]

Hi Nicolas,

> -Original Message-
> From: Nicolas Goaziou 
> 
> > I'm continuing on my proposal to introduce a "document" element in
> > org-mode and the idea of seeing everything before the first headline
> > as the base level 0 outline for a file. I've attached two patches that
> > I'd like some public review of before pushing to master.
> 
> I will not review fully the patches, as I have no time for that.
> However, I will make a few comments about it.
> 
> First, you should show a few examples of what an Org document would look
> like, compared to what we have already, focusing particularly on the
> advantages, and what is now invalid. It is a good thing to do if you
> expect comments, as you cannot ask everyone to eyeball through the whole
> patch set.

Sure, I'll do that as a reply to my own mail to not make that comment
disappear among the details here.

> Also, whatever the outcome of the discussion is, /nothing should go in
> master as long as Org 9.3 is not released/. This looks like a breaking
> change at the most lower level (syntax, parser...), I think it may
> trigger a new major release.

I'm starting out slow by making this a non-breaking change. At least
I'm trying to. I.e. everything that worked before should continue to
work.

> > Patch 0001 introduces the document element into org-element.el, and
> > some restructuring related to that.
> 
> This should be explained in comments, and, if it lands at some point,
> Worg pages about syntax and exporter should be updated, too.

Which comments do you mean? I've tried to be rigorous in the patch
notes. But you mean in the mail itself?

> 
> > ** (renamed, modified) org--setup-collect-keywords -> org-collect-keywords
> > Renamed and generalized org--setup-collect-keywords to make it work
> > for multiple purposes.  Is not limited to a fixed set of keywords any
> > longer.  New name: org-collect-keywords.
> 
> An important typo note: we use "Org mode", or "an Org document", not
> "Org-mode" and "an org-document". Hyphens are only used to refer
> explicitly to a Lisp symbol, or its value or function.

Ah of course. I keep forgetting that one.

> 
> > ** (modified) org-element-keyword-parser
> > Uses (new) org-keyword-regexp instead of hardcoding it's own regexp.
> 
> Keep in mind that Org Element library should ultimately be as
> independent as possible to the other parts of Org, including "org.el".

Got it - is it preferred to do it the other way around - I.e. define
the regexp in org-element.el rather than org.el if there is use of the
regexp in both files?

> > +;; Org-element can parse org-mode documents.  The top-node in the
> > +;; parse-tree will always have TYPE `org-data' and PROPERTIES nil.
> 
> See my remark about typography above.

Got it.

> 
> > +;; The following part creates a fully recursive org-mode parser.
> 
> Ditto.

Ok, ofc.

> > +(defun org-back-to-heading-or-point-min ( invisible-ok)
> > +  "Go back to heading or first point in buffer.
> > +If point is before first heading go to first point in buffer
> > +instead of back to heading."
> > +  (condition-case nil
> > +  (outline-back-to-heading invisible-ok)
> > +(error
> > + (goto-char (point-min)
> 
> Try to limit use of Outline functions. They are generally slower than
> their Org counterpart. This is not true in this case, but, one day, we
> might optimize `org-back-to-heading'.

Noted, except in this case I think it's warranted since
org-back-to-heading behaves exactly the same with the exception of
raising an error if before the first heading. Since both functions are
defined next to another I'm sure a later refactor will take care of
both since they're structurally identical and defined two lines apart.

> > +(defun org-at-keyword-p nil
> > +  "Is cursor at a keyword-line?"
> 
> Non-nil if ...

I've just "pattern-matched" myself into that docstring. It matches the
existing definitions of org-at-drawer-p, org-at-comment-p,
org-at-block-p. Which are defined around this.

> > +  (save-excursion
> > +(move-beginning-of-line 1)
> > +(looking-at org-keyword-regexp)))
> 
> While this is technically correct today, please don't write a predicate
> only based on regexps, use the parser for that. For example, the parser
> can understand
> 
>   #+begin_example
>   #+keyword: not a keyword
>   #+end_example
> 
> whereas your function cannot.

Hmm... I see your point. Have to think a bit here - because I have the
feeling that defining the predicate using org-element-at-point will
incur quite a performance hit. Or what is your intuition regarding
that?

> 
> Also, you could use `org-match-line' in this case.
> 
> 
> Regards,
> 
> --
> Nicolas Goaziou

Thanks for your initial thoughts.

/G

Re: [O] org-id fixups and minor changes

[Gustav Wikström]

I’ve committed a minor fix to add parts of a second to the ISO 8601 version if 
the ID.

https://code.orgmode.org/bzg/org-mode/commit/dea0c70c7b9036f386d36dfc8864ac0e431f9d25

/G

From: Carsten Dominik 
Sent: den 1 september 2019 10:36
To: Stig Brautaset 
Cc: Gustav Wikström ; org-mode list 
Subject: Re: [O] org-id fixups and minor changes


On Sun, Sep 1, 2019, 08:49 Stig Brautaset 
mailto:s...@brautaset.org>> wrote:
Hi Gustav,

Gustav Wikström mailto:gus...@whil.se>> writes:
> [...] I also wonder how common it will be to try to batch-add ID’s…?

Not especially uncommon, I think.  Both the org-rss and org-drill
packages batch-add IDs on first use.

And even if that would be uncomment - at least the defaults need to be safe.

Carsten


Regards,
Stig