Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Matt Price]

On Sat, Mar 26, 2022 at 4:31 AM Ihor Radchenko  wrote:

> No Wayman  writes:
>
> > I'm open to bringing doct's features into Org mode, but I'd prefer
> > it not to be spread out over another two years.
>
> Proper integration of doct into org-capture.el would indeed be ideal. I
> am happy that you are ok with such option.
>
> Practically, we can do it step-by-step in separate patches:
> 1. Introduce alist format for org-capture-templates. No new
>functionality yet, just change the existing supported template
>options to :keyword value pairs. Also, make sure that we keep
>backwards compatibility.
> 2. Extend the alist with new options, like ${keyword} expansion, hooks,
>contexts, children, etc Maybe in a sequence of patches.
> 3. Provide validation of capture templates.
>
> WDYT?
>
> Best,
> Ihor
>

I'll just say that I would be an enthusiastic adopter of the new syntax
(just one data point). I agree wholeheartedly that named properties in
plists are much easier to remember and adjust than positional arguments,
especially when there are lots of the latter.   So, I hope this move
sforward.

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Ihor Radchenko]

Ihor Radchenko  writes:

> 2. Extend the alist with new options, like ${keyword} expansion, hooks,
>contexts, children, etc Maybe in a sequence of patches.

For reference, someone proposed a patch to extend org-capture-templates
and support per-template hooks:
https://list.orgmode.org/orgmode/20220206140551.1263130-1-herr.valentin.m...@gmail.com/

Best,
Ihor

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Ihor Radchenko]

No Wayman  writes:

> I'm open to bringing doct's features into Org mode, but I'd prefer 
> it not to be spread out over another two years.

Proper integration of doct into org-capture.el would indeed be ideal. I
am happy that you are ok with such option.

Practically, we can do it step-by-step in separate patches:
1. Introduce alist format for org-capture-templates. No new
   functionality yet, just change the existing supported template
   options to :keyword value pairs. Also, make sure that we keep
   backwards compatibility.
2. Extend the alist with new options, like ${keyword} expansion, hooks,
   contexts, children, etc Maybe in a sequence of patches.
3. Provide validation of capture templates.

WDYT?

Best,
Ihor

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[physiculus]

+1

-- 
Jens Reimer

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[No Wayman]

Nicolas Goaziou  writes:

The fact that we need a tool like "doct" to write templates in a 
compact
form may be a sign that the data type is not good enough. 
Actually it

sounds like a failure somehow.


Agreed. That's why I wrote doct in the first place.
As others have expressed, there's far too many elements required 
to ask the user to recall what position they fall in.
A plist is better in that the keys describe their values and can 
be specified in any order.
This also makes it easier for people to share templates, because 
they can actually tell what the template is intended to do by 
looking at it.


Another thing that is confusing about "capture templates" is that 
the words "template" and "entry" are overloaded in the 
documentation. From the docstring of org-capture-templates:



Templates for the creation of new entries.
Each entry is a list with the following items:


So org-capture-templates is a "list of entries which create new 
entries via templates"?
Confusing for me now, let alone the first time I tried to write my 
own templates.


doct also does more to check the validity of the template's form 
when it is defined.
I prioritized this because the worst time to figure out your 
template is malformed is when you go to capture something in the 
middle of another, unrelated task.


Should templates definition be more compact out of the box? 
Could the
data type be made more powerful to permit more complex templates 
without
relying on doct? IOW, can a tool such a doct be made transparent 
to the

user?


My initial idea for integrating doct with Org was to just provide 
a user option, e.g. org-capture-use-doct-syntax.
Then the only thing needed would be a conditional in 
org-capture-upgrade-templates which would delegate to doct when 
said user option is non-nil. 
That would allow those who are interested in using the feature to 
opt-in without breaking existing templates. Then as support for 
older syntax is dropped, (if ever? it looks like 
`org-capture-upgrade-templates' dates back to early 2017, and I 
think it makes sense to retain such a path) those features could 
be moved into org-capture.


I'm open to bringing doct's features into Org mode, but I'd prefer 
it not to be spread out over another two years.

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Tim Cross]

Nicolas Goaziou  writes:
>
> Ihor Radchenko  writes:
>
>> Nicolas Goaziou  writes:
>>
>>> I think there should be a direct mapping between Customize interface and
>>> values. Adding this macro as a band-aid to simply configuration is not,
>>> IMO, a solution.
>>
>> I think that current customize interface for org-capture-template is
>> perfectly fine. doct aids users who set org-capture-templates
>> programatically.
>
> I am merely pointing out that the programatically defined value is then
> no longer compatible with Customize interface. It may be important, or
> not.
>

This could be an issue for users who use both customise and
programatically modify their configuration.  



> The fact that we need a tool like "doct" to write templates in a compact
> form may be a sign that the data type is not good enough. Actually it
> sounds like a failure somehow.
>
> Should templates definition be more compact out of the box? Could the
> data type be made more powerful to permit more complex templates without
> relying on doct? IOW, can a tool such a doct be made transparent to the
> user?
>

Yes, I would agree. Such tools are generally a 'code smell'. Underlying
data structures should be transparent and not require helper functions
to assist the user in creating or updating them.

While I can see how doct can help with some use cases, I'm a little
concerned about the mixing of both setting of the org-capture-templatges
variable and adding/manipulating hooks together in one function. It
feels like two separate concerns being meshed into one function/macro
for convenience and attempting to work around limitations in the
existing system.

As you say, doct might be useful as a transition aid, but perhaps the
correct fix is to make it largely unnecessary (which I would assume
would also provide the additional functionality it offers to users of
customise and not only to those who configure their setup
programatically).

>From a personal perspective, I've never needed the additional
functionality offered by doct and while writing a capture template does
require lots of referencing of documentation, I suspect that even with
doct, I would still need to do that as I simply don't write new capture
templates often enough to remember all the options (even if they are
named rather than positional arguments).

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Nicolas Goaziou]

Hello,

Ihor Radchenko  writes:

> Nicolas Goaziou  writes:
>
>> I think there should be a direct mapping between Customize interface and
>> values. Adding this macro as a band-aid to simply configuration is not,
>> IMO, a solution.
>
> I think that current customize interface for org-capture-template is
> perfectly fine. doct aids users who set org-capture-templates
> programatically.

I am merely pointing out that the programatically defined value is then
no longer compatible with Customize interface. It may be important, or
not.

>> If capture templates values are too complicated, what about simplifying
>> them, and possibly use this macro as a temporary solution to help
>> transition?
>
> Could you elaborate what you are referring to? doct is a simplification
> of otherwise more complex templates. Without doct, users may need to
> write addition Elisp to define complex templates. With doct, a number of
> things can be done in a more compact form.

The fact that we need a tool like "doct" to write templates in a compact
form may be a sign that the data type is not good enough. Actually it
sounds like a failure somehow.

Should templates definition be more compact out of the box? Could the
data type be made more powerful to permit more complex templates without
relying on doct? IOW, can a tool such a doct be made transparent to the
user?

Regards,
-- 
Nicolas Goaziou

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[João Pedro de Amorim Paula]

On 20 March 2022 18:19, Ihor Radchenko  wrote:

> Also, if anyone agrees with my arguments below, do not stay silent and
> drop a "+1" email below. Otherwise, this whole thing will be stalled.
> There is no point discussing technical aspects, if there is no interest.

It isn't really something that bothers me, or causes my any sort of
problems, given that I have the same set of capture templates for close
to 3 years now, and have never felt the need to change it.

That being said, getting to those capture templates was a pain indeed,
having to, as you alluded to in another e-mail, constantly revisit the
docstring for the precise position of each element, and more often than
not getting them wrong. So I'm all up for it!

Cheers,

-- 
João Pedro de A. Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Ihor Radchenko]

Nicolas Goaziou  writes:

> I think there should be a direct mapping between Customize interface and
> values. Adding this macro as a band-aid to simply configuration is not,
> IMO, a solution.

I think that current customize interface for org-capture-template is
perfectly fine. doct aids users who set org-capture-templates
programatically. doct also does not break the mapping between customize
and the values - it converts the doct syntax into the default
org-capture-template format + adds a number of shorthands for
capture-related hooks + to capture contexts + provides easier definition
of similar templates (via inheritance).

For example the below doct definition provides two similar templates
without a need to duplicate them:

(defcustom org-capture-ref-capture-template `( :group "org-capture-ref template"
:type entry
,@org-capture-ref-capture-target
:fetch-bibtex (lambda () 
(org-capture-ref-process-capture)) ; this must run first
:link-type (lambda () 
(org-capture-ref-get-bibtex-field :type))
:extra (lambda () (if 
(org-capture-ref-get-bibtex-field :journal)
 (s-join "\n"
 '("- [ ] 
[[elisp:(browse-url (url-encode-url (format \"https://sci-hub.se/%s\; 
(org-entry-get nil \"DOI\"][downlaod and attach pdf]]"
   "- [ ] 
[[elisp:org-attach-open][read paper capturing interesting references]]"
   "- [ ] 
[[elisp:(browse-url (url-encode-url (format 
\"https://www.semanticscholar.org/search?q=%s\; (org-entry-get nil 
\"TITLE\"][check citing articles]]"
   "- [ ] 
[[elisp:(browse-url (url-encode-url (format 
\"https://www.connectedpapers.com/search?q=%s\; (org-entry-get nil 
\"TITLE\"][check related articles]]"
   "- [ ] check if 
bibtex entry has missing fields"
   "- [ ] Consider 
subscribing to new citations"))
   ""))
:org-entry (lambda () 
(org-capture-ref-get-org-entry))
:template
("%{fetch-bibtex}* TODO %?%{space}%{org-entry}"
 "%{extra}")
:children (("Interactive org-capture-ref 
template"
:keys ,(car 
org-capture-ref-capture-keys)
:space " ")
   ("Silent org-capture-ref template"
:keys ,(cadr 
org-capture-ref-capture-keys)
:space ""
:immediate-finish t)))
  "Default capture template. The template is a template defined using
  `doct' syntax. See docstring of `doct' for details."
  :type 'list
  :group 'org-capture-ref)


> If capture templates values are too complicated, what about simplifying
> them, and possibly use this macro as a temporary solution to help
> transition?

Could you elaborate what you are referring to? doct is a simplification
of otherwise more complex templates. Without doct, users may need to
write addition Elisp to define complex templates. With doct, a number of
things can be done in a more compact form.

A bright example is defining a capture hook that will only be activated
for a specific template. doct allows to do it seamlessly, while vanilla
syntax will require user to set the hook manually and check the current
active template via elisp in a non-obvious ways that require knowing
org-capture internals.

Best,
Ihor

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Ihor Radchenko]

Mark Barton  writes:

> I was concerned about long term support and if I would be left with templates 
> I would have to rewrite again, plus I was too busy at the time to adopt it 
> and it stays in my list of TODOs that are not scheduled.

To clarify, I do not think that we need to switch to doct format of the
templates. The old format must be kept for backwards-compatibility.

>  That being said, I use org capture daily, but to the points you raise, I 
> avoid changing or adding to to my capture templates.

There is not much problem with current format if you already have
working capture templates. They should keep working even if doct gets
merged.

The problem with current format reveals itself once you need to tweak
the existing templates or need to write new templates. doct makes things
much easier and much more readable.

Best,
Ihor

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Mark Barton]

+1 as interested
I saw this project a while ago
https://github.com/progfolio/doct 

I was concerned about long term support and if I would be left with templates I 
would have to rewrite again, plus I was too busy at the time to adopt it and it 
stays in my list of TODOs that are not scheduled. That being said, I use org 
capture daily, but to the points you raise, I avoid changing or adding to to my 
capture templates.



> On Mar 20, 2022, at 3:19 AM, Ihor Radchenko  wrote:
> 
> I am pinging this thread again because I believe that doct syntax is
> much easier to write compared to current defaults. It should be added to
> Org core.
> 
> Also, if anyone agrees with my arguments below, do not stay silent and
> drop a "+1" email below. Otherwise, this whole thing will be stalled.
> There is no point discussing technical aspects, if there is no interest.
> 
> Citing again an example illustrating the difference between a slightly
> non-trivial capture template using current vs. doct syntax:
> 
> No Wayman  writes:
> 
>> Without DOCT:
>> 
>> ;;rest of template configured elsewhere...
>> ;;make sure you update this if you ever change the key for this 
>>  template!
>> (defun my-org-template-hook ()
>>  (when (string= (org-capture-get :key t) "t")
>>(message "Template \"t\" selected.")))
>> 
>> (add-hook 'org-capture-mode-hook 'my-org-template-hook)
>> 
>> With DOCT:
>> 
>> (doct `("Example" :keys "t" :file ""
>>:hook (lambda () (message "Template %s selected." 
>>(doct-get :keys)
>> 
>> DOCT ensures that function is only run for that template without 
>> having the user manually filter against `org-capture-plist's :key.
>> It also allows the user to do this inline with the rest of the 
>> declaration vs spreading it out.
> 
> I believe that the right way to introduce this syntax is creating an
> easy-to-use macro like (org-declare-capture-template ...), which takes
> care about fiddling with org-capture-templates variable and its format.
> 
> The current format of a number of Org customisations is cumbersome. For
> example, org-capture-templates and org-agenda-custom-commands often
> become long list of lists with each element that must have a very
> specific position. Every single time I need to write a new template, I
> have to carefully consult the docstring; and every single time I an
> reviewing my templates, I have to consult the docstring again simply to
> understand the meaning of the first, second, ..., nth element of the
> template. Having CL-style :keyword value plists is much more readable.
> 
> More generally, it would be useful to provide a doct-like functionality
> for org-agenda-custom-commands and similar variables (like
> org-agenda-custom-commands, or say, also org-latex-classes).
> 
> Best,
> Ihor
> 

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Nicolas Goaziou]

Hello,

Ihor Radchenko  writes:

> I am pinging this thread again because I believe that doct syntax is
> much easier to write compared to current defaults. It should be added to
> Org core.
>
> Also, if anyone agrees with my arguments below, do not stay silent and
> drop a "+1" email below. Otherwise, this whole thing will be stalled.
> There is no point discussing technical aspects, if there is no
> interest.

I think there should be a direct mapping between Customize interface and
values. Adding this macro as a band-aid to simply configuration is not,
IMO, a solution.

If capture templates values are too complicated, what about simplifying
them, and possibly use this macro as a temporary solution to help
transition?

Regards,
-- 
Nicolas Goaziou

Re: [RFC] DOCT: Declarative Org Capture Templates (easier template syntax)

[Ihor Radchenko]

I am pinging this thread again because I believe that doct syntax is
much easier to write compared to current defaults. It should be added to
Org core.

Also, if anyone agrees with my arguments below, do not stay silent and
drop a "+1" email below. Otherwise, this whole thing will be stalled.
There is no point discussing technical aspects, if there is no interest.

Citing again an example illustrating the difference between a slightly
non-trivial capture template using current vs. doct syntax:

No Wayman  writes:

> Without DOCT:
>
> ;;rest of template configured elsewhere...
> ;;make sure you update this if you ever change the key for this 
>   template!
> (defun my-org-template-hook ()
>   (when (string= (org-capture-get :key t) "t")
> (message "Template \"t\" selected.")))
>
> (add-hook 'org-capture-mode-hook 'my-org-template-hook)
>
> With DOCT:
>
> (doct `("Example" :keys "t" :file ""
> :hook (lambda () (message "Template %s selected." 
> (doct-get :keys)
>
> DOCT ensures that function is only run for that template without 
> having the user manually filter against `org-capture-plist's :key.
> It also allows the user to do this inline with the rest of the 
> declaration vs spreading it out.

I believe that the right way to introduce this syntax is creating an
easy-to-use macro like (org-declare-capture-template ...), which takes
care about fiddling with org-capture-templates variable and its format.

The current format of a number of Org customisations is cumbersome. For
example, org-capture-templates and org-agenda-custom-commands often
become long list of lists with each element that must have a very
specific position. Every single time I need to write a new template, I
have to carefully consult the docstring; and every single time I an
reviewing my templates, I have to consult the docstring again simply to
understand the meaning of the first, second, ..., nth element of the
template. Having CL-style :keyword value plists is much more readable.

More generally, it would be useful to provide a doct-like functionality
for org-agenda-custom-commands and similar variables (like
org-agenda-custom-commands, or say, also org-latex-classes).

Best,
Ihor