Re: How to go to an Org headline programmatically?

[Richard Lawrence]

Hi Marcin,

Marcin Borkowski  writes:

> I want to find an Org headline in Elisp.  No need to make it visible,
> for example - just move the point to the heading with a given title (it
> may even be within `save-excursion', as in "go there, do something, go
> back").  `org-link-search' seems to do what I want, but it does a lot
> more - is there some simpler function to do that?

Do any of

org-find-top-headline
org-find-exact-heading-in-directory
org-find-exact-headline-in-buffer

do what you want?

-- 
Best,
Richard

Re: Turn Emacs holidays into Org appointments?

[Richard Lawrence]

Hi David and all,

David Rogers  writes:

> Is there a way that I can use something like the Emacs holiday 
> forms ... but from inside an Org-mode appointment timestamp

Yes, this should be possible. 

> Here's a mock example of what I'm trying to do:
>
> * Easter
> <(holiday-easter-etc 0)>
> ** stuff for Easter
>
> * The Sunday after Easter
> <(holiday-easter-etc 7)>
> ** stuff for after Easter

Actually, you're pretty close already.

Org understands diary sexp entries in timestamps, and diary sexp entries
basically allow arbitrary elisp, so you should be able to use the diary
functions in combination with the holiday functions to do what you want.
See the 'Timestamps' entry in the Org manual, and the 'Special Diary
Entries' entry in the Emacs manual, for more.

Here is a quick and dirty proof of concept; see if it works for you:

* The Sunday after Easter
  <%%(apply #'diary-date (caar (holiday-easter-etc 7)))>

The angle brackets tell Org this is a timestamp. The '%%' says that the
following sexp is a diary sexp. And the code inside it feeds the (M D Y)
date representation returned by holiday-easter-etc into diary-date.

You may need to fiddle with this a bit...holiday-easter-etc is a bit too
closely tied to the calendar (it depends on a couple of calendar
variables, displayed-year and displayed-month, that cannot be
dynamically bound). But you could create your own version that doesn't
have this limitation. 

Hope that helps.

-- 
Best,
Richard

Re: [PATCH] Fix org-comment-line-break-function

[Richard Lawrence]

Kaushal Modi  writes:

> On Sat, Dec 4, 2021, 5:25 PM Tim Cross  wrote:
>
>> Given that Nicholas cannot remember the reason for the original function
>> and suspects it was meant to be an internal only function, I think this
>> patch is probably the best way forward and should be applied.
>
> Thanks. Nicolas asked me to add tests for this patch. But I need to look
> into how to add tests for behavior of bindings. Need to add tests for M-j
> binding behavior when cursor is within a comment or outside.

FWIW I have been running the equivalent of Kaushal's patch (I just
removed the local binding of comment-line-break-function to
'org-comment-line-break-function without deleting the function) and have
also set

(debug-on-entry 'org-comment-line-break-function)

I've been running this for at least a week with no issues, and have
never been dropped into the debugger. So at least the testing through my
normal work indicates there are no problems with the patch.

-- 
Best,
Richard

Re: [PATCH] Fix org-comment-line-break-function

[Richard Lawrence]

Tim Cross  writes:

> Well, that is the big question - why was
> org-comment-line-break-function added instead of just using the
> default comment-indent-new-line?

Looking back at commit d58d40f0c864ae3a6d7c66df34769619ad2486c1, I see
this comment was added by Nicolas (still in org.el):

;; `org-auto-fill-function' takes care of auto-filling. It calls
;; `do-auto-fill' only on valid areas with `fill-prefix' shadowed with
;; `org-adaptive-fill-function' value. Internally,
;; `org-comment-line-break-function' breaks the line.

which at least suggests ("Internally, ...") that
org-comment-line-break-function was never intended to be called to break
a comment via a user command like M-j, and was only intended to be
called during filling, in a context where fill-prefix would be set correctly.

In that case, the problem might be that org-setup-filling sets
comment-line-break-function to 'org-comment-line-break-function
(overriding the default value of 'comment-indent-new-line) "globally" in
Org buffers, causing it to be called even when fill-prefix is not
set correctly.

The documentation for (the variable) comment-line-break-function is a
bit confusing:

"Mode-specific function that line breaks and continues a comment.
This function is called during auto-filling when a comment syntax
is defined."

which also suggests that this function will only be called during
filling. But that is clearly not true (even if it once was), since
default-indent-new-line calls it directly.

So maybe org-comment-line-break-function was written under assumptions
that no longer hold?

It might also be worth pointing out here that
org-comment-line-break-function is *almost* a line-for-line copy of part of
default-indent-new-line, except the latter is more careful about
checking that fill-prefix is not nil before calling insert-* on it.

> Until this is determined, I think the only 'safe' approach would be to
> just advise those who are impacted by the M-j issue to set
> comment-line-break-function to comment-indent-new-line and then wait
> to see if someone who has more historical context to comment. 

This is a bit trickier than it sounds, because in Org buffers,
org-setup-filling calls

(setq-local comment-line-break-function 'org-comment-line-break-function)

which will override any global value for comment-line-break-function
(e.g. in your init file).  So you'd have to reset it in Org
buffers locally, via a hook that runs after org-setup-filling.

-- 
Best,
Richard

[PATCH] Fix org-comment-line-break-function (was: Is M-j broken for you in Org on Emacs 27 and 28?)

[Richard Lawrence]

Hi Tim and all,

Thanks for sticking with me here...

Tim Cross  writes:

> I just checked this when running emacs -Q and get the following
>
> comment-line-break-function is a variable defined in ‘simple.el’.
>
> Its value is ‘org-comment-line-break-function’
>
> and fill-prefix is
>
> fill-prefix is a variable defined in ‘simple.el’.
>
> Its value is nil

OK, those are the values I have too...

> and I don't get any error with M-j and cannot reproduce the issue you
> are encountering. .

Do you see an error if you explicitly call

(insert-before-markers-and-inherit nil)

?

Because that is what org-comment-line-break-function does when
fill-prefix is nil. That is the source of the error on all the versions
where I have reproduced it. If you don't see the error then, it would
indicate to me that something in your setup is suppressing it.

> Looking at the git log, I can only find these messages relating to
> default-indent-new-line
> ...
> which indicates the function was added in 2007 by RMS and made the
> default for M-j in 2019. 

Right, which is why I don't see it with M-j in Emacs 26, presumably.
Emacs 26.1, the version installed on my Debian system, was released May
28, 2018.

> my suspicion is that your org version is too old for the current Emacs
> versions... The other possibility is that you have a broken "mixed"
> installation of org.

I'm aware of the difficulties of a mixed installation and have been
careful to avoid them. I run Org from git, usually a recent pull of the
bugfix branch, which I load via use-package from my init file. But I
have also tested it with the built-in Org for various versions of Emacs
with emacs -Q. I do not have Org installed via ELPA.

Just to be extra, super sure, I built Emacs this afternoon from a
checkout of the repo, and the error is *still* there, with the same
cause. In that build, with emacs -Q, I have:

(org-version)
"9.5"

(emacs-version)
"GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5, cairo 
version 1.16.0)
 of 2021-11-30"

At this point I've replicated the bug on my machine in four different
builds of Emacs (version 26.1 from Debian, 27.2 and "emacs-next" from
Guix, and version 29.0.50 I built myself from source) with several
versions of Org (the built-in ones in these Emacsen and a recent build
of the bugfix branch). It is robustly reproducible for me, and the cause
is clear: default-indent-new-line calls org-comment-line-break-function,
which calls

(insert-before-markers-and-inherit nil)

which is a type error. I'm looking for help figuring out what the right
fix is. I attach a patch for the simplest fix I can think of; please let
me know if something else would be better.

-- 
Best,
Richard

diff --git a/lisp/org.el b/lisp/org.el
index 1a1375461..fdeec0d67 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -19695,7 +19695,8 @@ non-nil."
   (save-excursion (forward-char -1) (delete-horizontal-space))
   (delete-horizontal-space)
   (indent-to-left-margin)
-  (insert-before-markers-and-inherit fill-prefix))
+  (when fill-prefix
+(insert-before-markers-and-inherit fill-prefix)))
 
 
 ;;; Fixed Width Areas

Re: Is M-j broken for you in Org on Emacs 27 and 28?

[Richard Lawrence]

Tim Cross  writes:

> I think something is very wrong if your Emacs 28 has org 9.3. I'm pretty
> sure the earliest version which was bundled with Emacs 28 was 9.4 - it
> is certainly 9.5 now and that is the version that will be bundled with
> it when it is released.

Fair enough -- I did think that was weird. I installed Emacs 28 as
emacs-next via Guix, which I am new to; maybe it installed a very old
build, or maybe something is screwed up in the load-path. 

Still, I think this is not relevant to the problem. I see the same
problem on Emacs 27.2 (also installed via Guix) with the built-in Org
9.4.4. I also see it on both Emacs 27.2 and 28 when I load Org
from the latest bugfix branch in git ("Org mode version 9.5.1
(release_9.5.1-13-gc91271 @ /home/rwl/src/org-mode/lisp/)").

Several other people have confirmed that they see it too -- two in this
thread, one privately to me off-list.

So I don't think it's a problem with my setup. But I'm still trying to
get clear on where the problem could be. The functions involved
(default-indent-new-line, org-comment-line-break-function, and
insert-before-markers-and-inherit) mostly haven't been modified in a
long time. What *has* changed is that M-j now calls this stack of
functions, when it used to call comment-indent-new-line (in Emacs 26).

In fact, if I run M-x default-indent-new-line in an Org buffer in Emacs
26 with the built-in Org 9.1.9, I get the same error. I think this issue
has been lurking for a while -- it's just that the new binding brought
it to the surface.

-- 
Best,
Richard

Re: Is M-j broken for you in Org on Emacs 27 and 28?

[Richard Lawrence]

Sorry, forgot to reply to this:

Tim Cross  writes:

> Note that C-j in org mode is different from 'normal' C-j in that it is
> bound to org-return-and-maybe-indent. If you want M-j to act like C-j in
> org mode, you would need to rebind M-j to org-return-and-maybe-indent in
> an appropriate org mode startup hook. 

This is a good suggestion, thanks; maybe that will be the best solution
for me, if the answer is that the current behavior is not a bug. 

It now seems to me that it must be a bug, though, since Org sometimes
calls a built-in C function with an argument it cannot accept, and
several people have confirmed this. The main question for me at this
point is: does this happen because org-comment-line-break-function is
being called when it shouldn't be, or because fill-prefix is nil when it
shouldn't be, or something else?

-- 
Best,
Richard

Re: Is M-j broken for you in Org on Emacs 27 and 28?

[Richard Lawrence]

Tim Cross  writes:

> I'm running Emacs 28 and cannot reproduce the issue you observe.

Hmm, the plot thickens!

> Running emacs -Q I find M-j is bound to
>
> M-j runs the command default-indent-new-line (found in global-map),
> which is an interactive compiled Lisp function in ‘simple.el’.

I definitely see the error in emacs -Q with

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo 
version 1.16.0)

which only contains Org 9.3, in my installation. So the problem has been
around at least that long, but only surfaced for me because the binding
of M-j changed between Emacs 26 and 27.

> This binding is the same inside and outside of org mode.

Yes, but inside Org mode, default-indent-new-line calls
org-comment-line-break-function (because it is the value of
comment-line-break-function), which is where the error originates.

The last line of org-comment-line-break-function is:

  (insert-before-markers-and-inherit fill-prefix)

and fill-prefix is nil, at least in all the contexts where I've tried
this.

Since you're not seeing the error, could you please check in an Org
buffer whether:

  - comment-line-break-function is 'org-comment-line-break-function
  - org-comment-line-break-function contains the line above (it should;
it's still there in the master branch)
  - fill-prefix is nil when you type M-j?
  
Thanks!

-- 
Best,
Richard

Re: Is M-j broken for you in Org on Emacs 27 and 28?

[Richard Lawrence]

Colin Baxter   writes:

> I confirm that it also appears broken to me in emacs-27.2, with the same
> error as you found. I have never noticed it before, possibly because I
> use C-j rather than M-j.

Thanks for confirming. Do you know what the difference between C-j and
M-j is "supposed" to be? They both do very mode-dependent things. I
guess M-j is more explicitly aimed at continuing comments (which is
probably why I started using it), but it has always worked great for me
as a newline-and-indent-like-I-want command outside of comments too.

-- 
Best,
Richard

Is M-j broken for you in Org on Emacs 27 and 28?

[Richard Lawrence]

Hi Org community,

Some questions for those of you on Emacs 27 and 28:

Does M-j in an org-mode buffer do what you expect?
Does it throw an error?
What function is M-j bound to in Org?

Backstory:

I have long been on Emacs 26.3 (in Debian stable) but recently decided
to try a newer Emacs from GNU Guix. I immediately ran into an issue,
and now I'm trying to figure out if it's a bug, and if so where to file
it: in both Emacs 27.2 and 28.0.50, typing M-j in an Org buffer throws
(wrong-type-argument char-or-string-p nil)

The source of this problem appears to be that the keybinding for M-j
changed between Emacs 26 and 27. In Emacs 26 it calls
indent-new-comment-line. In Emacs 27 and 28 it calls
default-indent-new-line, and the call stack look like:

  insert-before-markers-and-inherit(nil)
  org-comment-line-break-function(nil)
  default-indent-new-line()
  funcall-interactively(default-indent-new-line)
  call-interactively(default-indent-new-line nil nil)
  command-execute(default-indent-new-line)

The error arises because insert-before-markers-and-inherit cannot accept
nil (the value of fill-prefix in this context).

I see this error in emacs -q with both Emacs 27 and 28 from Guix.

After some investigation, the functions involved here don't appear to
have changed at all recently (though see [1]); just the binding. This
leads me to ask: why hasn't this been discovered already? Which leads
me to wonder if I am using M-j in some non-standard way.

Some time in the distant past, I internalized the idea that M-j is a
better way to type a newline because (a) it doesn't involve a pinky
reach and (b) in most contexts in Emacs, it is more likely to "do what I
mean" than RET is. In particular, it continues comments and indents
properly. (I am also an evil-mode user and there is probably some part
of my brain that thinks "M-j is like j for insert mode".) But maybe that
was always wrong, and the recently changed binding is just an indication
that I was not using M-j as intended.

So which is it? Is this a bug in Emacs, in Org, or in my ingrained
typing habits? Many thanks for your advice.

-- 
Best,
Richard

[1] There is a commit which changed default-indent-new-line in August of
this year, but the changes don't seem relevant to the error I'm seeing,
since I also see it in Emacs 27.2:

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=b41f31d2b60269bd0e7addd1081f3738f91e76bc

Re: Citations merged!

[Richard Lawrence]

Hi everyone,

Nicolas Goaziou  writes:

> It took years, but citations are now full part of Org syntax.

Sorry to be late to the party, but congratulations and thanks to all of
those who worked on this, especially Nicolas, Bruce, Denis and András!
It's really wonderful to see this work come to fruition.

> Thanks to everyone involved over the time!
>
> Now, it needs to be documented, but that will come a bit later.

I'm sorry I wasn't able to help much with this over the last few months,
due to other work. But I should have a bit more time starting in August,
so if I can help with documentation, please let me know.

-- 
Best,
Richard

Re: allow HTML block to escape from outline-text div? WAS: BUG? unable to surround subtrees with html tag

[Richard Lawrence]

Hi Matt,

Matt Price  writes:

>> I would like to be able to surround some portion of a subtree with a tag,
>> e.g.:
>>
>> * parent
>>   some text
>> #+HTML: 
>> ** child 2
>>   some boxed content
>> ** child 2
>>more boxed content
>>  #+HTML:
>> ** child 3
>>   unboxed content

> I don't know if there is a way to somehow slide my own html in between the
> outine-text element and the outline-container element for a child subtree.
> If someone knows a way to do this, I'd appreciate a pointer, but for now I
> think I have to find another way to accomplish this.

Is it important that your two headlines in the boxed content export as ?

If not, you could just use a structure like

* Parent
** Box
*** Boxed Child 1
*** Boxed Child 2
** Unboxed Child

and use something like the ignoreheading filter (see Worg's "Org Hacks"
page) to prevent "Box" from producing a separate header, and maybe
various properties (e.g. UNNUMBERED?) to keep the boxed children from
appearing as part of the main outline. 

Otherwise, #+INCLUDE comes to mind as a possible solution.

Would one of those options work for you?

-- 
Best,
Richard

Re: Cannot schedule something for 2039?

[Richard Lawrence]

Hi Alan,

alan.schm...@polytechnique.org writes:

> I need to schedule something for 2039, but when I do it the date is set
> for 2037. I tried with a plain emacs config and I see the same issue. Is
> this a bug?

whoa, that's a weird behavior, but it seems not to be a bug. I learned
something today!

It looks like some Emacs implementations don't support dates after
2038-1-1, so Org doesn't let you specify them by default. See the
variable `org-read-date-force-compatible-dates'. I think you want to set
this variable to nil.

-- 
Best,
Richard

Re: org-cite: make 'suppress-author' a citation 'style'

[Richard Lawrence]

Denis Maier  writes:

> I think Bruce's point was that author suppresion list items other than 
> the first lies in the jurisdiction of the citeproc, and has to be 
> handled depending on citation style. (I.e., user input does /not/ matter 
> here.)

Ahhh OK, that makes it clearer. Thanks! Sorry for the noise -- I haven't
been able to follow this thread all that closely due to other work,
though I remain interested and willing to help.

-- 
Best,
Richard

Re: org-cite: make 'suppress-author' a citation 'style'

[Richard Lawrence]

"Bruce D'Arcus"  writes:

>> e.g. you can write
>>
>> Smith claims foo [(cite): -@Smith2019; -@Smith2020; see also @Jones2018].
>>
>> to render
>>
>> Smith claims foo (2019; 2020; see also Jones 2018).

> You identified the same case Andras and I discussed just above.
>
> I think the solution is simple; with the updated syntax:
>
> [cite/-:@Smith2019;@Smith2020; see also @Jones2018].
>
> So the rule is (and this is for the processor to worry about, of
> course, not org), when a suppress-author style is specified, the
> suppression only applies to the author of the first citation
> item/reference. That would generate the output you noted.

I think I see what you mean after parsing this a few times, but I would
rephrase the rule, since the way you phrased it initially seemed to
contradict the example. The point of my example was that author
suppression should also apply to the author of the *second* reference in
the list, though that happens to be the same author as the author of the
first item. So maybe the rule is more like: "when a suppress-author
style is specified, the suppression only applies to the initial segment
of citation items/references that have the same author as the first
reference"?

-- 
Best,
Richard

Re: org-cite: make 'suppress-author' a citation 'style'

[Richard Lawrence]

Hi all,

"Bruce D'Arcus"  writes:

>> We introduced :suppress-author because someone requested it at some
>> point. I don't remember who, but it may be worth asking that person.
>>
>
> I did some quick searching.
>
> Wow; this goes back a long time!
>
> Anyway, Richard Lawrence summarized previous discussions, which includes
> this detail, in this 2015 post:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2015-02/msg00932.html
>
> Not sure if he introduced the idea or not, but if not, he should know.

I don't remember exactly anymore, and a quick search does not turn up
the source, but I can say in general that the concern back then was to
make the citation syntax at least as flexible as the BibLaTeX citation
commands, to support users who currently use BibLaTeX (and don't have
much interest in non-LaTeX output).

> I think there are two points to consider before removing suppress-author
>> syntax:
>>
>> 1. does it make sense to apply it independently on references within
>>a full citation?

I don't remember what my thinking was exactly, but it was probably
something like this: (a) we need citations with multiple references; (b)
in some such citations we may need to suppress the author of some
references without suppressing the author of others; (c) thus
:suppress-author must be a reference-level property, not a property of
the whole citation object and (d) uniformity demands that every
reference should have it (even if it doesn't make sense in practice to
apply it to references in the middle of the list).

That said, here is a possible use case for this: suppose you are
referencing multiple works from a given author and want to avoid
printing the author's name multiple times, but also want to include a
further reference to a work by another author:

e.g. you can write

Smith claims foo [(cite): -@Smith2019; -@Smith2020; see also @Jones2018].

to render

Smith claims foo (2019; 2020; see also Jones 2018).

Does that help?

-- 
Best,
Richard

Re: Problems while trying to load feature

[Richard Lawrence]

Hi Ypo,

Ypo  writes:

> Problems while trying to load feature ‘ol-org-w3m’ [etc.]
>
> It says "ol-..." because I tried to solve it after reading the mail 
> list. It is said that it happens because in version 27.1 files changed 
> their name from "org-bbdb" to "ol-org-bbdb", but as you can see, the 
> problems keep popping up. I'd say those files don't even exist on my 
> computer.
>
> /GNU Emacs 27.1 (build 1, i686-w64-mingw32)//of 2020-08-21/
>
> /org:  Version: 20210308/

I think we need some more information to help you:

1) How did you install Org? (Git? MELPA?) It looks like you are using a
very recent version, more recent than what ships with Emacs or is
available on ELPA.

2) It's not clear to me whether you have verified that you have the
files. Have you checked that these files exist and are somewhere on your
Emacs load-path?

3) Do you still see this problem if you run emacs -q? 
What happens if you then execute 

(require 'org)

via M-: or via C-x C-e in *scratch*?

-- 
Best,
Richard

Re: [PATCH] capture: Fix handling of time range for :time-prompt

[Richard Lawrence]

Kyle Meyer  writes:

> Pushed (3e64d3475).

Thank you!

>> As far as I can tell, that is not fully possible today, even with this
>> patch. The reason is that time *range* information entered at the prompt
>> generated by :time-prompt gets thrown away. The reason for *that* is
>> that org-read-date is called with t as its to-time argument (the second
>> argument), so that the date is returned in Emacs' internal time
>> representation, which doesn't represent ranges.
>
> Hmm.  Is it really about the TO-TIME argument?  If org-read-date is
> called with TO-TIME as nil, doesn't it still throw away the end of the
> range?
>
>   (let ((org-time-was-given nil)
> (org-end-time-was-given nil))
> (org-read-date nil nil nil "Date for tree entry:"))
>   ;; enter "3pm-4pm" => "2021-02-01 15:00"
>
> Or, actually, it stores the end in org-end-time-was-given, but it does
> that regardless of the TO-TIME argument.

Ah, that's interesting, thanks for pointing it out. I thought I had
checked to see if org-end-time-was-given had a value after org-read-date
gets called for a capture :time-prompt, but now I see that of course
that wouldn't work without the surrounding let binding...   

OK, so with your patch, the way is clear to stick this value in
org-capture-plist and make use of it when filling %T and friends in
templates. I'll see if I can come up with a patch to do that, and
report back.

-- 
Best,
Richard

Re: [PATCH] capture: Fix handling of time range for :time-prompt

[Richard Lawrence]

Dear Kyle and all,

Thanks for following up! Sorry it's taken me so long to reply.

Kyle Meyer  writes:

> Testing that against the cases in your initial report, I believe it
> leads to the same results as your patch, so here's a cleaned-up version.

I confirm that this cleaned up version works for me and gets the same
results for my test cases. I think it should be applied (modulo one
nitpick below). Are you willing to apply it, Kyle? I don't have commit
rights myself.

> -- >8 --
> Subject: [PATCH] capture: Fix handling of time range for :time-prompt
>
> * lisp/org-capture.el (org-capture-set-target-location): Bind
> org-end-time-was-given around the org-read-date call to get a return
> value that uses the start time rather than doing custom adjustment of
> the return value.
>
> If org-capture-set-target-location detects that the answer to
> org-read-date has a time range, it strips the end time from the answer
> and calls org-read-date-analyze again.  (org-read-date already calls
> it underneath.)  The regexp it uses, however, can easily match a date,
> leading to a bogus result.
>
> org-read-date-analyze is already capable of processing the time range
> in a way that matches org-capture-set-target-location's intent: when
> org-end-time-was-given is bound, org-read-date-analyze splits off the
> end value of the range and stores it in org-end-time-was-given.  Drop
> the custom logic and let org-read-date-analyze handle the range.
>
> Reported-by: Richard Lawrence 
> Ref: https://orgmode.org/list/87h7obh4ct.fsf@aquinas
> ---
>  lisp/org-capture.el | 35 +++
>  1 file changed, 15 insertions(+), 20 deletions(-)
>
> diff --git a/lisp/org-capture.el b/lisp/org-capture.el
> index f40f2b335..d7b69f228 100644
> --- a/lisp/org-capture.el
> +++ b/lisp/org-capture.el
> @@ -1025,28 +1025,23 @@ (defun org-capture-set-target-location ( 
> target)
>  (time-to-days org-overriding-default-time))
> ((or (org-capture-get :time-prompt)
>  (equal current-prefix-arg 1))
> -;; Prompt for date.
> -(let ((prompt-time (org-read-date
> -nil t nil "Date for tree entry:")))
> +   ;; Prompt for date.  Bind `org-end-time-was-given' so
> +   ;; that `org-read-date-analyze' handles the time range
> +   ;; case and returns `prompt-time' with the start value.
> +   (let* ((org-time-was-given nil)
> +  (org-end-time-was-given nil)
> +  (prompt-time (org-read-date
> + nil t nil "Date for tree entry:")))
>(org-capture-put
> :default-time
> -   (cond ((and (or (not (boundp 'org-time-was-given))
> -   (not org-time-was-given))
> -   (not (= (time-to-days prompt-time) (org-today
> -  ;; Use 00:00 when no time is given for another
> -  ;; date than today?
> -  (apply #'encode-time 0 0
> - org-extend-today-until
> - (cl-cdddr (decode-time prompt-time
> - ((string-match "\\([^ ]+\\)-[^ ]+[ ]+\\(.*\\)"
> -org-read-date-final-answer)
> -  ;; Replace any time range by its start.
> -  (apply #'encode-time
> - (org-read-date-analyze
> -  (replace-match "\\1 \\2" nil nil
> - org-read-date-final-answer)
> -  prompt-time (decode-time prompt-time
> - (t prompt-time)))
> +  (if (and (or (not org-time-was-given))

Nitpick: (or (not org-time-was-given)) is equivalent to (not org-time-was-given)

> +   (not (= (time-to-days prompt-time) (org-today
> +  ;; Use 00:00 when no time is given for another
> +  ;; date than today?
> +  (apply #'encode-time 0 0
> + org-extend-today-until
> + (cl-cdddr (decode-time prompt-time)))
> +prompt-time))
>(time-to-days prompt-time)))
> (t
>  ;; Current date, possibly corrected for late night
>
> base-commit: 9e8215f4a5df7d03ac787da78d28f69a4c18e7d3

As for this:

> The original change came in b61ff117b (org-capture.el:
> Set a correct time value with file+datetree+prompt, 2012-09-24), and I
> believe the related thread is
>

Re: clock-table and hooking that into org-capture file+olp+datetree

[Richard Lawrence]

Hi Christopher and all,

"Christopher Causer"  writes:

>> Note that org-datetree-find-date has a slightly annoying interface, in
>> that you need to provide a list of three integers representing a
>> calendar date. 
>
> Yes, that is a little awkward. What I did think of using was substrings to 
> extract the date from the picker interface.
>
> #+BEGIN_SRC emacs-lisp
> (defun org-date-picker-to-list ()
>   (let* ((date-string (org-read-date))
>(year (substring date-string 0 4))
>(month (substring date-string 5 7))
>(day (substring date-string 8 10)))
> (mapcar 'string-to-number (list month day year
> #+END_SRC
>
> Does that look sensible? 

As sensible as anything :) 

I recently added my version, using decode-time and nth, as a helper
function to my .emacs. I feel sure that it must already exist
*somewhere* in Emacs but I searched quite a while for it and didn't find
it. I will use this again below:

#+begin_src emacs-lisp 
  (defun time-as-calendar-date (time)
"Convert time in Emacs' time format to a calendar date list (MONTH DAY 
YEAR)"
(let ((parsed-time (decode-time time)))
  (list
   (nth 4 parsed-time)
   (nth 3 parsed-time)
   (nth 5 parsed-time
#+end_src

> My two next things to tackle are 
>
> 1. A hook to run the function when I run org-capture.

Here you might find it useful to grab the value of the :default-time key
from org-capture-plist (which should contain either the time you entered
at the date prompt during capture, or the current time, in Emacs' time
format). Then, using the functions above, you can say something like
this in your hook:

#+being_src emacs-lisp
(let* ((default-time (plist-get org-capture-plist :default-time))
   (date (time-as-calendar-date default-time)))
  (org-update-clocktable-on-date date))
#+end_src

> 2. Changing the org-clock-report options in your function above, but not the 
> defaults.
>
> For the second point, is there some trick to swap a global variable for the 
> run of a function? The variable in this case would be org-clocktable-defaults.

Emacs Lisp has dynamic scope by default, which makes this is pretty easy
in general: just set the value you want to use in a let form around the
code that uses this variable, like:

(let ((org-clocktable-defaults your-custom-value-here))
  ...)

> Thank you so much Richard. You've probably saved me days of going through the 
> org-mode documentation and source. 

No problem! That's what this list is for. I recently spent a fair amount
of time digging through the datetree stuff myself, so I was glad to have
a chance to share what I learned.

-- 
Best,
Richard

Re: clock-table and hooking that into org-capture file+olp+datetree

[Richard Lawrence]

Hi Christopher,

"Christopher Causer"  writes:

> Hello everyone! Here's a reasonably easy (I think) question because I'm quite 
> new to Emacs and org-mode.
>
> I have an org-capture template using file+olp+datetree[1], which works great 
> at filing my thoughts for the day. Separately I know I can generate clock 
> tables[2] based on dynamic blocks to show me what I've been doing with my 
> time for any given period. What I'm struggling with is to glue parts of these 
> together to achieve the following:
>
> 1. I org-capture to a subheading of datetree. When it does so it either 
> creates or updates an org-clock-report just below the datetree header  (the 
> bit that says "2020-11-12 Thursday", for example.) I guess this would be the 
> parent of what I'm capturing.
>
> 2. For all my historical journal entries, if I could move point to a headline 
> with a date such as the example below and it would pull the date out and add 
> a clocktable below via an interactive function that would be my ideal. This 
> is less of a problem for me as I don't have much in the way of history in my 
> diary yet or my other org files.
>

If I understand right, what you need for both of these things is a
function to jump to a date in your diary datetree and update the
clocktable there. Right?

Some functions that will help with this:
- org-datetree-find-date-create
- org-narrow-to-subtree

So, something like this should get you started:

#+begin_src emacs-lisp
(defun org-update-clocktable-on-date (date)
  (save-excursion
;; open the file containing the datetree:
(find-file (concat org-directory "/diary.org"))
;; jump to the subtree for the given date:
;; note: date must look like (m d y) where all three values are integers
(org-datetree-find-date-create date)
;; narrow to the subtree for this date, so we don't update
;; any other clocktables
(org-narrow-to-subtree)
;; update the clock report, or create it if it doesn't exist
;; note: we pass a prefix argument to tell org-clock-report to
;; update the first clocktable it finds in the (narrowed) buffer
(org-clock-report t)
;; widen to the whole buffer again
(widen)))
#+end_src

Then you can call this function, providing the date, in different
contexts where you want to create or update the clocktable.

Note that org-datetree-find-date has a slightly annoying interface, in
that you need to provide a list of three integers representing a
calendar date. One easy way to do that interactively is with
calendar-read-date, which prompts you for the year, month and day, so
you could say

(org-update-clocktable-on-date (calendar-read-date))

calendar-read-date is not as nice to use interactively as org-read-date,
but as far as I know, there is no easy way to get the calendar (m d y) format
out of its return value, which is either a string like "2021-01-30" or a
value in Emacs' internal time representation format.  But you can do
something like

(let*
;; prompt for the date and decode the resulting internal time as a list:
((decoded (decode-time (org-read-date nil t nil "Update on date:")))
;; unpack the date as a list (m d y) from the decoded time:
 (date (list (nth 4 decoded) ; month
 (nth 3 decoded) ; day
 (nth 5 decoded ; year

  (org-update-clocktable-on-date date))

Hope that helps get you to your next step!

-- 
Best,
Richard

[PATCH] incorrect timestamps with :time-prompt and datetrees

[Richard Lawrence]

Hi everyone,

Bumping this, since I forgot to put "PATCH" in the subject line before.

Richard Lawrence  writes:

> Here is a patch for this issue. It uses a narrower regex to match a time
> range. This regex requires time ranges to have ":MM" or an AM/PM
> specification in the end time, to prevent mangling strings that are
> interpreted as dates, like "11-12".
>
> This patch is a minimal change that gets the code working in the way
> that seems to have been intended, so it seems worth applying to maint.
>
> However, the way the code is intended to work doesn't seem right to me,
> because it simply throws away time range information at the time prompt.
> If you enter a time range like "13:00-14:00" at the time prompt, you
> will get a timestamp with "13:00" for the time when the %T template is
> expanded. (This is because org-capture-set-target-location uses the
> beginning of the entered time range to set :default-time, which must be
> an encoded time value, and there is no obvious way to set a time range.)
> This is a surprising contrast with the behavior of %^T, which preserves
> the time range information in the timestamp entered. But fixing this
> will be a larger change and possibly requires some discussion.

-- 
Best,
Richard

>From a6c223664aad6096943e236c9a51c30246e57669 Mon Sep 17 00:00:00 2001
From: Richard Lawrence 
Date: Wed, 6 Jan 2021 11:53:42 +0100
Subject: [PATCH] org-capture: fix expansion of %T when capturing to a datetree

* org-capture.el (org-capture-set-target-location): Use a narrower
regular expression to replace a time range by its start time when
setting :default-time, so that dates do not get mangled.

TINYCHANGE
---
 lisp/org-capture.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/org-capture.el b/lisp/org-capture.el
index f40f2b335..df0eccdbb 100644
--- a/lisp/org-capture.el
+++ b/lisp/org-capture.el
@@ -1038,12 +1038,12 @@ Store them in the capture property list."
 			 (apply #'encode-time 0 0
 org-extend-today-until
 (cl-cdddr (decode-time prompt-time
-			((string-match "\\([^ ]+\\)-[^ ]+[ ]+\\(.*\\)"
+			((string-match "\\(--?\\([012]?[0-9]\\)\\(\\(:[0-5][0-9]\\)\\|\\(am\\|AM\\|pm\\|PM\\)\\>\\)\\)\\(.*\\)"
    org-read-date-final-answer)
 			 ;; Replace any time range by its start.
 			 (apply #'encode-time
 (org-read-date-analyze
- (replace-match "\\1 \\2" nil nil
+ (replace-match "\\6" nil nil
 		org-read-date-final-answer)
  prompt-time (decode-time prompt-time
 			(t prompt-time)))
-- 
2.20.1

Re: Bug: incorrect timestamps with :time-prompt and datetrees

[Richard Lawrence]

Hi everyone,

Richard Lawrence  writes:

> I can now confirm that this is the problem. It looks like what is happening 
> here
> is that the regex is meant to match a time range, but ends up matching
> the date: thus a string like "12-31 13:00" gets mangled to "12 13:00"
> and sent into org-read-date-analyze.

> So I guess the solution is...a better regex is needed to cause this
> branch to fire only in the intended case, namely, a time range. 

Here is a patch for this issue. It uses a narrower regex to match a time
range. This regex requires time ranges to have ":MM" or an AM/PM
specification in the end time, to prevent mangling strings that are
interpreted as dates, like "11-12".

This patch is a minimal change that gets the code working in the way
that seems to have been intended, so it seems worth applying to maint.

However, the way the code is intended to work doesn't seem right to me,
because it simply throws away time range information at the time prompt.
If you enter a time range like "13:00-14:00" at the time prompt, you
will get a timestamp with "13:00" for the time when the %T template is
expanded. (This is because org-capture-set-target-location uses the
beginning of the entered time range to set :default-time, which must be
an encoded time value, and there is no obvious way to set a time range.)
This is a surprising contrast with the behavior of %^T, which preserves
the time range information in the timestamp entered. But fixing this
will be a larger change and possibly requires some discussion.

-- 
Best,
Richard

>From a41c751f15488a8b0d48d6b1b1744dbbc003f9f0 Mon Sep 17 00:00:00 2001
From: Richard Lawrence 
Date: Wed, 6 Jan 2021 11:53:42 +0100
Subject: [PATCH] org-capture: fix expansion of %T when capturing to a datetree

* org-capture.el (org-capture-set-target-location): Use a narrower
regular expression to replace a time range by its start time when
setting :default-time, so that dates do not get mangled.

TINYCHANGE
---
 lisp/org-capture.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/org-capture.el b/lisp/org-capture.el
index f40f2b335..df0eccdbb 100644
--- a/lisp/org-capture.el
+++ b/lisp/org-capture.el
@@ -1038,12 +1038,12 @@ Store them in the capture property list."
 			 (apply #'encode-time 0 0
 org-extend-today-until
 (cl-cdddr (decode-time prompt-time
-			((string-match "\\([^ ]+\\)-[^ ]+[ ]+\\(.*\\)"
+			((string-match "\\(--?\\([012]?[0-9]\\)\\(\\(:[0-5][0-9]\\)\\|\\(am\\|AM\\|pm\\|PM\\)\\>\\)\\)\\(.*\\)"
    org-read-date-final-answer)
 			 ;; Replace any time range by its start.
 			 (apply #'encode-time
 (org-read-date-analyze
- (replace-match "\\1 \\2" nil nil
+ (replace-match "\\6" nil nil
 		org-read-date-final-answer)
  prompt-time (decode-time prompt-time
 			(t prompt-time)))
-- 
2.20.1

Re: Bug: incorrect timestamps with :time-prompt and datetrees

[Richard Lawrence]

Richard Lawrence  writes:

> This is org 9.4 running from maint (commit ab00524fc). I spent a while
> stepping through org-capture and org-read-date but haven't found the
> problem yet. I suspect this snippet from a cond form in the middle of
> org-capture-set-target-location:
>
> #+begin_src
> ((string-match "\\([^ ]+\\)-[^ ]+[ ]+\\(.*\\)"
> org-read-date-final-answer)
> ;; Replace any time range by its start.
> (apply #'encode-time
> (org-read-date-analyze
> ;; it looks to me like this is maybe sending the wrong value
> ;; into org-read-date-analyze: 
> (replace-match "\\1 \\2" nil nil  
>org-read-date-final-answer)
>
> #+end_src

I can now confirm that this is the problem. It looks like what is happening here
is that the regex is meant to match a time range, but ends up matching
the date: thus a string like "12-31 13:00" gets mangled to "12 13:00"
and sent into org-read-date-analyze.  If I comment out this branch of
the cond form, and enter "12-31 13:00" at the org-read-date-prompt, then
both the timestamp and the location in the datetree are correct.

So I guess the solution is...a better regex is needed to cause this
branch to fire only in the intended case, namely, a time range. Should
it be checking for "am"/"pm" or ":" or similar? Otherwise it seems
impossible to distinguish a date specification like "11-12" from a time
range. 

-- 
Best,
Richard

Bug: incorrect timestamps with :time-prompt and datetrees

[Richard Lawrence]

Hi all,

I ran into a subtle bug yesterday. Basically, when using org-capture to
capture

  - an entry into a datetree,
  - on a date other than today (using :time-prompt in org-capture-templates)
  - with a capture template that inserts a timestamp (%T)

then I get incorrect results for either the timestamp or the location in
the datetree, depending on how I enter the date.

Here is a minimal working example:

#+begin_src emacs-lisp
  (setq org-capture-templates
  `(
("d" "Diary")
("da" "Appointment/Event" entry
 (file+olp+datetree "/tmp/diary.org")
 " %^{Description}\n %T"
 :time-prompt t)))
#+end_src
Notice that the template contains %T, to expand to a timestamp with a
time, and also that the capture target is a datetree and :time-prompt is
true. My understanding is that this should both file the entry to the
entered date in the datetree, and fill the %T template with a timestamp
for the entered date (and time, if any).
  
Here are the results from running a few captures with this setup on
Dec 24 around 11AM. I tried various ways of scheduling an event on Dec 31
at 13:00; what I entered into the prompt of org-read-date is shown in
the headline:

#+begin_example

* 2020

** 2020-12 December

*** 2020-12-24 Thursday
 Entered "31-12" 
 <2020-12-31 Thu 11:06>

This gets filed to the wrong day in the datetree, but the date in the
timestamp is correct. The time is also correct (it's the current
time, since no time was entered).

 Entered "31-12 13:00" 
 <2020-12-31 Thu 13:00>

This gets filed to the wrong day in the datetree, but the date and time
in the timestamp are correct.
 
*** 2020-12-31 Thursday
 Entered "12-31" 
 <2020-12-31 Thu 00:00>

This gets filed to the correct day in the datetree and the date in the
timestamp is correct. The time is 00:00 because no time was entered.
(Why isn't the time 11:06, though, like in the first example?)

 Entered "12-31 13:00" 
 <2021-01-12 Tue 13:00>

This is the most problematic case: it gets filed to the correct day in
the datetree, but the date in the timestamp is incorrect!

 Entered "2020-12-31 13:00" 
 <2020-12-31 Thu 13:00>

If I enter the date in full ISO format, the location in the datetree and
the timestamp are both correct.

#+end_example

Possibly relevant here is the value of calendar-date-style, which is
'american for me. I tested briefly with 'european but it did not make a
difference for the "31-12" cases.

This is org 9.4 running from maint (commit ab00524fc). I spent a while
stepping through org-capture and org-read-date but haven't found the
problem yet. I suspect this snippet from a cond form in the middle of
org-capture-set-target-location:

#+begin_src
((string-match "\\([^ ]+\\)-[^ ]+[ ]+\\(.*\\)"
org-read-date-final-answer)
;; Replace any time range by its start.
(apply #'encode-time
(org-read-date-analyze
;; it looks to me like this is maybe sending the wrong value
;; into org-read-date-analyze: 
(replace-match "\\1 \\2" nil nil  
   org-read-date-final-answer)

#+end_src

Will report here if I find out more exactly.
Happy holidays, everyone!

-- 
Best,
Richard

Re: typos in doc.org

[Richard Lawrence]

hj-orgmod...@hj.proberto.com writes:

> https://www.orgmode.org/worg/sources/doc.org :
>
> availables -> available ,  uesr -> user :

Fixed - thanks for the report!

-- 
Best,
Richard

Re: Bug: unexpected new page started after bullet line [9.1.9 (release_9.1.9-65-g5e4542 @ /usr/share/emacs/26.3/lisp/org/)]

[Richard Lawrence]

Hi Rob,

Rob Sargent  writes:

> This construct causes a new page to be started
> * head
> ** subhead
> - bullet

... in which export backend? You say you are using LibreOffice, but you
provided an example Org file with many LaTeX-related options. Do you see
the page break in a LibreOffice document? in a LaTeX-compiled PDF? both?

If in LibreOffice, do you see the same behavior if you don't have all
the LaTeX-related options in place?

FWIW, I tried exporting this construct to both LaTeX and ODT *without*
the LaTeX options on Org 9.4 and do not see the page breaks you are
seeing.

-- 
Best,
Richard

Re: Customize resheduling behaviour

[Richard Lawrence]

Hi Stefan,

Stefan Huchler  writes:

> I have a task sheduled like that:
>
> * TODO  Order from a delivery service
>   SCHEDULED: <2020-10-06 Di .+2w>
>
> ...
> Is there a way to reshedule it, if it fails 1 week later and if I
> complete it, reshedule 2 weeks later?

I don't think there's any way to do this automatically when you change
the TODO state of the entry.

The simplest way I can think of is to do C-c C-s +7 (i.e., call
org-schedule and give a new date a week later at the prompt). If doing
this on a different day than the scheduled day, you'll need to change
the interval you enter (e.g., if you missed it by 2 days, enter +5) or
type in the new date manually.

Another option is C-c C-x c (i.e., org-clone-subtree-with-time-shift).
Clone the tree one time, with a shift of one week, then delete the old
tree. The advantage of this option is that the time shift will be
relative to the old scheduled/deadline date, not the date on which you
run the command, so you can always say +1w.

Hope that helps!

-- 
Best,
Richard

Re: Bug: Math mode doesn't work if followed by a dash [9.4 (nil @ /home/gutin/.emacs.d/.local/straight/build/org-mode/)]

[Richard Lawrence]

"Berry, Charles"  writes:

> The case Gutin describes conforms to the documentation, viz. `$x\beta$-` 
> should produce math mode LaTeX as I read the next paragraph. 
>
> From (info "(org) LaTeX fragments"):
>
>• Text within the usual LaTeX math delimiters.  To avoid conflicts
>  with currency specifications, single ‘$’ characters are only
>  recognized as math delimiters if the enclosed text contains at most
>  two line breaks, is directly attached to the ‘$’ characters with no
>  whitespace in between, and if the closing ‘$’ is followed by
>  whitespace, punctuation or a dash.

Hmm, good point.

It looks to me like the relevant function is
org-element-latex-fragment-parser, and the code for that hasn't changed
much in several years (last change was 2017). (Also, I was wrong,
parsing latex fragments is not done with just a regexp.) I can confirm
that this function parses "$foo" in "$foo$ bar" as a latex fragment, but
not in "$foo$-bar"

Not sure if the problem here is the code or the documentation. Perhaps
the documentation should be updated to reflect the current behavior?

-- 
Best,
Richard

Re: Bug: Math mode doesn't work if followed by a dash [9.4 (nil @ /home/gutin/.emacs.d/.local/straight/build/org-mode/)]

[Richard Lawrence]

Hi Gutin,

gutin  writes:

> What I meant is that if you type
>
>   $*$-algebra
>
> and hit C-c C-x C-l, then the "$*$" doesn't get replaced with a
> mathematical image. A similar problem happens when you export to Latex:
> The dollar signs get escaped.

I believe this is intentional. There are too many other uses of dollar
signs that people might want to use in Org documents that might be
broken if Org treated them as math mode delimiters; so the regexp that
matches math mode delimiters is deliberately limited in scope.

The recommended workaround is to use the \(...\) style of delimiter for
math mode instead, which is much less ambiguous. Here is a function that
makes this easy, by inserting these delimiters when you type "$". It's
been posted on this mailing list before, and I use a version of it
myself; it's very useful!

   ;; from Nicolas Richard 
   ;; Date: Fri, 8 Mar 2013 16:23:02 +0100
   ;; Message-ID: <87vc913oh5@yahoo.fr>
   (defun yf/org-electric-dollar nil
 "When called once, insert \\(\\) and leave point in between.
   When called twice, replace the previously inserted \\(\\) by one $."
  (interactive)
  (if (and (looking-at ")") (looking-back "("))
  (progn (delete-char 2)
 (delete-char -2)
 (insert "$"))
(insert "\\(\\)")
(backward-char 2)))
   (define-key org-mode-map (kbd "$") 'yf/org-electric-dollar

Hope that helps!

-- 
Best,
Richard

Re: Bug: Math mode doesn't work if followed by a dash [9.4 (nil @ /home/gutin/.emacs.d/.local/straight/build/org-mode/)]

[Richard Lawrence]

Hi Gutin,

gutin  writes:

> If a pair of dollar signs is followed by a dash, then math mode doesn't
> work.

Can you say a bit more about what you are trying to do, what you are
expecting to happen, and what you are seeing instead?

Since you refer to math mode, I assume you are exporting to LaTeX. I
just tested this:

some text
$$-1$$
other text

and it exports fine to LaTeX on my machine (Emacs 26.1, Org 9.4).
What's broken for you?

-- 
Best,
Richard

Re: newline or no newline at end of capture: expected behavior

[Richard Lawrence]

Hi Samuel,

> On 9/23/20, Samuel Wales  wrote:
>> i have the same question for headline-only capture
>> buffers.  i.e. what is the expected result with and without trailing
>> newline.

Just to clarify: are you aware of the :empty-lines, :empty-lines-before
and :empty-lines-after keys that you can set in org-capture-templates?
Are your questions about the default values of these keys?

I just tried a simple setup in emacs -Q with Org 9.4, namely:

(setq org-capture-templates
  '(("s" "simple" entry (file+headline "~/org/notes.org" "Foo")
  "* %?" :empty-lines 1)))

and Org seems to correctly respect the value of these keys when they are
specified explicitly. That is, with :empty-lines 0, no newlines are
added; with :empty-lines 1, a blank line appears on each side of the
captured entry; with :empty-lines-after 1, a blank line appears after,
but not before, the new entry; and so on.

Can you use these keys to get the behavior you are looking for? Or is
your question about something else?

-- 
Best,
Richard

Re: Get Grades Done: the joys of Org's simple power

[Richard Lawrence]

Hi Devin,

Devin Prater  writes:

> Ah, I’m using Safari on MacOS 10.15. I can try with Chrome as well, though.

Did it work in Chrome, and/or when you moved the script to the end of
the file?

For what it's worth, here is a new version that should work better
regardless of where it's placed in the document.  The only change is
that the code which replaces the text representation of the checkboxes
with HTML checkboxes has been wrapped in a function that fires on the
"DOMContentLoaded" event.

Hope that's helpful!

-- 
Best,
Richard

#+begin_export html

  function updateCookiesIn(div) {
  const headline = div.querySelector("h1, h2, h3, h4, h5, h6");
  if (!headline) return;
  const cookies = Array.from(headline.querySelectorAll("code"))
.filter(c => c.innerText.startsWith("[") && 
c.innerText.endsWith("]"));
  const fracCookies = cookies.filter(c => c.innerText.includes("/"));
  const pctCookies = cookies.filter(c => c.innerText.includes("%"));

  // The ugly query strings here restrict the selection to checkboxes at 
*this* level of the hierarchy
  const allTasks = div.querySelectorAll(`#${div.id} > div > ul 
input[type=checkbox], #${div.id} > div > ol input[type=checkbox]`);
  const completedTasks = div.querySelectorAll(`#${div.id} > div > ul 
input[type=checkbox]:checked, #${div.id} > div > ol 
input[type=checkbox]:checked`);

  const newFrac = `[${completedTasks.length}/${allTasks.length}]`;
  const newPctText = allTasks.length
? (100 * completedTasks.length / allTasks.length).toFixed(0)
: "100"; // Org shows 100% for a cookie when there are no 
checkboxes 
  const newPct = `[${newPctText}%]`;

  fracCookies.forEach(c => c.innerText = newFrac);
  pctCookies.forEach(c => c.innerText = newPct);
  }

  function replaceWithCheckbox(code) {
  const isChecked = code.innerText.includes("X");

  const checkbox = document.createElement("input");
  checkbox.setAttribute("type", "checkbox");
  if (isChecked) checkbox.setAttribute("checked", "checked");
  checkbox.onclick = function (e) {
  const container = findContainingSection(e.target);
  if (!container) return;
  updateCookiesIn(container);
  };

  code.replaceWith(checkbox);
  }

  function findContainingSection(el) {
  if (!el.parentElement) return null;

  const parent = el.parentElement;
  const classes = Array.from(parent.classList);
  if (classes.some(cl => cl.startsWith("outline") && 
!cl.startsWith("outline-text"))) {
  return parent;
  } else {
  return findContainingSection(parent);
  }
  }

document.addEventListener("DOMContentLoaded", function(event) { 
  const orgCheckboxes = document.querySelectorAll(".off > code, .on > code");
  orgCheckboxes.forEach(replaceWithCheckbox);
  
  const orgSections = document.querySelectorAll("div.outline-1, div.outline-2, 
div.outline-3, div.outline-4, div.outline-5, div.outline-6");
  orgSections.forEach(updateCookiesIn);
});


#+end_export

Re: Get Grades Done: the joys of Org's simple power

[Richard Lawrence]

Hi Devin and all,

Devin Prater  writes:

> Yeah, I was hoping to just have an HTML page or something that could
> be put on Github or just sent in an email attachment, where checking
> checkboxes would update the fraction cookie.

I hacked together a quick solution for this. You should be able to just
drop this Javascript into an Org file. When it is included in HTML that
has been exported via the standard Org exporter, it replaces the static
"[ ]" and "[X]" code elements with actual HTML checkboxes, and updates
any progress cookies in the relevant headlines when those boxes are
checked or unchecked.

I haven't tested it extensively, and the code can surely be improved,
but it works for the cases I could think to test.

#+begin_export html

  function updateCookiesIn(div) {
  const headline = div.querySelector("h1, h2, h3, h4, h5, h6");
  if (!headline) return;
  const cookies = Array.from(headline.querySelectorAll("code"))
.filter(c => c.innerText.startsWith("[") && 
c.innerText.endsWith("]"));
  const fracCookies = cookies.filter(c => c.innerText.includes("/"));
  const pctCookies = cookies.filter(c => c.innerText.includes("%"));

  // The ugly query strings here restrict the selection to checkboxes at 
*this* level of the hierarchy
  const allTasks = div.querySelectorAll(`#${div.id} > div > ul 
input[type=checkbox], #${div.id} > div > ol input[type=checkbox]`);
  const completedTasks = div.querySelectorAll(`#${div.id} > div > ul 
input[type=checkbox]:checked, #${div.id} > div > ol 
input[type=checkbox]:checked`);

  const newFrac = `[${completedTasks.length}/${allTasks.length}]`;
  const newPctText = allTasks.length
? (100 * completedTasks.length / allTasks.length).toFixed(0)
: "100"; // Org shows 100% for a cookie when there are no 
checkboxes 
  const newPct = `[${newPctText}%]`;

  fracCookies.forEach(c => c.innerText = newFrac);
  pctCookies.forEach(c => c.innerText = newPct);
  }

  function replaceWithCheckbox(code) {
  const isChecked = code.innerText.includes("X");

  const checkbox = document.createElement("input");
  checkbox.setAttribute("type", "checkbox");
  if (isChecked) checkbox.setAttribute("checked", "checked");
  checkbox.onclick = function (e) {
  const container = findContainingSection(e.target);
  if (!container) return;
  updateCookiesIn(container);
  };

  code.replaceWith(checkbox);
  }

  function findContainingSection(el) {
  if (!el.parentElement) return null;

  const parent = el.parentElement;
  const classes = Array.from(parent.classList);
  if (classes.some(cl => cl.startsWith("outline") && 
!cl.startsWith("outline-text"))) {
  return parent;
  } else {
  return findContainingSection(parent);
  }
  }

  const orgCheckboxes = document.querySelectorAll(".off > code, .on > code");
  orgCheckboxes.forEach(replaceWithCheckbox);
  
  const orgSections = document.querySelectorAll("div.outline-1, div.outline-2, 
div.outline-3, div.outline-4, div.outline-5, div.outline-6");
  orgSections.forEach(updateCookiesIn);

#+end_export

Hope that helps!

-- 
Best,
Richard

Re: Get Grades Done: the joys of Org's simple power

[Richard Lawrence]

Hi Devin,

Devin Prater  writes:

> I tried that on my file, but the checkboxes didn’t update. I’ll give you the 
> kind of file I’m working with:

OK, sorry about that!  Two things:

1) I see now that my code assumes the script block runs immediately, so
it doesn't work if the script runs before the document is completely
loaded. I'll fix that and send an update.

2) When I move the script to the end of your file, it works here for me
(on Firefox Nightly). But it could also be that there are browser
differences getting in the way; I used some newer Javascript features, I
think. What browser are you using?

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

Nicolas Goaziou  writes:

> I think there are really two paths here: either we only support the
> common denominator between all processors, like, e.g., Pandoc, or we
> handle every possible command, knowing that most of them will not be
> portable anyways.

Yes, I think that is the core issue: which way to do we want to go here?

My opinion has always been that it makes more sense to just support the
common denominator at the level of Org citation syntax, for two reasons:
(1) it makes implementing a good solution that will work for a lot of
cases much more feasible, and (2) anyone who really needs more than the
common denominator -- that is to say, anyone who needs BibLaTeX -- can
already write arbitrary LaTeX snippets directly in an Org document. Thus
the latter group doesn't really lose anything if the syntax only
supports the common denominator, while everyone else has a lot to gain
from an implementation of citation syntax that can be exported on other
backends.

On the other hand, this opinion is narrowly focused on the issue of how
citation syntax can be rendered into citations when exporting a
document. When I think about the other uses for the syntax that e.g.
John Kitchin has talked about in this thread, and that something like a
future org-ref could support, then I see that people who need to export
citations as BibLaTeX *would* still be missing out if they couldn't use
the citation syntax.

So, I think it is good if the syntax can support advanced BibLaTeX users
too, and it looks to me like the "cite/xxx" syntax can do that. I have
no objections to the syntax you've proposed, Nicolas.

I *do* think it's worth marking a clear distinction between citation
syntax that can vs. cannot be expected to export correctly on non-LaTeX
backends. It looks to me at the moment like that distinction will be
expressed as the difference between "cite" and "cite/xxx". For me,
that's a reason to make "cite/text" a special case and give it its own
syntax, since this is such an important and widespread use case, and CSL
supports it. But I don't feel that strongly about this. For me, it would
be fine if it's mentioned as a special case in the manual.

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

"Bruce D'Arcus"  writes:

>> > My understanding, though, is that org "cite" would default to your
>> > last example I quote above (in natibib, citep); that there's no need
>> > for a dedicated "cite/paren" command, either reserved or not.
>>
>> Not necessarily. "cite" means default value, whatever that is. It could,
>> for example, mean: "cite/text" for every citation, if that is what you
>> use the most. In that case, "cite/paren" is necessary, to override it
>> locally. It could also be, e.g., "cite/footnote", then both "cite/text"
>> and "cite/paren" could be of some use. That was suggested by Richard
>> Lawrence in this thread, if my memory serves me right.
>>
>> Does that make sense?
>
> I think so. I'll defer to Richard on this, since he was making this point.

Sorry to take so long to reply. The point I made earlier was that, as
far as I understand, the choice of CSL stylesheet is the main factor
determining how a given citation gets rendered into the output (assuming
you process citations with CSL). So yes, it makes sense to have "cite"
mean default value as determined by the choice of stylesheet.

I've been skimming the CSL documentation, and I'm realizing that I
actually don't have a very good understanding of how these different
types of commands would be represented at the level of a CSL processor.

Bruce, is it possible to have a CSL stylesheet that would be able to
accommodate both e.g. "cite/paren" and "cite/footnote" in the same
document? Can a stylesheet support an arbitrary numbers of different
citation types like this, and can a CSL processor choose among those
types based on their *names*?

If so, then I think Nicolas' proposal to have "cite" mean default and
make non-default citations available as "cite/xxx" makes sense
(especially with the other syntax supporting suppress-author, etc.).

If not, then the "cite/xxx" syntax makes less sense to me; it just sort
of looks like a different way of writing BibLaTeX commands, and will be
hard to support when LaTeX is not the output format. I would be hesitant
in that case to make "cite/xxx" the standard way to express "this
citation should be rendered in manner xxx, instead of the default".

Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

"Bruce D'Arcus"  writes:

> I can't see that it's necessary to have a fourth, because I think the
> result of that would be this, which doesn't make any sense.
>
> 4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
> author-in-text+suppress-author command
>
> Let us know what you think?

I think this could sometimes make sense. Granted, it wouldn't be very
often, but if e.g. you are citing something inside a wider parenthetical
remark, like:

  (Blah blah. However, Doe showed that not-blah; see her -@doe17.)

I can imagine that some style guides might forbid putting nested
parentheses in that position, so having a way to render "2017" instead
of "(2017)" would be useful.

Another case: I can imagine citation styles that use e.g. a work's title
(instead of its year) as the non-author identifier, in which case it
would often make sense to say things like

  Doe depicts blah in her -@doe17

as a way to output things like

  Doe depicts blah in her /Wondrous Novel/

Again, I don't know how important this is, or how widely used it would
be, but those are at least a couple of possibilities.

On the other hand, I notice that pandoc does not distinguish these
cases, at least with the default citation style; pandoc renders both
-@doe17 and [-@doe17] like "(2017)", so maybe it's not that important.

> ... notwithstanding that, I think Nicolas' latest proposed syntax
> would support this anyway.
>
> [citet:-@doe17]

Great. No objections from this corner, then!

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

Hi Bruce and all,

"Bruce D'Arcus"  writes:

> Just to align what you're saying and what I'm saying:
>
> I see three commands in the pandoc syntax: standard/parenthetical,
> author-in-text, and suppress-author; that look like so:
>
> [@doe17]
> @doe17
> -@doe17
>
> Implicit in what you wrote is the last one is not needed.
>
> The question, then: Is that what you're saying; we don't need suppress-author?

Ah, no, I didn't intend it like that. I am not very familiar with the
implementation details of pandoc-citeproc and wasn't aware that
suppress-author was a different type of citation command. I was
(vaguely) thinking of the third case as a "variant" of an in-text
citation type, rather than a separate type.

Actually, the Pandoc example you give seems to support this way of
thinking about it:  

> Doe, by contrast, found negative results [-@doe17].

That is a fourth case, right? "[-@doe17]" is not equivalent to "-@doe17"?

In other words, what we have here are two orthogonal distinctions:
parenthetical vs. in-text, and normal vs. author-suppressed. So, at
least on my funny way of counting ;), that's two citation "types", with
two "variants" within each of those types.

> one of the CSL implementers (Frank Bennett) figured out how to make
> the above example an author-in-text variant, so that you don't need
> suppress-author, and the entire sentence is the citation.
>
> He did this by adding an optional "infix" variable to the citation.
>
> So in that example, you would have:
>
> - command: "author-in-text"
> - citekey: "doe17"
> - infix: "by contrast, found negative results"
>
> This is arguably an edge case, but it does relate to the question of
> whether we need two (standard and author-in-text) or three commands
> (adding the suppress-author).
>
> One could make the reasonable argument (I think, though not everyone
> would agree) that the workaround for the above example is to use
> author-in-text command but restructure the sentence:
>
> @doe17, by contrast, found negative results.
>
> From that perspective, I guess we indeed need only two commands:
> standard (parenthetical) and author-in-text.

This way of writing the sentence seems less obvious to me than the
pandoc syntax. It also has the potential disadvantage that the choice
between rendering

"Doe (2017), by contrast, found negative results"

and

"Doe, by contrast, found negative results (2017)"

now has to be made at the level of the stylesheet instead of at the
level of the sentence where that citation appears. My instinct is that
this choice is informed by individual writing style and better made at
the level of the sentence. But you probably have a better sense than I
do of whether this is something that should at least sometimes be
controlled by the stylesheet. (Are there e.g. journals that always want
in-text citations to look like the latter case? I have no idea.)

In any case, if I'm right that this choice is usually better made at the
sentence level, then I think the syntax needs to support all four cases.
.The pandoc syntax does this, and I think the suppress-author variation
is probably needed often enough that we should have something similar.

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

Joost Kremers  writes:

> Good points. I guess what this boils down to is whether Org wants 
> to be like LaTeX, where simple things are doable and complicated 
> things possible, or Pandoc, where simple things are simple indeed 
> and complicated things essentially impossible.
>
> To clarify: in LaTeX (biblatex) you can mix footnote and in-text 
> citations in a single document, Pandoc doesn't allow that. 
> Pandoc's functionality is sufficient for a great majority of 
> cases, but if you want or need to go beyond it, things get very 
> difficult.

Right. The Pandoc syntax trades some of the flexibility of (Bib)LaTeX
for the ability to render the citations it *does* support in a whole
bunch of non-LaTeX formats.

I personally think this is a good tradeoff, and one I would like to see
Org adopt. In both Org and Pandoc, you can use embedded LaTeX if you
need it. If you need the full power of BibLaTeX citations, then you are
confined to LaTeX export anyway, so you might as well just use BibLaTeX
commands in your document. But if you fall into the great majority of
use cases, you can use specialized citation syntax, and thereby get
reasonable behavior in other export formats too.

> My suggestion would still be not to hard-code a limit on possible 
> citation commands. Org itself should probably just provide the 
> basics, but users and add-on packages should be allowed to define 
> more specific commands with readable names and there should be a 
> well-defined interface for doing so (just like users and packages 
> can add new link types, for example).

I agree with this. I see no problem with having an analogue of
org-add-link-type for citations, and I think it's reasonable to have the
syntax allow for such extensions, so that e.g.

[cite/my-custom-cite-type: ...]

can still be recognized by the parser as a citation, which extensions
can then give a semantics to. But I think there needs to be a clear
syntactic delimitation between citations that are expected to work "out
of the box" (which to me primarily means: exported correctly in the
built-in backends) vs. those that need some additional extension to
export correctly or support additional behavior (which doesn't need to
be available on all backends, and could e.g. support BibLaTeX-only
users).  Otherwise the problem of getting citations working is too big a
project.

Best,
Richard

Re: wip-cite status question and feedback

[Richard Lawrence]

Hi all,

Nice to see this issue being discussed again!

I don't have a lot to add and at the moment I don't have a lot of time
to contribute, but I wanted to make one point about this issue:

Joost Kremers  writes:

> On Mon, Apr 13 2020, Nicolas Goaziou wrote:
>> denis.maier.li...@mailbox.org writes:
>>> What about allowing something more verbose? Perhaps
>>> "cite-intext:" or "cite:intext:"?
> [...]
>>> The simple syntax is great for most cases, but if you want to
>>> support some of those not so common biblatex commands, this might be
>>> better.
>>
>> Alphanumeric suffix provides 62 combinations, which should hopefully
>> be enough for any citation back-end out there (I'm looking at you
>> biblatex). It's not terribly readable, tho, as you point out.
>
> 62 combinations might sound like a lot, but if you want your cite
> commands to be mnemonic, you'll run out of options much more quickly.

This came up in the discussion in 2015, too. So maybe this can help
avoid repeating a long discussion about this:

I think it is worth pointing out to Bib(La)TeX users that it is useful
to avoid a proliferation of citation commands in Org syntax. The syntax
discussed so far achieves this by "factoring out" formatting information
that BibLaTeX puts into the command into other parts of the syntax and
into the choice of citation stylesheet. For example, instead of having
\footnotecite and \parencite as separate commands, you can just have a
single cite command, and the choice of stylesheet determines whether
citations get formatted as footnotes or as in-text parenthetical
citations or as something else.

This obviously has the drawback that if you only have single citation
command, you only get to make the choice about formatting once for the
whole document (via the stylesheet). So, I think the relevant question
is: how many different basic citation types are needed *within a single
document*, keeping in mind that these basic types will be formatted in
different ways, depending on the choice of stylesheet?

My experience is that it's typically just two (e.g. parenthetical and
author-in-text), and my memory of the earlier conversation was that most
people agreed. This is also borne out in the Pandoc syntax. As long as
we have two basic types of citations, the finer points of formatting
them can be achieved via other syntax, including the choice of
stylesheet.

-- 
Best,
Richard

Re: Release 9.3

[Richard Lawrence]

Dear Bastien and all,

Bastien  writes:

> I've released Org 9.3.
> ...
> As usual, this is the work of Nicolas acting as a maintainer
> and of many other contributors!

Thank you, Bastien, Nicolas, and everyone else! I haven't been very
active on the list of late, but I am using and loving Org as much as
ever. I appreciate all your hard work, and I wanted to say so.  Thanks,
and congratulations on this milestone!

-- 
Best,
Richard

Re: mobile org

[Richard Lawrence]

Neil Jerram  writes:

> If org-web and organice are browser-based, why do they need syncing?  Could
> the server be your regular non-mobile Org machine?

Presumably yes, if you're willing to keep it online and run a WebDAV
server on it.

> On Tue, 19 Nov 2019 at 08:57, Diego Zamboni  wrote:

>> Looking at the github history, organice seems to be a fork of org-web, but
>> with more recent development.

Yes, I think I saw somewhere that it's a "friendly" fork.

-- 
Best,
Richard

Re: mobile org

[Richard Lawrence]

Jude DaShiell  writes:

> Will mobile org work in Android PI?  If not, any good alternatives beyond
> orgsly available?  I don't like to have to use dropbox and would like to
> plug a usb cord into my computer and synchtronize my orgmode files that
> way if possible.

I saw this recently, and haven't seen it mentioned on the list:

https://organice.200ok.ch/

Here's an intro video: https://invidio.us/watch?v=aQKc0hcFXCk

It's a browser-based interface rather than an app, so it should work
just about anywhere. It doesn't solve the syncing problem itself;
instead it allows you to sign in with Dropbox, Google Drive, or WebDAV.
Still not great if you're trying to avoid the proprietary services and
sync with a cable, but it looks like an interesting alternative to
Orgzly.

-- 
Best,
Richard

Re: included text

[Richard Lawrence]

"Fraga, Eric"  writes:

> Mind you, an alternative could be #+CALL-ing a src block
> that generates the text as output?

On these lines: I've been thinking recently that it would be great to be
able to tangle/export other kinds of blocks (quotes, examples, etc.) via
the same kind of mechanism that src blocks use.

This would be really useful for me for making e.g. handouts from the
file that contains notes for a talk or a paper.

I've done this in the past with src blocks containing Org source code.
So instead of writing

#+begin_quote
Yada yada
#+end_quote

I write

#+begin_src org :tangle handout.org
,#+begin_quote
Yada yada
,#+end_quote
#+end_src

and create the handout.org file via org-babel-tangle.

This works but it is pretty clunky, and you lose a lot of the benefit of
having the original quote block. For one thing, it no longer exports as
a quote from the original file, but as source code. I'd rather just
write:

#+begin_quote :tangle handout.org
Yada yada
#+end_quote

or even better:

#+name: yada-quote
#+begin_quote
Yada yada
#+end_quote

and then in handout.org, something like:

#+include: talk.org::#yada-quote

Is there anyone who would be interested in this besides just me and
Samuel? (Can this already be done somehow in a non-clunky way?? I find
John's suggestion intriguing but too clunky: it seems like I shouldn't
have to pass a block of text through a shell or Python interpreter just
to include it verbatim in another Org file!)

-- 
Best,
Richard

Re: [O] CUSTOM_ID vs ID

[Richard Lawrence]

Hi Nate,

Nathan Neff  writes:

> However, I usually *do* manually assign IDs (not CUSTOM_IDs) myself.
> The reason is so I can reasonably search for the ID within my org
> files and that the link ID makes some sense to me.
>
> For example, if I see a link to "id:keyboard_shortcuts" I can tell where
> it's going.

If this is what you want IDs for, then you probably want CUSTOM_ID.
There is built-in Org syntax for links based on CUSTOM_ID: write your
links like [[#some-custom-id]] or [[file:foo.org::#some-custom-id]].

And if you use CUSTOM_ID, then if you ever need ID later (e.g., for
something like org-caldav), you won't have a collision between the two
different properties.

-- 
Best,
Richard

Re: [O] Show weekday in daily agenda view

[Richard Lawrence]

Hi Johanna,

"Prof. Dr. Johanna May"  writes:

> thanks for pointing me at the variable org-agenda-format-date. There
> was  a line in my dotemacs that included a formatting without the
> weekday (something like %y-%w-%d).

Ah, ok, great!

> I could not figure out how the name-of-the-weekday format would be
> called. But for now I'm fine with returning to the default which gives
> me an English date including the weekday.

You probably want "%a" or "%A" somewhere in there, if you want a
locale-specific name of the weekday.  See the docstring for the
format-time-string function.

-- 
Best,
Richard

Re: [O] Show weekday in daily agenda view

[Richard Lawrence]

Hi Johanna,

johanna@th-koeln.de writes:

> maybe I did not search the right way in google alias startpage. But I
> could not figure out how to write next to the date e.g. 2000-04-01 the
> weekday, i.e. the specific day of the week. It would come very handy to
> have the date displayed as 2019-05-14 Tu and 2019-05-15 We and so on. Or
> even have it spelled out as 2019-05-14 Tuesday.
>
> Probably there is a way of setting this. How does it work?

Hmm.  My daily agenda (the default one) starts like this:

Day-agenda (W20):
Tuesday14 May 2019

and tasks, etc. are listed below that.  I don't think I've ever
customized this.  Are you seeing something different?  Or maybe you mean
something else by "agenda"?

If you're talking about Org's built-in agenda, you might want to take a
look at the `org-agenda-format-date' variable.  Mine is set to the
org-agenda-format-date-aligned function, which I assume is what outputs
the example above.

If you're just talking about timestamps in Org files, you might want to
look at the `org-time-stamp-custom-formats' and
`org-display-custom-times' variables.

Hope that helps!

-- 
Best,
Richard

Re: [O] Order of items after a headline in an Org outline?

[Richard Lawrence]

Hi David,

David Masterson  writes:

> Is there a documented requirement of the order of things in a Org
> outline?  

My understanding is that a lot of work in recent years has gone into
making Org's syntax more consistent, and this is an ongoing effort.  So,
sometimes things still work if they're not in the proper order, but I
wouldn't count on them remaining so.  Your best bet is just to use Org's
functions for inserting and modifying these things, e.g., org-schedule
to schedule something.

You might find this helpful:
https://orgmode.org/worg/dev/org-syntax.html

> For instance, do these things have to come directly after the
> headline and, if so, in what order?  Will Org get confused otherwise?

The correct order is:

> * SCHEDULED and DEADLINE
> * PROPERTY drawer

and then notes.  (Timestamps can occur anywhere within notes.)  e.g.:



* Sample Headline
  DEADLINE: <2019-02-11 Mon> SCHEDULED: <2019-02-10 Sun>
  :PROPERTIES:
  :FOO:  bar
  :END:

Here I am typing some notes, late in the evening on [2019-02-09 Sat]



Hope that helps!

-- 
Best,
Richard

Re: [O] Org-Publish of a PDF ??

[Richard Lawrence]

Hi David,

David Masterson  writes:

> When I publish my project, I find that my org files are first generated
> into tex and pdf files in directory1 and then the tex/pdf files are
> copied to directory2.  What I would like is for the tex/pdf files to be
> directly generated in directory2 with no "extras" in directory1.

I think you might be thinking about this the wrong way around.  I may be
wrong about this, as I'm not a heavy user of the publishing framework,
but here's my two cents.

As far as I understand it, the publishing framework is basically
designed around the idea of copying finished products to a desired
location (a directory, either locally or on a webserver).  It's built as
a layer on top of Org's exporters, not as a way to make those exporters
behave differently.  If you want to control the *build* process for the
PDF, as opposed to just where it finally ends up, you probably want to
set options for the latex exporter, not the publishing framework.

In particular, you might want something like:

#+EXPORT_FILE_NAME: builddir/file.tex

in the Org files you're exporting, where builddir is somewhere outside
of your directory1.  (I'm *pretty* sure that will work, but I haven't
tested it.)

Hope that helps!

-- 
Best,
Richard

Re: [O] “Fuzzy” times (“evening”, “morning”, “night”…)

[Richard Lawrence]

Hi Leo,

Leo Gaspard  writes:

> In the process of migrating all my self-organization to org-mode, I
> noticed there is something that cannot currently be encoded in
> timestamps: fuzzy times, where an appointment is made for “Dec 4, Tue,
> evening” but with the hours not yet fixed.
> ...
> To be perfectly honest, my ulterior motive here is to auto-generate
> tasks “Decide of an exact time for [task]” a few days before the
> timestamp. However, I haven't investigated yet whether that'd actually
> be doable and I'm still pretty new to the lisp/emacs/org-mode
> ecosystems, so this may be completely impossible.
>
> What do you think about this idea? Is there a better way to do what I'm
> trying to do with current tools?

What about just adding a tag (:evening:, :morning:, etc.) to represent a
fuzzy time, with a plain date stamp, like <2018-12-04>?

That would allow you to easily create a custom agenda view containing
just entries with fuzzy times (if they have a timestamp with just a
date, you can even filter for those that are coming up in the next
couple of days).  Then when it's convenient, you can look at this agenda
view and schedule things that are still fuzzy more precisely.

Hope that helps!

-- 
Best,
Richard

Re: [O] In LaTeX export, can I control what heading type a headline goes to?

[Richard Lawrence]

Hi Bill,

William Denton  writes:

> Is there a way, exporting as a book, to make Org skip "part" and make a 
> top-level Org headline turn into a chapter?  Is there a built-in way, or do I 
> need to make my own class in org-latex-classes that has the structure I want?

Another simple solution that doesn't require changing the default
configuration is: just don't use top level headlines in your document!
I don't think Org will care if your highest-level headlines start with "**".

That may or may not work for your particular situation, but it seems
like a good option to keep in mind!

-- 
Best,
Richard

Re: [O] [POLL] Should Org tempo be enabled by default? (expand templates thru e.g. "<s[TAB]")

[Richard Lawrence]

Jon Snader  writes:

I use the 

Re: [O] Should wip-cite branch be merged to master?

[Richard Lawrence]

Hi everyone,

I don't have too much to add to this, though I do want to thank 
everyone for their continued interest in citations, and the work 
that has been done!


Since the question of syntax has come up again, I guess I want to 
address one point that András made:


András Simonyi  writes:

From the citeproc-el/CSL point of view, the current syntax is 
perfect with the notable exception of the provided citation 
commands. Currently only `cite' and `(cite)' are supported, 
where the latter seems to be intended to provide the 
parenthetical version of a basic citation, e.g. in an 
author-date style `cite' would produce something like `Smith 
2018` while `(cite)' `(Smith 2018)'. Now I think that for 
author-date styles `cite' should produce the parenthetical 
version and that `(cite)' probably shouldn't be among the 
commands at all. The main reason is that most citation 
processors (biblatex, CSL processors etc.)  support not only 
author-date citation styles but footnote-based ones as well, and 
the concept of a `parenthetical citation' doesn't really make 
sense for the latter. 


I think this actually gets at one of the fundamental problems with 
citations, namely, they resist attempts to separate 'content' from 
'style'.  It is hard to insert citations into a document without 
making any assumptions about how they will be rendered by the 
citation processor.  And so it is hard to come up with a syntax 
that is general enough to represent widely different citation 
styles, in a way that makes those assumptions explicit in the 
syntax of the unprocessed citation, so that the content of the 
citation can be separated from how it will be rendered.  Thus we 
see the huge proliferation of different citation commands in 
e.g. BibLaTeX, natbib, and similar packages.  András' proposal 
would reintroduce some of the complexity of BibLaTeX citation 
commands at the level of Org syntax.


Maybe having multiple citation commands is ultimately the best way 
to go, but I want to point out that one of the goals of the 
original discussion to come up with an Org syntax that is simpler 
and does a better job of separating content and style.  Anyway, 
that was one of my goals in synthesizing people's various 
suggestions and needs into a concrete syntax proposal.  I think we 
should try to avoid proliferation of citation commands.


If that is a reasonable goal, then I think there is really just 
one distinction that deserves to be represented at the level of 
citation commands: the distinction between 'in text' and 
'parenthetical' citations, that is, the distinction between 
'cite:' and '(cite):' in the current syntax in wip-cite.  That 
brings me to this point:


A more abstract characterization which is applicable to all 
styles is that normally references are not part of the main 
text, they are set off either by brackets or in a note. Since 
this is the most frequent, basic form, I think this the one 
which should be produced by the `cite' syntax, that is, when 
used in normal text `cite' should produce something like `(Smith 
2018)' for author-date styles and a note with the reference for 
note styles.


I think András' abstract characterization is exactly right: the 
issue here is not really about whether the citation has brackets 
around it.  Instead, it is about whether the rendered citation is 
intended to be *read as part of the surrounding sentence* 
('in-text'), or whether it is *grammatically independent* of the 
main text ('parenthetical').  I disagree, however, that 
'parenthetical' citations are the normal or usual case; I think 
that depends completely on one's field, writing style, etc.  (In 
my dissertation, for example, in-text citations outnumber 
parenthetical citations roughly 3:2.) 

The 'in-text' vs. 'parenthetical'/'out-of-text' distinction is 
really the fundamental one that you can't abstract from as an 
author.  And it seems to me that apart from this decision, most of 
the other decisions about styling can handed off to the citation 
processor, via a choice of CSL file, at least for the out-of-text 
case: is it a note-based style, or a Harvard-like one? etc.


The in-text case is a little trickier because you need a way to 
represent the different kinds of data that might be intended to 
fit into the surrounding sentence (author name? year? both? just a 
number? etc.).  But there are many many possibilities here, and 
for that reason I think it is better to avoid making them 
primitive citation commands.  Instead we should just stick to one 
command for the in-text case, and have some extensible way to 
represent the data that should be rendered, e.g. maybe like


[cite: @Jones2018 :year]

or (as I think was proposed at one point):

[cite:author @Jones2018]

Again, maybe it's worth having some shortcuts here for the common 
cases, but I think in general we want to try to avoid 
proliferation of basic citation commands.  So for that reason I 
think we should just stick 

Re: [O] aligning images, html attributes ignored when exporting to html (longish, sorry)

[Richard Lawrence]

Hi Andreas,

Andreas Reuleaux  writes:

Leaving out the double quotes may be comfortable, but I am not 
sure if this was a good idea: makes it difficult to see which 
parts belong to the style, and where the alt attribute starts: 

 #+attr_html: :style float: right; border: 1px solid brown; :alt 
 foo  
this would have been easier to read I think 

  #+attr_html: :style "float: right; border: 1px solid brown;" 
  :alt "foo"


This would be a good place to use a CSS class.

In your CSS file (or between 

Re: [O] Org citations, CSL and citeproc-el

[Richard Lawrence]

Hi András and all: 

Sorry for the delay.  


András Simonyi  writes:

Another question is the syntax for specifying the bibliography 
to be used and, maybe, the location where the bibliography 
should be placed.  Org-ref (ab)uses links for these purposes as 
well. Was there a syntax proposal for these things too? 


As far as I know, there was not.  This was not settled because in 
order to settle it, you need to know a bit more about the backend. 
(Where is the citation data stored? do you have to get it from an 
external program? if so, which? etc. etc.)


I think there was some agreement that we can do this with option 
keywords.  There is an existing syntax for specifying 
bibliographies in contrib/ox-bibtex.el, which probably provides a 
good model:


#+BIBLIOGRAPHY: bibfilename stylename optional-options  When I was 
playing around with hooking up the citation syntax to an external 
citation processor, I was using something similar: I had keywords 
like


#+CSL_FILE: the-csl-style-file.csl +BIBDB: bibtex testdoc.bib 


in the header, and used

#+BIBLIOGRAPHY: here

to specify where the bibliography should be placed in the 
document.  That is obviously not the nicest solution, but the 
basic idea might be a good starting point. 



--
Best,
Richard

Re: [O] Org citations, CSL and citeproc-el

[Richard Lawrence]

Dear Simonyi,

Simonyi András  writes:

a few days ago I've released the first public version of 
citeproc-el (https://github.com/andras-simonyi/citeproc-el), a 
CSL 1.01 citation processor library for Emacs.


Wow!  I don't know if you are aware, but we had discussed the 
possibility of building something like this on the list a few 
years back.  At the time it seemed like a lot more work than 
anyone was willing to do, and so the effort stalled.  Thanks so 
much for working on this -- it looks like you've done a ton of 
work!


The resulting link syntax is rather cumbersome so I'd like to 
ask your opinion about introducing an alternative org-mode 
citation syntax that handles all of these elements.  One option 
would be to use something very similar to pandoc's citation 
syntax (which I tried to follow as much as possible in the cite 
link descriptions of citeproc-orgref). 


We had a (very) long discussion about implementing a new citation 
syntax for Org in the spring of 2015.  Most of it took place in 
February and March; see:


https://lists.gnu.org/archive/html/emacs-orgmode/2015-02/threads.html
https://lists.gnu.org/archive/html/emacs-orgmode/2015-03/threads.html 


And here is how I summarized the state of that discussion in June:

https://lists.gnu.org/archive/html/emacs-orgmode/2015-06/msg00426.html

The upshot was that we came to agreement on quite a few points 
about what citation syntax should look like, and those points have 
been implemented in Org syntax in the wip-cite branch of the Org 
repo.  (This branch is now long out of date and at the very least 
in need of a rebase onto current master, I suspect.)  It would be 
really great to get things moving again.  Using your code to 
provide citation processing during Org export for the syntax 
that's already been implemented would be the place to start.


There are other aspects of the syntax we agreed on that are not 
implemented yet; as I recall, the idea was to get a minimal 
agreeable subset working, and then add to it based on real-world 
experience and feedback.


A more general question I'd like to raise how (or whether) you 
see citeproc-el's (and CSL's) potential place in the org-mode 
ecosystem.  There are a lot of directions which the further 
development could take (BibLaTeX support, citeproc-YAML 
bibliographies, CSL editing and CSL extensions etc.) and I'd be 
grateful to receive your input on which ones I should focus on. 


Here's my two cents on this. From what I recall about the 
discussion (it's hard to believe it's already been almost three 
years...), I would focus on two things:


1) BibLaTeX support.  Most Org users who want citation support 
want it primarily for LaTeX export, and BibLaTeX provides a good 
model of what we should try to achieve in Org citation support.


2) org-bibtex support.  Org-bibtex is a library that represents 
citation data via Org's property syntax.  Some people use this and 
it would be great to have it integrated with the new citation 
syntax and export.  That would provide an end-to-end citation 
solution that is completely native to Org and Emacs. 

Again, thanks for jump-starting this effort!  I look forward to 
contributing to it this spring.


--
Best,
Richard

Re: [O] Properties lost when closing, scheduling, etc.

[Richard Lawrence]

Hi John,

John Goerzen  writes:

Unfortunately, org-mode only recognizes the PROPERTIES when they 
occur immediately after the headline.  So whenever I take an 
action on an item like this, the properties become invisible to 
org-mode.   


What version of Org are you running?  This sounds like old 
behavior (or maybe a problematic mix of new and old behavior). 
PROPERTIES drawer syntax changed (incompatibly) in 8.3; see 
http://orgmode.org/Changes_old.html.


If you are running an older version, may want to try upgrading and 
then running the org-repair-property-drawers function defined on 
that page. 


Hope that helps!

Best,
Richard

[O] Agenda filtering (take 2)

[Richard Lawrence]

Hi everyone,

About a year ago, I wrote: 


Is there a simple way for me to say to the agenda, "Show me all 
(and only) the NEXT tasks that are part of a project whose 
deadline is before (say) 2016-11-01"?  It seems like this 
should be possible with the built-in agenda but I can't quite 
figure it out.  The tricky thing is filtering by the deadline 
of the parent project, which might be several levels up. 


I'm still stuck on this, so I'd like to ask this question again. 
Bastien responded: 

I would add a category to each project, then use something along 
this: 

("N" "My important tasks" tags-todo 
 "CATEGORY={cat1\\|cat2}+TODO={NEXT}+DEADLINE<=\"<+3d>\"") 


But that doesn't seem to work for me.  The problem is that the 
tasks I want to list in the tags-todo search don't themselves have 
deadlines; only their parent projects do.


I have a file with projects that look like this:

** TODO Foo University 
  DEADLINE: <2017-11-01>

*** NEXT Determine application requirements for Foo U.
*** TODO Submit application

I'd like to see a list of just NEXT tasks in this file, sorted by 
the deadline of their parent projects.


I thought that property inheritance would be the way to do this. 
So I tried: 

   ("jn" "Job application NEXT tasks" tags-todo 
"+jobmarket+application+TODO={NEXT}" 
((org-use-property-inheritance t)) 
 (org-agenda-sorting-strategy '(deadline-up 


as well as

   ("jn" "Job application NEXT tasks" tags-todo 
"+jobmarket+application+TODO={NEXT}" 
((org-use-property-inheritance '("DEADLINE")) 
 (org-agenda-sorting-strategy '(deadline-up 

in org-agenda-custom-commands.  But neither seems to work.  Can 
anyone provide any insights here?  Is it possible for sub-tasks 
within a project to inherit their deadline? 


Many thanks as always!

Best,
Richard

Re: [O] Auto-filling in org-mode without affecting the headlines

[Richard Lawrence]

Hi Narendra,

Narendra Joshi  writes:

I would like to have auto-fill enabled in org-mode but would 
like to prevent headings from being auto-filled. Is something 
available in org-mode that would allow me to do this? 


I have used auto-fill-mode in Org buffers for quite a while now 
without any problems, and a quick test on my end shows that long 
headings don't get auto-filled.  Are you having problems with 
this?  If so, what version of Org and Emacs are you using?   -- 
Best,

Richard

Re: [O] [RFC] Remove Org Struct mode

[Richard Lawrence]

Nicolas Goaziou  writes:


I would like to remove Org Struct minor mode from Org code base.


Like others, I only use Org Struct mode to get some Org-like list
handling and such when I'm writing email, and as Nicolas observes, it
barely works.  So I'm OK with this, especially if it makes Org easier to
maintain!

--
Best,
Richard

Re: [O] LaTeX Export

[Richard Lawrence]

Hi Scott,

Scott Randby  writes:

I tried this in the properties associated with a subtree:  

:export_latex_header: 
\hypersetup{pdfauthor={foo},pdftitle={foo},pdfsubject={foo},pdfkeywords={foo},pdfproducer={foo},pdfcreator={foo}} 

Unfortunately, all the other LaTeX headers are unrecognized when 
I do this.  


If you put the subtree-specific hypersetup in each subtree's 
EXPORT_LATEX_HEADER_EXTRA property, and the common stuff in 
#+LATEX_HEADER lines, I think it should do what you want.  Org 
should combine these properties rather than using the subtree 
property to override the global ones. 


Hope that helps!

--
Best,
Richard

Re: [O] How to paginate Babel Lisp output?

[Richard Lawrence]

Hi Jean,

Jean Louis  writes:

I have a very simple lisp output within Org Mode file, and it 
delivers list of lists with 4-5 elements, and it displays nicely 
as a text. 


(Do you mean: as a table?) 

#+BEGIN_SRC lisp :exports results  (load 
#P"/home/data1/protected//lisp/load-some-files.lisp" :verbose 
nil :print nil) (sku-list-values-by-location :location 1369) 
#+END_SRC 

But when exporting LaTeX I get the result that the table is 
basically too long, even being printed over the page number and 
below, and does not even continue on the next page. 

Is there a way or customization to make it break pages 
automatically when there is long output from source blocks? 


If you are already getting your results as an Org table, try 
adding 


#+ATTR_LATEX: :environment longtable

to that table.  This will use the LaTeX longtable package and
environment to format the table, which should take care of breaking it
across pages.

Best,
Richard

Re: [O] Price/ list comparison

[Richard Lawrence]

Hi Eduardo,

"Eduardo V."  writes:

> I constantly buy diodes, resistors, etc. for projects and i mostly save the
> receipts.
>
> I want to know if there is an org-mode way (or any other software) that
> let's me input all the data form the receipts and tell me which components
> are cheaper in X store.
>
> A plus if it has a way of telling me how many components i have left.

You might want to check out Ledger: http://ledger-cli.org/

It is a system for doing accounting with plain text, and has lots of features 
that I think you would find useful, including tracking prices and commodities.  
You should definitely be able to get it to tell you the things you're looking 
for, once you enter your receipts.  Also, it has good integration with Emacs, 
and (to some extent) with Org. 
 
Best,
Richard

Re: [O] Getting "Chapter" before a heading number

[Richard Lawrence]

Hi Peter,

Peter Davis  writes:

> I'm preparing a proposed table of contents for a book, and I'd like the HTML 
> output to appear like:
>
> Chapter 1 Blah
>   1.1 blah
>   1.2 blah blah
>   1.3 blah blah blah
>
> Chapter 2 Blah Blah
>   2.1 blah
>   2.2 blah blah
>
> etc.
>
> Is there a way to get HTML output to do this? I'd like the "*" at the start 
> of a line to denote a new chapter, and nested headings
> to go inside. I haven't been able to find anything like this.
>

This might not be exactly what you want, but it's pretty trivial to do
this with CSS.  In your stylesheet, you want something like (untested):

span.section-number-2:before { content: "Chapter" }

Basically, this tells the browser to put the word "Chapter" before the
number of a first-level headline (which, in a default setup, is exported
as a span tag with class section-number-2).

If you'd rather that "Chapter" appears in the actual HTML, you probably
want to use a filter on headlines during export.

Hope that helps!
 
Best,
Richard

Re: [O] sending an email to an orgmode file?

[Richard Lawrence]

Hi Xebar,

Xebar Saram  writes:

> im looking for a simple solution that will allow me to send an email from
> my mobile phone and habr that email either be appended to a txt(org) file
> or perhaps another solution to get my mobile on the go notes onto my laptop
> orgmode file. i tried orgzly and mobile org but i would prefer a simple
> solution as to send an email
>
> anyone have any thoughts on this?

I'd say a lot depends on the mail setup you use on the machine where
you use Org.  Emacs-based mail clients will be easier to integrate with
Org.  But really you're just limited by how much effort you want to sink
into this, at least if you store your mail in some format that is easy
to read with various tools.

If you use Gnus, one solution (I think?) is Gnorb:

https://elpa.gnu.org/packages/gnorb.html

I don't actually use this or know much about it but other people on this
list can tell you more about it.

In my own setup, I've settled on just making it easy to linking to an
email from a new TODO entry.  This is a very general mechanism that
works well for me, even though it is not completely automatic.  I
describe my setup below.

I use the Emacs interface for notmuch as my mail client.  I integrate it
with Org via org-notmuch and Org capturing.  Basically, I have a key that
automates capturing an email as a TODO item, containing a link to the
relevant mail:

#+BEGIN_SRC elisp
;; in my Org setup:

(require 'org-notmuch)

(setq org-capture-templates
; ...
  `(("t" "Task, Note, Project, etc.")
("tm" "Mail" entry
 (file ,(concat org-directory "/refile.org"))
 (file ,(concat org-template-directory "/mail-with-link.txt")

;; in my notmuch setup:
  (define-key notmuch-search-mode-map "P"
(lambda ()
  "postpone message (remove inbox tag, create task to reply)"
  (interactive)
  (notmuch-search-tag '("-inbox" "-unread"))
  (notmuch-search-show-thread)
  (org-capture nil "tm")))
#+END_SRC

The mail-with-link.txt template is simple:

#+BEGIN_SRC org
* %a:MAIL:
  :PROPERTIES:
  :Entered: %U
  :END:
  %? 
#+END_SRC

I mostly use this to record emails that I need to *reply* to as TODO
items, so a link to the original mail is enough for me.  

This probably isn't quite what you're looking for, but with more effort
on the notmuch side, a setup like this could be used to extract the
content of the email into the Org capture.  And you could also automate
this by looping over the messages that match a certain notmuch query,
etc.  
 
HTH,
Richard

Re: [O] Citation syntax: now supported in Pandoc

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou  writes:

> I rebased wip-cite onto master. I didn't test the branch but "make test"
> reports no problem.

Great, thanks!

I guess this does not include the preliminary work on the org-cite
library (from Aaron Ecay's wip-cite-awe branch), or the work I was doing
on interfacing with a CSL processor via JSON.  Both of those pieces of
work are in the branch in [1].

> I'd rather have some testing on the Org side. This is not a Pandoc-devel
> mailing list. Besides, we really need to move forward on this branch.

Sure, of course.  In order for there to be testing on the Org side,
though, we need the support for citation syntax that the wip-cite branch
provides to get hooked up to the various exporters.  This is what we
made a start on in the org-cite library, before some sticky design and
dependency issues interrupted the momentum.  Will anyone have time to
work on those soon?

Best,
Richard

[1] https://github.com/wyleyr/org-mode/commits/wip-cite-rebase

Re: [O] Citation syntax: now supported in Pandoc

[Richard Lawrence]

Hi Matt,

Matt Price  writes:

> Richard, is wip-cite up to date with recent git versions of org-mode?

Nope.  At least, I have not done any work on this, and I don't know of
anyone else who has either.  I suppose I should rebase it onto master
again, but it's hard to justify the effort unless someone is going to
actively work on it, because otherwise it will quickly get out of date.
(Unfortunately, I can't put much time into this, at least not in the
short term.)

> I'd like ot try it out but would rather not go back to a pre-9.0
> version or org, having done a certain amount of work to update to the
> new framework...

Well, I'd still recommend trying it with Pandoc (instead of trying to
get wip-cite to work).  If there are issues with Pandoc's Org reader
post-9.0, those should be filed as bugs against Pandoc.

Best,
Richard

Re: [O] latex export: can I add an optional argument to an item?

[Richard Lawrence]

Hi Alan,

Alan Schmitt  writes:

> I would like to generate the following latex export:
>
> \begin{itemize}
> \item[\Checkmark] foo
> \end{itemize}

It's not exactly pretty, but you can use a description list while
resetting it to the itemize environment, like:

#+ATTR_LATEX: :environment itemize  
  - \Checkmark :: foo

That produces:

\begin{itemize}
\item[{\Checkmark}] foo
\end{itemize}

At least, that works for me (I have not yet upgraded to Org 9.0;
so I don't know if this will continue to work, though I don't know of
any reason why it shouldn't). 

Hope that helps!

Best,
Richard

[O] Citation syntax: now supported in Pandoc

[Richard Lawrence]

Hi everyone,

With Org 9.0 out, I thought it might be a good time to revisit the issue
of citation syntax.  Not much has happened with this recently, but I do
have one bit of progress to report:

About six months ago, Albert Krewinkel, who maintains the support for Org
syntax in Pandoc [1], wrote to me about supporting the citation syntax
we discussed previously, and that is partially implemented in
the wip-cite branch.  You can see examples of the syntax at [2].

Albert dove in, and was able to quickly add support for the syntax to
Pandoc's Org reader.  This support has been merged into released
versions of Pandoc.  Albert actually announced this a few months ago on
Reddit [3], but it looks like he did not email this list.

I finally got a chance to check out his work yesterday.  I converted one
citation-heavy chapter of my dissertation to use the new syntax, and
tested out Pandoc's support, with a couple of different CSL stylesheets,
including a note-based style.  It works great.  Thanks, Albert, for your
hard work!

There are some minor issues that probably need more discussion after
real-world use (like whether/when a conjunction should be inserted
automatically before the last reference in a multi-cite citation).  But
as far as I can tell, Albert has correctly implemented everything we
actually discussed and agreed on.

In short, Pandoc now has support for Org citation syntax, and that
support is fully integrated with Pandoc's citation processing
capabilities.  So there is now at least one "real" (non-prototype)
solution for processing Org citations via CSL, with output to HTML,
LaTeX, ODT, .docx, and the other formats that Pandoc supports.

This makes it a lot easier for people to test out the syntax in the real
world, so I encourage you to do so, and report back on what you find!

Best,
Richard

[1] http://pandoc.org/
[2] https://github.com/wyleyr/org-citeproc/blob/master/tests/testdoc.org
[3] 
https://www.reddit.com/r/orgmode/comments/4tes4x/ann_pandoc_v1172_has_been_released_it_now/

[O] Agenda filtering

[Richard Lawrence]

Hi everyone,

I have an Org file with a lot of projects that look like this:

* TODO Project A
  DEADLINE: <2016-10-28 Fri>
** NEXT Subtask 1
** TODO Subtask 2

* TODO Project B
  DEADLINE: <2016-10-31 Mon>
** DONE Subtask 1
** NEXT Subtask 2

Some of these projects have deadlines far in the future, some near.  I'd
like to focus on just the NEXT tasks for projects with upcoming
deadlines.

Is there a simple way for me to say to the agenda, "Show me all (and
only) the NEXT tasks that are part of a project whose deadline is before
(say) 2016-11-01"?  It seems like this should be possible with the
built-in agenda but I can't quite figure it out.  The tricky thing is
filtering by the deadline of the parent project, which might be several
levels up.

Many thanks for your insights!

Best,
Richard

Re: [O] How to change agenda sorting order temporarily?

[Richard Lawrence]

Hi Martin,

Martin Beck  writes:

> I'm working with a lot of pre-defined agendas, but I sometimes would
> like to change the sorting order of one of those to different criteria
> (by time, alphanumerically, etc.).
> I did not find a way to do that in org-mode directly - did I overlook
> something, or do I really have to change the configuration of the
> whole agenda and rebuild it?

Did you find the variable `org-agenda-sorting-strategy'?  That offers a
lot of possibilities.

> Related problem:
> If I change the configuration of the agenda by changing the
> customizing options, those changes do only have an effect, if I kill
> the agenda buffer and launch the agenda again.
> It would be nice, if I just could rebuild the existing agenda with "r"
> or "g", but that does not seem to be sufficient?

I'm afraid I don't know the answer to this, but would also be interested
to hear what it is.
 
Best,
Richard

Re: [O] Annotating org exporters

[Richard Lawrence]

Hi Sebastian and all,

Sebastian Fischmeister  writes:

> I'm still undecided between a regex replace and org-entities. Is there
> a straightforward way to define own directives for orgmode to then
> support something like the following?
>
> #+LaTeX_EXPORT: "=>":"$\rightarrow$"
>
> or more generic
>
> #+EXPORT_EXPAND: latex:"=>":"$\rightarrow$"
>
> Then I could just make these expansions part of the standard header in
> my org files.

Well, there are macros, which do something very similar.  As far as I
know, they are the only way to achieve this kind of thing without some
Elisp.

You could do something like:

#+MACRO: => @@latex:$\rightarrow$@@

but that actually doesn't seem to work as is, I suspect because "=>" is
not an allowable macro name.  (It works fine with an alphabetic name
like "ARR".)  But even if you got it to work, you'd then have to type

Some text {{{=>}}} other text after the arrow

in your document, which sort of defeats the point of the nice, simple
"=>".  If it were me, I'd opt for the regex replacement, and avoid
typing all the braces.
 
Best,
Richard

Re: [O] Annotating org exporters

[Richard Lawrence]

Hi Sebasitan,

Sebastian Fischmeister  writes:

> Is there a simple way to build regex-based extensions to the exporters?
> For example, I would like to convert this string "=>" to $\rightarrow$
> when converting the document to latex.

There's an example of how to do something like this in the "Advanced
Configuration" section of the Export section in the manual.  Maybe you
could adapt it like this?

 (defun my-latex-filter-rightarrow (text backend info)
   (when (org-export-derived-backend-p backend 'latex)
 (replace-regexp-in-string "=>" "$\rightarrow$" text)))

 (add-to-list 'org-export-filter-plain-text-functions
  'my-latex-filter-rightarrow)


Hope that helps!

Best,
Richard

Re: [O] pandoc-style citations

[Richard Lawrence]

Hi Alex,

Alex Fenton  writes:

>> Export is where efforts stalled last year.
>
> That's understandable, given that, as you say, it's a complex problem 
> given the range of citation styles and output formats. It's still a 
> shame given the work that you (pl.) have put into integrating citations 
> into the org parser & element tree so they are first class objects.
>
> I don't know whether it's conceivable that the data structures and 
> parsing could be integrated into org, with the (presumably) relatively 
> easy latex output, which I suspect is the commonest use case, and then 
> with some kind of "adequate" output for other targets (html, text, odt) 
> - perhaps an output that would require further post-processing by a 
> third-party tool such as citeproc or pandoc.
>
> Latex-outputters would be better off and other targets no worse off than 
> present, and it might act as a spur to solve the other target formats 
> one by one way. But I can see that this has been to some extent 
> considered and can also see the arguments against.

Unfortunately, even the LaTeX case is not so easy unless you assume
users are managing the bibliography database, and the mapping of
citation keys into that database, themselves.  It's easy to map the Org
syntax to the LaTeX syntax if you can just pass the keys through, but
that isn't a great assumption to make.  And things are a lot more
complicated once you have to worry about looking up the keys somewhere,
generating an appropriate .bib file, etc. We will have to do this stuff
for other backends anyway, so it seems sensible to do it for LaTeX too,
but that's what makes the project so big (especially if we can't just
pick one database format to support).

Maybe you're right that a good first step would just be to generate
LaTeX syntax, and just let users manage the bibliography database in
whatever way works for them, without trying to interface with it.  This
could be done quickly -- indeed, I think it is already most of the way
done.  And as you say, that would go a long way toward meeting many
people's needs.

I hope I will have some time next week to look at this.  I'll report
back!

Best,
Richard

Re: [O] pandoc-style citations

[Richard Lawrence]

Hi Alex,

Alex Fenton  writes:

> I see that there were several extensive and fruitful discussions on this 
> list last year on citation syntax. There seemed to be a reasonable 
> degree of consensus that pandoc-style citation syntax was at the least a 
> good model.
>
> I'd like to know if there are any implementations out there of elisp to 
> parse pandoc citation syntax and turn it into latex \cites. My question 
> is not so much "when/if this will be in org mode" but rather whether 
> there's something I can drop in now (likely as a link type).

As far as Elisp implementations go, I know of no specific parser for
Pandoc citation syntax.  But there is support for a Pandoc-like syntax
(discussed in the threads you read) in the wip-cite branch of Org's
repository.  This provides support for multi-cite citations in Org
syntax, but it isn't hooked up to the export framework at all.

Export is where efforts stalled last year.  Aaron Ecay, Vaidheeswaran C,
and I all worked on different proof-of-concept implementations to hook
up the citation parser to a citation processor and the various
exporters.  This is a non-trivial problem, and it seems that not many
people have a lot of time to work on it (including me), so if you want
to help, that would be great!  Aaron's work is in the Org repository
(see the wip-cite-awe branch).  Vaidheeswaran's is elsewhere; I don't
know where exactly, but you can search the list for a link.  My own is
here, in the wip-cite-rebase branch:

https://github.com/wyleyr/org-mode/.

There is also Pandoc itself, which can read (some) Org syntax.
Depending on what your document looks like, you might have good luck
just using Pandoc to convert it to LaTeX.

> I have a lot of longish citations with multiple references each with 
> their own pre- and post- ("'blah blah blah @ref1, p.23, also @ref2, for 
> a contrary view see @ref3 pp148-152") that end up as \cites. However my 
> home-brew link solution, stuffing the multiple pre- and posts- with 
> separator into the link description is unwieldy - difficult to write, 
> hard to read and easy to get wrong or breaking output.

You may also want to look at John Kitchin's org-ref, which I believe
works similar to your homebrew link solution, but has a lot of features
and may provide a better interface for what you're trying to do:

https://github.com/jkitchin/org-ref
 
Best,
Richard

Re: [O] [OT] A new web browser‽

[Richard Lawrence]

Eric Abrahamsen  writes:

> I installed the eww-lnum package right away, as that provides the main
> functionality I liked from Conkeror: hit a key, and pick a link to do
> something with. Rather strangely, the KeySnail plugin for Firefox seems
> to do everything *but* this, which I thought was weird since, if you're
> going to control your browser through the keyboard, following links is
> generally the main thing you want to do. I've only had it for a few
> hours, though, so maybe I'm missing something.

IIUC, Firefox actually has this behavior out of the box: press ' (single
quote) and start typing the link text.  I was surprised and pleased to
learn about that when I first found it.  That may be why it isn't
included in KeySnail.
 
Best,
Richard

Re: [O] Export Subtree to LaTeX: no EXPORT_LATEX_CLASS_OPTIONS ?

[Richard Lawrence]

Hi Axel,

Axel Kielhorn  writes:

> I want to export a subtree with a columnview dynamic block in
> landscape orientation.  I don’t want just a rotated table, the whole
> page should be in landscape mode.  I may have to tweak the page layout
> using geometry as well, thus I probably need EXPORT_LATEX_HEADER.

> * Projektboard :noexport:
> :PROPERTIES:
> :EXPORT_FILE_NAME: Projektboard
> :END:
> #+EXPORT_LATEX_CLASS_OPTIONS: [landscape]
> #+EXPORT_TITLE: Projektboard
> ...

I think what you want to do is put the EXPORT_LATEX_CLASS_OPTIONS (and
the TITLE) as a property in the property drawer, like this:

* Projektboard :noexport:
:PROPERTIES:
:EXPORT_LATEX_CLASS_OPTIONS: [landscape]
:EXPORT_TITLE: Projektboard
:EXPORT_FILE_NAME: Projektboard
:END:

That has the intended effect when I test it here, and my setup is
similar to yours.  In general, when you just want to affect the export
options of a subtree, you use properties, rather than keywords.

Best,
Richard

Re: [O] Attach a Word doc in org mode file

[Richard Lawrence]

Hi Eliya,

(I'm sending my reply to the list; please use reply-all or similar so
your responses go to the list, too.)

Eliya Voldman  writes:

> I'm using Windows OS.
> I'm using Emacs package from Cygwin hence I open any file types e.g. .doc
> using %cygstart.exe file.docx

I don't know anything about Cygwin (or Windows, really), but this looks
strange to me.  What does cygstart.exe do?  I would think you would want
a command like

word.exe file.docx

Maybe Cygwin has another way of starting Word?

> I never used to open file from Emacs buffer and I don't know how to open it
> from Emacs itself.
> It looks that text file Org opens with no problem but other type of file
> should be opened differently ( I think so )

Yes, presumably.

> I checked C-h v org-file-apps but it does not look straightforward for me
> how to set it up to open Word type file.

If it's not working automatically, then you probably need to add a
clause like

("\\.docx\\'" . "word.exe %s")

But I don't know what the second part (the command) should be; I'm not
even sure that it's possible to tell Word to open a file from the
command line.  I don't use Word, but maybe someone else will know.

By the way, I'm not sure what your reasons for running Emacs from Cygwin
are, but you *might* have better luck if you just run Emacs for Windows.
If Emacs is not running inside Cygwin, the system "open" command might
automatically work on Word files.  This is just a guess.

> When I tried to use 'link' ... it worked and followed the link but opened
> not the Word file but rather directory with a bunch of .xml files hence I
> was not able to open that file.

That's because, if I understand correctly, a .docx file is really just a
.zip archive with a bunch of .xml files inside.  Emacs knows how to open
a .zip archive, so it's showing you the internal .xml files in the
.docx.  (Which you *could* theoretically edit by hand, using Emacs...)

But that's not what you want.  You want to launch Word, right?  Or do
you have another program that you want to use to read Word files?

> Well I understood from your reply that *only* plain text file could be
> opened in Org major mode ( and it worked for me)
> I tried to use Emacs to open Word file by using C-c C-o but it again opened
> me a list of .xml file and not the Word file.

Yes, same explanation: Emacs sees the .docx as a .zip archive containing
those .xml files.

> Hence the question is: how to open Word file from emacs or Org mode?

The next thing you need to figure out is how to launch Word from the
command line in your environment, I think.

Best,
Richard

Re: [O] Attach a Word doc in org mode file

[Richard Lawrence]

Hi Eliya,

Eliya Voldman  writes:

> How could I attach a Word document in my .org file? I could do it for plain
> text file like C-c C-a but not .doc type.
>
> In other words: attach Word doc is not a problem but the problem is to open
> it.

What OS are you on?  Windows?  OS X?

I don't know how exactly Org decides how to open files on your system,
but if you're having trouble, I guess you should start by looking up the
value of the org-file-apps variable.  Try:

C-h v org-file-apps

> Are there other options to point to Word file in my system other the
> attach?

You can use a link; see this section of the manual:

http://orgmode.org/manual/Hyperlinks.html#Hyperlinks

> BTW: is it possible to attach and then open in org mode files other that
> plain text?

It depends what you mean by "open in org mode".  The only files it makes
sense to open as Emacs buffers with Org as the major mode are plain text
files in Org mode format.  But you can open other types of files in
Emacs in other modes.  And from within an Org mode buffer, you should be
able to launch programs other than Emacs (such as Word) that can read
other kinds of files, for example by following a link with C-c C-o.

Hope that helps!

Best,
Richard

[O] Bug: [PATCH] Tiny timezone bug in ox-icalendar [8.3.3 (release_8.3.3-2-g6bc48c @ /home/rwl/src/org-mode/lisp/)]

[Richard Lawrence]

Hi all,

I finally got annoyed enough to debug a problem with the timezone field
in the .ics export of my agenda.

Turns out the problem was simple: org-icalendar--vcalendar was being
called with the wrong argument order in
org-icalendar-export-current-agenda, so the timezone was going into the
description field rather than the timezone field.

Here's a patch that fixes that.  Note that it should be applied to
maint.  I did not generate a commit against master, but I can if need
be.

Best,
Richard

>From 410ba46791e2f4cf470f8727298c3faa5b717e56 Mon Sep 17 00:00:00 2001
From: Richard Lawrence <richard.lawre...@berkeley.edu>
Date: Thu, 21 Jan 2016 17:28:58 -0800
Subject: [PATCH] ox-icalendar:  fix timezone export bug

* ox-icalendar.el (org-icalendar-export-current-agenda): Correct
  argument order in call to org-icalendar--vcalendar (timezone should be
  third, description fourth).

TINYCHANGE
---
 lisp/ox-icalendar.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index be2f3e6..a7cb4fb 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -909,8 +909,8 @@ This function assumes major mode for current buffer is
(org-icalendar--vcalendar
 	org-icalendar-combined-name
 	user-full-name
-	org-icalendar-combined-description
 	(or (org-string-nw-p org-icalendar-timezone) (cadr (current-time-zone)))
+	org-icalendar-combined-description
 	contents)))
 (run-hook-with-args 'org-icalendar-after-save-hook file)))
 
-- 
2.1.4

Re: [O] Can weekly agenda show children nodes?

[Richard Lawrence]

Hi Tomasz,

Tomasz Piotrowski  writes:

> I usually plan my activities for a single day without specifying time of
> day it should be accomplished, e.g.,
>
> * <2016-01-04 Mon>
> ** Item 1
> ** Item 2
>
> In agenda, children nodes of
>
> * <2016-01-04 Mon>
>
> are not shown, i.e., only * <2016-01-04 Mon> is displayed and is empty.
>
> Can I tweak agenda to show
>
> ** Item 1
> ** Item 2

I am not sure whether it is possible to do exactly what you asked; maybe
someone else will know.  But I wanted to suggest an alternative: rather
than using the headline hierarchy to represent your schedule, use
timestamps within the individual activities, like:

** Item 1
   <2016-01-04>
** Item 2
   <2016-01-04>

This will cause them to appear in the agenda on that date, as you
wanted.  It also has the advantage that rescheduling the individual
activities for a different day is as easy as changing the timestamp --
you don't have to move the headline around in your file, and can just
let the agenda take care of assembling your schedule for the day.

You may also find the SCHEDULED and DEADLINE keywords helpful; see the
"Deadlines and scheduling" section of the manual.

Hope that helps!

Best,
Richard

Re: [O] Org list source block

[Richard Lawrence]

Hi Jarmo,

Jarmo Hurri  writes:

> I am trying to define a checklist and then use the same list in my
> exported document in multiple locations. ...
> What is the correct solution to this?

Would an #+INCLUDE file work here?  See

http://orgmode.org/manual/Include-files.html#Include-files

Best,
Richard

Re: [O] LaTeX export with section number, name and page in internal links

[Richard Lawrence]

John Kitchin  writes:

>> I do something like this with custom link types.
> Aha! I am not the only one ;)
>
>>
>> First of all, have a look at the variable
>>
>> org-latex-prefer-user-labels
>
> Is this a new 8.3 variable? It doesn't seem to be in my 8.2.10 MELPA
> version.

Yes, I believe it is new in 8.3, although it originated in a patch I
wrote quite a while ago (February 2014?) when I needed this behavior.

> I like your idea. I think you could simplify it to just:
>
> (org-add-link-type
>  "sec"
>  (lambda (path)
>(org-open-link-from-string (format "[[#%s]]" path)))
>  ...

Ah, nice, that is way simpler.  Thanks!

Best,
Richard

Re: [O] LaTeX export with section number, name and page in internal links

[Richard Lawrence]

Hi Ilya,

Ilya  writes:

> I export my Org-Mode notes with internal links to LaTeX and I want it to
> look like this ''Section 1.1 [Section name], page 99'' (Like in the Org
> Manual). I use this construction:
>
> #+BEGIN_EXAMPLE
>  * Chapter 1
>  ** Section 1.1
> :PROPERTIES:
> :CUSTOM_ID: section-1
> :END:
>  * Chapter 2
>  ** Section 2.1
> I want reference to Section 1.1 from here (See #section-1)
> #+END_EXAMPLE
>
> But as a result I get only the number of the section ''1.1'', not the
> ''Section 1.1 [Section name], page 99''.
> What options I need to use?

I do something like this with custom link types.

First of all, have a look at the variable

org-latex-prefer-user-labels

if you haven't already.  Setting it will cause Org to use CUSTOM_ID
properties to generate labels, so you don't need to manually insert your
own.

I use the following bit of Elisp to define some link types for referring
to sections this way.  You could modify this to insert the LaTeX command
you're interested in (as opposed to just \ref{}).  With your example
above, you'd write something like

#+BEGIN_EXAMPLE
I want reference to Section 1.1 from here (See [[sec:section-1]]).
#+END_EXAMPLE

Here's the code:
#+BEGIN_SRC elisp
;; Link types for targeting sections, tables, etc.
;; These assume that headlines with CUSTOM_ID defined will export using
;; that value as their \label keys.
(defun org-find-headline-by-custom-id (prefix path)
  "Find a headline in the current buffer by CUSTOM_ID value PREFIX:PATH."
  (save-excursion
(goto-char (point-min))
 (and
  ; borrowed from org.el; there doesn't seem to be a function that searches
  ; for a headline with a specific property value
  (re-search-forward
   (concat "^[ \t]*:CUSTOM_ID:[ \t]+" prefix ":" path "[ \t]*$") nil t)
  (setq pos (match-beginning 0
   (if pos
   (progn
 (goto-char pos)
 (org-back-to-heading t))
 (message (format "Headline with CUSTOM_ID %s:%s not found." prefix path

(defun org-export-dissertation-link (prefix path desc format)
  "Export a link to a dissertation section, etc.

In LaTeX, the exported link will look like:
  DESC~\\ref{PREFIX:PATH}
"
(when (member format '(latex linguistics))
  (format "%s~\\ref{%s:%s}" desc prefix path)))

; Sections:
(org-add-link-type
 "sec"
 (lambda (path)
   (org-find-headline-by-custom-id "sec" path))
 (lambda (path desc format)
   (org-export-dissertation-link "sec" path (or desc "Section") format)))

; etc. etc.
#+END_SRC elisp

Best,
Richard

OpenPGP Key ID: CF6FA646
Fingerprint: 9969 43E1 CF6F A646

(See http://www.ocf.berkeley.edu/~rwl/encryption.html for more information.)

Re: [O] More questions about CSL and org-mode

[Richard Lawrence]

Hi John,

John Kitchin  writes:

> Thanks.
>
> Its an interesting jam. You want to have multiple outputs as a
> possibility, but there isn't a robust markup that readily works across
> all backends.

Yes, indeed.  

> On export the in-text citations are transformed to unique text blobs,
> e.g. uuids, and the document exported. The only important features of
> these blobs is that they do not get changed on export, and they are
> unique because we replace them later.
>
> The strings in the bibliography entry are "exported" to convert the
> org-markup to the output format. The in-text citations, expanded
> bibliography and style are sent to the citation processor, which outputs
> replacements and a formatted bibliography in the desired output format.
>
> Finally, you replace each uuid with the appropriate replacement, and
> insert the bibliography where it belongs. That should be the final
> document.

IIUC, the problem with this approach is that it will not work well when
the citation style is note-based rather than inline.  The main
motivation for going "back to Org" is that note-based styles require the
document structure to change as a result of citation processing: new
footnotes have to be inserted, and existing ones have to be renumbered.
That is relatively hard to do if the rest of the document is already in
the target format (except with LaTeX).  By doing citation processing
early in the export process and converting the results to Org, we can
rely on Org's footnote processing to handle this later in the export
process.

As far as I can see, if it weren't for note-based styles, this approach
would work fine.  (Indeed, it is pretty much what the existing org-cite
code does, except that the mapping between citations and their
replacements is done with Lisp data structures rather than via string
replacement in the output buffer.  I stopped work on that right about
the time I realized the existing approach wouldn't work very well with
note-based styles.)

But given the problem about nested formatting, going back to Org at the
level of text replacements doesn't work.  In other words: both of the
simple-minded approaches (process citations directly to text in the
target format, or process them to Org text, then let Org convert them to
the target format) face problems.

I think probably what we'll have to do to accommodate both note-based
styles and the possibility of nested formatting is to get the results of
citation processing in some unambiguous format like HTML or JSON, then
parse it, and then use the result to directly modify the parse tree for
the Org document before continuing the export process.  I can't see an
easier way...can anyone else?

Best,
Richard

Re: [O] More questions about CSL and org-mode

[Richard Lawrence]

Richard Lawrence <richard.lawre...@berkeley.edu> writes:

>> IIUC, the current aim is to get a citeproc that will do the following on
>> export:
>> 1. replace in-text citation syntax with org-formatted replacements
>> 2. Insert an org-formatted bibliography somewhere in the document
>> 3. proceed with org-to-something export, with built-in
>> exporters.
>
> That's basically my understanding too.  There is one snag with the
> "org-formatted replacement" plan, though, which I saw in a Zotero dev
> discussion yesterday.  

Here's the reference for that discussion, by the way:

https://groups.google.com/d/msg/zotero-dev/Bz_IenruxX4/24QWuyEIp_IJ

Best,
Richard

P.S.  John, thanks for your continued research on this.  I see that our
procrastination habits are on the same schedule. :)

Re: [O] More questions about CSL and org-mode

[Richard Lawrence]

Hi John,

John Kitchin  writes:

> Hi all,
>
> This is mostly for the people working on citations in org-mode.
>
> I have been reading about CSL more this weekend. IIRC, one of the
> reasons to develop the new citation syntax was to get the ability to
> have pre/post text in citations more conveniently than what is currently
> possible.

Yes, that is my understanding, too.

> I have not seen any possibility for this with CSL, however. Is my
> understanding correct? Is this a problem, or something partially handled
> by org-export and partially by a citeproc?

The CSL processors I've looked at support prefix and suffix text for
individual references within a citation.  See, for example, the
citeproc-js documentation:

http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#citation-data-object

prefix, suffix, and some other fields are supported.  pandoc-citeproc
supports the same set of fields.

However, my understanding is that neither citeproc-js nor
pandoc-citeproc support a BibLaTeX-style "common" prefix/suffix that
belongs to the citation as a whole, rather than the individual
references within it, as is available in the multi-cite commands.  We
currently have support for such common prefixes/suffixes in Org syntax. 

My solution to this in my org-citeproc wrapper for pandoc-citeproc is to
prepend the common prefix to the prefix for the first reference in a
citation, and append the common suffix to the last reference.  This is
not a great solution, because it is not really defined what kind of
punctuation (if any) should separate the common prefix from the first
item's prefix, and so on.  But I figured that was not an important issue
to address until we actually have people making use of common prefix and
suffix syntax who are not exporting to LaTeX...

> IIUC, the current aim is to get a citeproc that will do the following on
> export:
> 1. replace in-text citation syntax with org-formatted replacements
> 2. Insert an org-formatted bibliography somewhere in the document
> 3. proceed with org-to-something export, with built-in
> exporters.

That's basically my understanding too.  There is one snag with the
"org-formatted replacement" plan, though, which I saw in a Zotero dev
discussion yesterday.  CSL processing might result in multiple levels of
formatting, e.g. nested italics like

Something with an internal Title

and that won't translate very well back to Org syntax in general:

/Something with an internal /Title//

The suggestion was to just use HTML output, and then parse the HTML to
get a data structure that could be directly rendered into HTML, LaTeX,
etc., which support nested italics just fine.  I think we could do this,
though maybe there's a better solution.  That is, we can take HTML from
the citation processor and go directly to org-element objects, without
producing and re-parsing citations in Org format.

> The current contenders for a citeproc are Zotero and Pandoc.
>
> Has anyone looked at https://pypi.python.org/pypi/citeproc-py/
> or https://github.com/inukshuk/citeproc-ruby
>
> The ruby one looks pretty advanced.

I haven't looked at them closely.  My impression was that the Python
version was quite incomplete; and unfortunately, I don't know Ruby, so I
would be the wrong person to evaluate it (or write code for it).

Best,
Richard

Re: [O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi Matt and all,

Matt Lundin  writes:

> But for bibtex users, wouldn't there presumably have to be another
> zotero plugin that would allow for live, automated importing of bibtex
> into zotero? (If anyone knows whether such a plugin exists, please do
> let me know.)

Well, my hope is that this could be added to zotxt without much effort,
so we could still just depend on Zotero and zotxt.  The translation
capability already exists in Zotero; it's just a matter of exposing it
as an API, and I imagine that Erik would happily accept a patch to zotxt
that does so.

> A thought experiment... Do we need a fork of pandoc-citeproc?

(The name "org-citeproc" might be a bit misleading: it's not a fork of
pandoc-citeproc, but a small wrapper script that takes care of
marshalling data into and out of Pandoc, in a format that's easy to
generate and read from Emacs lisp.  But because it's written in Haskell,
building it still requires a full Haskell build chain, and distributing
it is non-trivial.)

> Or could we rather write an emacs-lisp wrapper that would feed
> citation data and a bibliography to pandoc and receive a string
> containing citations formatted in org syntax (one of pandoc's
> outputs). This could be done via an export filter, with all the
> necessary manipulation being done on the emacs/org side of things.

Yes, you're basically describing the approach that I eventually realized
org-citeproc should take: use the full capabilities of Pandoc to render
citations and bibliography in Org format, then re-parse these on the Org
side.  I did start to work on this, though I didn't finish and I'm not
sure if I pushed it to the public repo.

If we want to use pandoc-citeproc directly, instead of wrapping it with
something like org-citeproc, what we'd need to do is be able to
translate an Org document (or at least the citations within it) both to
and from pandoc-compatible JSON, since pandoc-citeproc reads and writes
in that format.  This can certainly be done in Emacs Lisp, and maybe it
would be worth doing, because it would mean that we'd gain a nice
serialization format for Org documents.

I am not opposed to this idea -- indeed, I kind of like it, which is why
I started work on org-citeproc in the first place.  Still, it would be a
non-trivial amount of work to develop this solution even to the point
that it can do what Zotero and zotxt can do right now.

> Javascript interpreters/engines are widely available for all platforms
> if we create a wrapper script around citeproc-js. Node itself is also
> easily available for most platforms. But we wouldn't need to set it up
> as a node server à la citeproc-node.

My concern here is with the wrapper script.  Yes, it's pretty easy to
install a javascript interpreter; but getting from there to the point
where you have a fully-working toolchain for processing citations from
Org mode is the problem.  What I think we should avoid is a process that
looks like:

1) Install node (or whatever interpreter)
2) Install citeproc-js and the wrapper script
3) Make sure the wrapper script is available as an executable that can
be called from Emacs
4) ...

Steps 2 and 3 are a bit much to ask people to do manually, and we don't
have the option of doing them through the system package manager.  They
could of course be automated by packaging up citeproc-js and a wrapper
script into a node package.  But again, that requires some work that no
one so far has volunteered to do; and anyway, it doesn't get away from
the worry about having to install packages/plugins from heterogeneous
sources.  (Also, we'd have to develop the wrapper script, and our own
way of translating items stored in BibTeX format into a format
compatible with citeproc-js.  This means duplicating efforts that have
already been put into both pandoc-citeproc and Zotero.)

It's a question of where to focus the limited resources we've got.  My
impression is that going with the combination of Zotero and zotxt will
represent the least amount of effort to get citations working on
non-LaTeX backends, for both Org developers and users.  I understand
that it won't be ideal for everyone, and there is time enough later on
to develop an alternative toolchain if that would better suit the needs
of people who don't want to depend on a GUI, etc.  I fully support that.
But until more people have time to work on this, it seems to me that
Zotero and zotxt represent the most practical path forward.
 
Best,
Richard

Re: [O] Fwd: Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi Matt and all,

Matt Lundin  writes:

> One question (based on complete ignorance of either zotero or zotxt) is
> whether those of us who maintain bibtex databases solely in emacs would
> have to interact with the zotero GUI.

My goal would be that you wouldn't need to interact with it at all.
(You would need to have Firefox running in the background, but that's it
-- no need to actually interact with the window as a user.)

> Let's say I have a bib file and an org document containing
> citations. Would everything place transparently in the background
> without ever having to interact with zotero? Can zotero query a bib
> file without requiring the user manually to import/export the
> database?

I do not know of a way to do this at the moment.  But it may already
exist somewhere, and if it doesn't, I don't think it will be too hard to
build.  (If anyone knows better, please let me know!)
 
Best,
Richard

Re: [O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi Rasmus and all,

Rasmus  writes:

> Also, last I checked Zotero also existed as a standalone manager.  If this
> also works re your points below, using Zotero is a non-issue.  There
> exists a plugin to keep a bibtex file up to date.  I don’t know if it’s
> two way, though.

I haven't looked into whether things work the same way when Zotero is
installed as a standalone program, but I don't imagine they'd be
significantly different.  (What do you mean by "using Zotero is a
non-issue"?  Do you think installing as a standalone program would be
better, or did I misunderstand?)


>> Previously, I thought that it would be a similar amount of work to
>> communicate with Zotero from Emacs as any of the other CSL
>> implementations out there.  However, after looking at zotxt a bit more
>> closely, I discovered that it has an (undocumented) API endpoint [3]
>
> This sounds amazing, but also dangerous.  Do you know whether stabilizing
> the API has been discussed upstream?

As Matt mentioned, "upstream" in this case is zotxt, not Zotero.  I have
heard from Erik that he's open to changing and stabilizing this API.

>> that pretty much does exactly what we need: it accepts a list of
>> citation objects, and returns a list of formatted citations and a
>> formatted bibliography, which can be inserted into the exported
>> document.
>
> Could you give an example of the sort of input you give?  I.e. is it based
> on keys?  From my bibtex-centric world view I imagine something like:
>
>I send key or pointer @K to a DB entry as well as a CSL-file pointer C,
>and maybe a desired output format F.  I get a string back that is the
>formatting of the data behind @K formatted according to the rules in C,
>adapted to F.

Yes, that's correct, except that ideally you send the data for all
citations at once (because context is important).  So for example, if
Erik accepts a patch for some small changes I wrote, one can query

/zotxt/bibliography?outputFormat=html=chicago-fullnote-bibliography

sending POST data like

[
{ "citationItems": [{"key": "0_ZOTKEY1"}],
  "properties": {"noteIndex": 0}},
{ "citationItems": [{"key": "0_ZOTKEY2"}]
  "properties": {"noteIndex": 0}},
...
]

and get back data that basically looks like

{
  "bibliography": [
{
  ...
  "bibstart": "\n",
  "bibend": ""
},
[
  "  Formatted entry for 0_ZOTKEY1\n",
  "  Formatted entry for 0_ZOTKEY2\n",
  ...
]
  ],
  "citationClusters": [
"Formatted citation for 0_ZOTKEY1",
"Formatted citation for 0_ZOTKEY2",
...
  ]
}

> Is that correct?  If so, does it support html, text and odt?

At the moment it supports html and text.  I suppose it could be made to
support ODT, though I'm not sure how difficult it is.  However, I think
a better solution would actually be to have it return *Org* markup, and
then replace citation objects in the document with that.  The main
problem this solves concerns note-based styles: we can insert Org
footnotes into the document at the beginning of the export process, and
then let Org figure out how to number the total set of footnotes in its
usual way.  I think it's a lot harder to deal with citation footnotes if
they come already-formatted in the output format.

>> Erik has also written a package for communicating with zotxt from Emacs,
>> zotxt-emacs [4], which is available on MELPA.  This package already
>> contains a lot of useful functions for querying the Zotero database and
>> inserting reference data into documents, including links in Org
>> documents.  I think it would be pretty straightforward to extend this
>> package to provide a nice UI for writers who are inserting citations
>> into Org documents, including search-based lookups of keys, etc.
>> Perhaps org-ref could also be taught to communicate with zotxt (with or
>> without zotxt-emacs) without too much work.
>
> I guess we’d need to convince Erik to move it to GELPA.  Fortunately (from
> that point of view), neither zotxt nor zotxt-emacs seems to have too many
> contributors.

Is the concern here that Org should not officially depend on anything
that isn't in GELPA?  Or just that it would make installing easier for
users?

>> I know that many people (perhaps especially the `power users' who have
>> been active in the citations discussion so far) prefer to maintain their
>> reference database without the aid of a GUI reference manager like
>> Zotero.
>
> I think this solves part of that issue.  Again it could be a dangerous
> solution.  Alternatively, perhaps it’s possible to feed-in a bibtex file
> to Zotero via an API.  This would also limit the damage.

I am hoping it will prove simple to feed a .bib file to Zotero via an
existing API, or to add such an API to zotxt.  But at the moment that's
just a hope.  (If anyone knows more about how to access a BibTeX
translator in a Zotero plugin, please get in touch!)

> IMO we can leverage zotero as a tool, but we cannot enforce it 

Re: [O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi Matt and all,

Matt Lundin  writes:

> Given these complexities, it seems that if we went the zotero route we
> could end up with a fairly large installation chain (firefox, zotero,
> zotxt, plugin for zotero). And this would require installing items from
> multiple, heterogeneous sources.

Well, I would guess that many people who are interested in this already
have Firefox installed, and after that, you just need to install two
Firefox plugins: Zotero and zotxt.  Open a couple of links, give your
permission, and that's it.

If you're skeptical, I encourage you to try it:

https://www.zotero.org/download/
https://addons.mozilla.org/en-US/firefox/addon/zotxt/

It's pretty easy.  And removing the plugins via about:addons is just as
easy.

> I wonder at this point whether pandoc-citeproc (packaged with pandoc)
> would actually be the simpler route. It can parse bibtex files directly
> and (as a filter within pandoc) can output formatted citations in org
> format.

We have discussed this before, and in fact, I already started work along
this route: see https://github.com/wyleyr/org-citeproc

I stopped because people objected that distributing a Haskell program is
too difficult.  Even if you can install pandoc-citeproc via your
system's package manager, to build org-citeproc against it you need a
complete Haskell build environment, which is (somewhat notoriously)
difficult to work with, and too much to expect for the average person
who just wants citation support in their Org documents.  Nor has anyone
volunteered to take care of building and distributing a binary for every
platform we'd want to support (including, I assume, Windows and OS
X...).

> As a GNU/Linux user, I would find installing zotero and all the add-ons
> messier and more cumbersome than installing pandoc and/or node-js (were
> we to use citeproc-js) from the command line.

I'm a Debian user, so I can appreciate your concern here.  But it's only
simpler to use the system package manager if all the dependencies are
already packaged for $YOUR_DISTRO, in a version that's up-to-date enough
for you to use.  Given the diversity of Org users, it seems likely that
we won't be able come up with a solution that goes via system package
managers that will work for everybody, at least not without a lot of
work.

The nice thing about Firefox (and these days, Emacs) is that it's a sort
of cross-platform package manager.  If the citation processing
dependencies are just Firefox plugins, they'll be much more accessible
to a much wider group of people without a lot of work on our part.  So,
that's why I'd prefer depending on Zotero to depending on something like
org-citeproc or citeproc-node.

Best,
Richard

Re: [O] Fwd: Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi Eric and all,

Eric S Fraga  writes:

> On Tuesday,  1 Dec 2015 at 07:12, Matt Price wrote:
>> I love Zotxt. my only concern is for those people who run a remote Emacs.
>> If, say, you run emacs on a server that you access by ssh, it will not be
>> possible to process your citations with org.
>
> I'm not sure I understand this.
>
> Is the implication that one will need to be connected to the Internet to
> be able to export citations?  If so, this is not good.

No, you should not need to be connected to the Internet.  As I envision
things, Emacs will just speak to the Zotero installation on the local
machine.  That's what zotxt does now.  So, there should be no problem as
long as Firefox, Zotero and zotxt are running on the same machine that
is running Emacs.

I suppose the concern is that someone running a remote Emacs is likely
doing so on a machine that does not have a GUI...and hence may not have
Firefox installed, and consequently won't have Zotero or zotxt.  In that
case, citation processing would have to go another route.

Are there a lot of people in that situation?
 
Best,
Richard

Re: [O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi John and all,

Thanks for your input!

John Kitchin  writes:

> This really means you are using bibtex as the reference database
> backend, and probably helm-bibtex to insert citations as org-mode links.
> Bibliography generation is probably managed by LaTeX/Bibtex. org-ref
> mostly provides the functionality of the links, and some utility
> functions to help build and maintain bibtex files, and to check all the
> cross-reference, label and citation links in the document.
>
> Ideally, something like zotxt would enable Zotero as the reference
> database backend with some selection mechanism (it could be helm, or
> ido, or some other interface) to insert citation syntax

This is pretty much what the zotxt-emacs package does.  It communicates
with the zotxt API, and it defines a minor mode for use with Org.  This
minor mode provides functions for querying the zotxt database (by
default, it searches by author, title and year) and for inserting and
updating Org links that point to items in the Zotero database.

It should be straightforward to extend these capabilities to insert
citation syntax instead of, or in addition to, links.

> , and at export time also leverage Zotero to generate the
> bibliography. I guess this would not too often be export to LaTeX,
> since that would require syncing with a Bibtex file. For quick and
> dirty work this would be fine, but my experience is converted bibtex
> files are never clean enough for production publication, and require a
> lot of manual cleaning of capitalization, journal abbreviations, math,
> etc... It is ok to do it once, but not every time you build a
> document!

Hmm, that is interesting and a bit concerning.  Zotero has the ability
to export sets of items to a BibTeX file.  Ideally this kind of cleanup
should, and in principle could, be dealt with `upstream' in the Zotero
database.  The CSL representation of items allows for pretty
fine-grained and explicit control over things like capitalization and
abbreviations.  But I do not know how much of that is exposed in Zotero,
and I don't know how well it is preserved upon export to BibTeX.

Certainly, I'd want export to LaTeX to be just as quick and painless as
export to other backends in general.  Some manual cleanup of the .bib
file seems acceptable to me as a final step before publication, but I
think it would count against Zotero if it was too annoying to

Zotero can also import BibTeX, so it would also be possible in principle
to go in the other direction: keep your database as .bib, but convert it
on the fly when exporting to other backends.

> A minimal, generalized interface that would probably work with org-ref
> would look something like (* is some prefix name like org-cite-):
>
> *-select-citations-from-db opens some selection interface with
>   candidates from the db-backend ((org-)bibtex, Zotero, Mendeley, RIS,
>   ...), allows user to select some citations with some filtering
>   capability then inserts citation syntax into org file. This function
>   should allow multiple selections, and allow appending to an existing
>   citation. org-ref supports reftex and helm-bibtex for this, and ebib
>   would also work.

As I mentioned above, zotxt-emacs already provides an API in this direction.

> *-export-citations This is the most difficult part, this code would have
>   to functionally do what LaTeX/BibTex accomplishes, and probably with
>   multiple passes to collect citations, get formatted bibliographic
>   entries, replace citations in the text (eg with numbers, au-year,
>   etc...), and insert the formatted bibliography in the right place. If
>   Zotero can give the formatted entry for each citation, this might be a
>   good start. It would be integrated into the export process. It might
>   even be backend-agnostic if the formatting was just plain or in org
>   markup text.

Yep, this is the hard one, but there is some more-or-less working code
to do this (via my org-citeproc script) in the in-progress org-cite.el.
I'm proposing reworking this code to communicate with Zotero, rather
than org-citeproc or another citeproc implementation.

> *-on-cite-click-func This is technically optional, but it is what makes
>   org-ref so great. Figures out what citation was clicked on, and runs
>   some action on that citation, e.g. open citation, find related
>   articles in google scholar, open browser to citation url, open notes
>   to the citation, etc... This is org-ref's most clever code, figuring
>   out what was clicked on, and providing multiple actions on that
>   citation key.

This needs more work, in org-element.  Right now, as far as I know,
there's no support for making citation-reference objects clickable.  On
the other hand, it doesn't seem like it should be that difficult.

> It is pretty easy to use zotero. I couldn't figure out zotxt though.

Hmm, where did you have trouble?  I was able to install it via this link

[O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi everyone,

For the past few days, I've been looking more closely at using the
combination of Zotero [1] with Erik Hetzner's zotxt plugin [2] as a
means of processing citations when exporting to non-LaTeX backends.  I
am now thinking that this is probably our best option, but I'd like to
know what other people think before I sink a lot of work into it.

Here are the reasons I think this is the best option:

1) It is really easy for users.

For those unfamiliar, Zotero is a reference manager, and zotxt is a
Zotero plugin that makes it easier to work with Zotero from plain text
documents.  Both are Firefox plugins, which means they can be installed
by a non-technical user with a couple of clicks.  It also means that
users get updates automatically.  I think this is *really* important.

Pretty much all the other options we have talked about seem like they
will require multi-step, non-trivial installation procedures ("First
install {Node.js/Haskell/JVM ...}, then install
{citeproc-node/pandoc-citeproc/citeproc-java...}, then install our
wrapper script...").  Updating could require other manual operations of
similar complexity.  Avoiding that kind of procedure will make citations
a lot more usable from Org for everyone.

Also, unlike the other options, Zotero is a full-featured reference
manager, not just a batch processor.  So we as users get a useful piece
of software with a simple installation procedure; the other options
require a complex installation procedure for a less-useful program.

2) It is quite complete.

Previously, I thought that it would be a similar amount of work to
communicate with Zotero from Emacs as any of the other CSL
implementations out there.  However, after looking at zotxt a bit more
closely, I discovered that it has an (undocumented) API endpoint [3]
that pretty much does exactly what we need: it accepts a list of
citation objects, and returns a list of formatted citations and a
formatted bibliography, which can be inserted into the exported
document.

This endpoint still needs a little bit of work, to generalize it and
make it easier to get the data in the format we need.  (That is probably
why it is undocumented in the README.) But it requires much less work
than I thought it would, and much less work than it would be to get a
full-featured setup with something like citeproc-node.

Erik has also written a package for communicating with zotxt from Emacs,
zotxt-emacs [4], which is available on MELPA.  This package already
contains a lot of useful functions for querying the Zotero database and
inserting reference data into documents, including links in Org
documents.  I think it would be pretty straightforward to extend this
package to provide a nice UI for writers who are inserting citations
into Org documents, including search-based lookups of keys, etc.
Perhaps org-ref could also be taught to communicate with zotxt (with or
without zotxt-emacs) without too much work.

3) It uses citeproc-js.

In previous discussions, I think we agreed that it would be best for us
to use citeproc-js as a CSL processor, since it is the `canonical' CSL
implementation, as opposed to pandoc-citeproc or citeproc-java.  Zotero
just uses citeproc-js internally to process citations, so it meets this
requirement. 


I know that many people (perhaps especially the `power users' who have
been active in the citations discussion so far) prefer to maintain their
reference database without the aid of a GUI reference manager like
Zotero.  I still think Zotero + zotxt is the best option for non-LaTeX
citation processing, even for these folks.  The ease of installation
(and removal) of the required programs alone makes it worth it, even if
you never actually populate a Zotero database.  So given what I know at
the moment, I think our efforts would best be directed at making the
in-progress org-cite library communicate with Zotero via zotxt.  What do
you think?

Best,
Richard

[1] https://www.zotero.org/
[2] https://gitlab.com/egh/zotxt/
[3] See the bibliographyEndpoint function in:
https://gitlab.com/egh/zotxt/blob/master/extension/bootstrap.js
[4] https://gitlab.com/egh/zotxt-emacs

Re: [O] contributing to work on citations

[Richard Lawrence]

Hi Matt and all,

Matt Price  writes:
>> OK, I tried rebasing on current master, is online here & pull request has
>> been sent ot richard:
>
> https://github.com/titaniumbones/org-mode

Sorry it took me so long to get around to this.  Because there have been
commits to master in the meantime, and because I wanted to clean up the
(unnecessarily complicated) history of my branch, I have performed
another rebase of my branch onto master and put the result in a new
branch, here:

https://github.com/wyleyr/org-mode/tree/wip-cite-rebase

This branch linearizes the history of the various changes that went into
org-element (parsing citations) and org-cite (handling citations), and
cleans up my commit messages, which were produced by an old magit and
pretty ugly to deal with.

Hopefully that will make it easier for someone to continue this work!

Best,
Richard

Re: [O] Some projects

[Richard Lawrence]

Hi Rasmus and all,

Rasmus  writes:

> I would feel more comfortable relying on a JS library.  Perhaps it’s also
> easier to find people who are willing to work on/knows JS over the long
> haul...

Yes, I agree with both you and Aaron here.  JS is easier to distribute
and probably easier to find people to maintain.

>> OTOH, pandoc-citeproc includes a bibtex parser; we’d need to write a JS
>> one and wire it up to citeproc-js.  When I looked (quite some time ago),
>> there did not seem to be any good bibtex parsing libraries in JS (and
>> several third-rate ones).
>
> Bibtex support is essential, of course.
>
> Can someone remind me why citeproc-java isn’t good?  AFAIR, it has a
> bibtex parser.  But probably it lacks in some other dimension...

My recollection is that its command-line tool is not all that flexible,
and thus getting it to produce the kind of output we'd need requires
writing and maintaining Java, which no one really wanted to do.  Also, I
think the thought of firing up the JVM to process citations in an Org
document seemed a little...expensive.

Best,
Richard

Re: [O] Some projects

[Richard Lawrence]

Hi Nicolas and all,

Nicolas Goaziou <m...@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawre...@berkeley.edu> writes:
>
>> Yes, this is my understanding, too.  In particular, there does not seem
>> to be an Elisp CSL library, and it would be a lot of work to write
>> one.
>
> Here is a genuine question: what is the difficult part, or if there is
> none, the longer parts of the work? AFAIU, CSL is XML and we can use
> libxml for the low-level parsing.

I can't speak to how difficult it would really be.  But I do think it
would be a lot of work, and tedious, unglamorous work at that.
Essentially it means implementing the CSL specification:

http://docs.citationstyles.org/en/stable/specification.html

The specification describes the XML format for CSL styles, which IIUC
are something like XSLT programs: they describe how to transform an
input document tree into an output document tree.  So, implementing CSL
is something like implementing a very special-purpose declarative
programming language, which has to handle declarations about every
detail of citation formatting (where to put periods, how to capitalize
names like "von Neumann", whether to use "&" or a translation of "and"
for the current locale, etc.).  Lisp is a good language to do this kind
of thing in, but parsing the XML is surely the easy part.

> Could you give some examples of the UI offered by the citeproc-hs
> library?

The library I ended up wrapping is actually pandoc-citeproc, which is a
fork of citeproc-hs (which seems to be unmaintained).

pandoc-citeproc essentially exports one main function, processCites.
This function transforms a Pandoc data structure containing Citation
nodes into another Pandoc where the citations have been rendered into
the text and the bibliography is appended at the end of the document.

processCites also requires a list of references (read from a
bibliography database, e.g. .bib files, via the readBiblioFile function)
and a CSL style (read from a file via the readCSLFile function).  So
basically, the pandoc-citeproc API consists of these three functions:
processCites, readBiblioFile, and readCSLFile.

Best,
Richard

Re: [O] Some projects

[Richard Lawrence]

Hi all,

Aaron Ecay <aarone...@gmail.com> writes:

> Hi Nicolas,
>
> Thanks for writing this up.  It is important to think about, and
> ultimately solve, all the issues you raise.

Yes, thanks for this, Nicolas!  

> 2015ko urriak 25an, Nicolas Goaziou-ek idatzi zuen:
>> 
>> ** Citations
>> 
>> Development apparently stopped for some reason. We have a citation
>> syntax for Org in wip-cite and some work done in wip-cite-awe and
>> probably elsewhere.
>> 
>> I think we could at least provide features defined in Org Ref using the
>> new syntax (minus hydra/helm related functions).
>> 
>> We don't need a silver bullet. Just something with a non-empty user
>> base, and extensible. In any case, the work done so far shouldn't be
>> wasted.
>
> I was working on this rather intensively at one time, but I had to stop
> because other aspects of life intruded.  I have just been coming back
> towards a situation where I can imagine myself having some (still small,
> but non-zero) chunks of time to devote to working on org.  So I hope I
> will be able to pick this back up, but (regrettably) I’m not able to
> make any promises.
>
> Based on my recollection, here’s what the problems were when I stopped:
>
> - The only “off the shelf”-capable citation processing library that we
>   found last time is in Haskell, which introduced some difficulties for
>   distributing the resulting tool.  I know some projects
>   (e.g. git-annex) are written in Haskell and distributed as static
>   binaries for windows/mac/linux/etc.  We’d need to figure out how to do
>   this, or find another citation processing library in an
>   easier-to-distribute language.

Yes, this is my understanding, too.  In particular, there does not seem
to be an Elisp CSL library, and it would be a lot of work to write one.

The other CSL library that looks complete and usable is citeproc-js; but
like the Haskell library (pandoc-citeproc) it would need to be wrapped
somehow so that it can talk with Org.

It should be relatively straightforward for someone who knows Javascript
to write such a wrapper, if anyone wants to work on that.  But this does
not really solve the problem with distribution.  Either of the
off-the-shelf CSL libraries will require both a wrapper and a platform
for building/installing/running the wrapper and library as a complete
external tool.

> (I should say, all the work on the external tool was done by Richard
> Lawrence; I worked on the exporter for the citation syntax including
> the interface with an external tool.)

The tool I was working on is here:
https://github.com/wyleyr/org-citeproc

The branch of Org that it needs is here:
https://github.com/wyleyr/org-mode

At the moment, it supports single- and multiple-work citations in inline
styles (e.g. Chicago/Harvard type citations, and I think also styles
that use numbered references to the bibliography).  It doesn't presently
work with note-based styles, and making it work will require some
modifications on the Org side.  Specifically, the Org side will have to
get a bit smarter about how it inserts the formatted citations into the
document (Org needs to understand them as footnotes so that they get
correctly numbered, etc. amongst other non-citation footnotes).

> - There is a difference between citations as done by latex/bibtex/etc.,
>   and those done in every other format (handled through CSL).  Assuming
>   latex users want to keep their native processing rather than
>   delegating to CSL, we need to solve the myriad small inconsistencies
>   between these two tools.  I think this is an area where it’s important
>   to get things right: users of citations generally have exacting
>   requirements.  “Approximately Chicago-style” or “almost MLA” aren’t
>   worth anything.

I guess I would just add that it is not clear how much we need to solve
here, at least in the short term.  I can't remember whether we found any
concrete examples of needs people have that BibLaTeX can handle but
CSL cannot, or vice versa.  Anyway, there is a core set of citation
features that both types of backends handle readily, and I think it
would be a big win to have these accessible via a common syntax in Org.
No silver bullets is indeed the maxim to keep in mind.

> (I should also say, if someone else is interested in working on this
> please don’t hesitate to jump right in.  I will help you however I can!)

I also want to echo this.  I don't really have much time to work on this
myself right now (trying to get the ol' dissertation finished this year)
but I will help out however I can.

Best,
Richard

Re: [O] Showing Property in headline; generating table from properties; exporting table to CSV

[Richard Lawrence]

Hi Matt and all,

Matt Price mopto...@gmail.com writes:

 On Thu, Aug 6, 2015 at 9:13 AM, John Kitchin jkitc...@andrew.cmu.edu
 wrote:

 How do you enter your grade? I use a function, bound to a convenient key
 like s-s g, which sets the grade property. You could have that function
 change the heading TODO state to DONE so you know it is done, and maybe
 add a tag with the grade, or just append the grade on the end of the
 headline.

 Adding the grade as a tag doesn't seem quite right, as I often change
 grades after a rewrite.  I'd need to get rid of the original tag.

Here's another idea for getting the grade into the headline: set the
grade as a priority, like

** [#A] John Doe

Pros: it's a simple hack that doesn't require you to do any additional
parsing; it's easy to change up or down either interactively or
programmatically (cf. org-priority* functions); and it's easy to sort
headlines by priority in a custom agenda view, to give you a quick
overall picture of your grade distribution.

Cons: this would only work for A/B/C/D/F grades, and even then, it
doesn't represent +/- variants.  So it's not very flexible or granular,
but it might be enough.  

Best,
Richard

P.S. Like others, I too have written my own grading system:

https://github.com/wyleyr/schoolutils

It stores grades and student data in a SQLite database, and allows you
to write end-of-term grade calculations in Python, instead of messing
with spreadsheets. (Overkill? Maybe so... :)

It currently only has a command line interface, but I have been thinking
about adding an Org interface that would do basic CRUD operations on
grades via Babel.  If anyone is interested in such a thing, let me
know...