Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 10:06 AM John Kitchin  wrote:
>
> Bruce and I looked into this UI approach in 
> https://github.com/jkitchin/org-ref-cite/issues/9. Bruce and I discussed and 
> worked on this for almost two weeks. There are 70 comments in this issue.
>
> There are opportunities now to annotate completion targets, which you can see 
> in the link above. The annotations are not selectable though during 
> completion, and this implementation was not too fast as I recall.

FWIW, an alternative I was playing with is something like this, which
makes use of the new oc-biblatex styles defcustom:

(defcustom style-select-latex-commands nil
  "Whether to use latex commands for style selection."
  :group 'style
  :type '(boolean))

(defun style-latex-alist ( swap)
  "Return org-cite-biblatex-styles as alist.
By default, each car is the latex command, and the cdr the
org-cite style with variant. With SWAP, they are reversed."
  (let ((raw-styles org-cite-biblatex-styles))
(mapcar
 (lambda (s)
   (let* ((style (elt s 0))
  (variant (elt s 1))
  (command (elt s 2))
  (cstyle (concat style (when variant "/") variant)))
 (if swap
 (cons cstyle command)
 (cons command cstyle
 raw-styles)))

(defun style-select ()
  "Select oc style."
  (let* ((latex-commands style-select-latex-commands)
 (styles
 (if latex-commands (style-latex-alist)
   (style-latex-alist t)))
 (choice
  (completing-read
   (if latex-commands "Biblatex command: "
 "Style: ") styles)))
(cdr
 (if style-select-latex-commands (assoc choice (style-latex-alist))
   (rassoc choice (style-latex-alist))

> You probably should also augment the tooltips like this because you have to 
> be able to tell what a citation format is in the future too, not just at 
> insert time, e.g. suppose you are reading work of a collaborator. It was hard 
> to write, and ambiguous in ways, e.g. what is the export backend you want? 
> The annotations should differ for LaTeX (where you want to see the latex 
> command) vs HTML (where you probably want to see the formatted CSL cite)...

I was thinking it'd be enough to have a tooltip preview of the
citation, and allow the actual preview to be configurable.

Bruce

> We did not surmount these challenges at the time. Maybe others will succeed 
> in this another day.
>
> John
>
> ---
> Professor John Kitchin (he/him/his)
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
>
>
>
> On Mon, Mar 21, 2022 at 8:42 AM Bruce D'Arcus  wrote:
>>
>> On Mon, Mar 21, 2022 at 8:23 AM John Kitchin  wrote:
>>
>> >> A package could be created, say `org-cite-literal-biblatex' which is just 
>> >> a copy
>> >> of `oc-biblatex.el' with a different default `org-cite-biblatex-styles' 
>> >> and
>> >> `org-cite-biblatex-style-shortcuts' (or just sets those variables in
>> >> `org-cite-biblatex'). As far as I can tell this would provide exactly the
>> >> functionality you say org-cite can’t provide but org-ref does.
>> >
>> >
>> > I wrote this package you suggest in org-ref-cite. In discussions during 
>> > that development, it was clear the preference was on the more abstracted, 
>> > and uniform syntax across backends cite commands in org-cite, and not this 
>> > kind of variant. Of course one can do this. It is not that org-cite can't 
>> > provide it, it is that it doesn't at this time.
>>
>> Just for some broader context on this particular issue.
>>
>> The advantage of the org-cite style/variant design reflected in the
>> included export processors (natbib, biblatex, csl) is that the same
>> styles will mostly generate the same final output.
>>
>> But that portability will only work with those styles and variants.
>>
>> With the new org-cite-biblatex-styles defcustom, however, one can
>> augment or completely replace all those. But if you care about that
>> portability, you'd want to be aware of this, and think about it.
>>
>> So per Timothy's point, you actually don't even need a new processor
>> for biblatex if you want to include all the extensive list of biblatex
>> commands.
>>
>> Natbib AFAIK is already fully covered.
>>
>> There's another POV on this though:
>>
>> If one doesn't like to see the org-cite styles, because of familiarity
>> with LaTeX commands etc., I would argue that can be addressed in the
>> style part of an insert processor and/or in an activate processor.
>> E.g. I would argue this is a UI issue; not fundamentally about the
>> styles names.
>>
>> Bruce

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Thu, Mar 31, 2022 at 11:27 AM Max Nikulin  wrote:



> >> Emphasis and bold markers may appear in plain text export. Behavior of 
> >> styles is
> >> not uniform in respect to adding (unbreakable?) space before citation.
> >
> > Sorry; not following here again. Isn't the space before a citation
> > determined by the user?
>
> I was lucky enough to pick a couple of styles having different behavior.
> Notice additional unbreakable space before "[1]" in the second example.
> I have checked a couple of IEEE papers and they have spaces before
> citations, so to switch from IEEE to APS style it is necessary to remove
> spaces before citations.

The latter style includes a non-breaking space in the citation definition ...

https://github.com/citation-style-language/styles/blob/21e2177295be5b98ef49b00dd4b8cc7e68d2143d/american-physics-society.csl#L96

... which strikes me as odd, though would explain what you see.

The former has no space.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 29/03/2022 23:14, Bruce D'Arcus wrote:

On Tue, Mar 29, 2022 at 11:23 AM Max Nikulin wrote:

So it is not more general. Switching CSL style means necessity to update
styles in each citations (unless it is possible to specify global or
per-cite mapping).


Not really. Arguably the most important style is "text", which applies
to any output style; author-date, note-based, numeric.

When you start getting into some of the others, the range of styles a
given style may apply to shrinks.


I am from the world of in-text numeric styles. I always considered them 
as almost footnote style since the only difference is brackets vs. 
superscript. Author-date style looks quite distinct from such point of view.



I am familiar with bst language used by BibTeX and I am surprised that
initials instead of full names are not enforced by CSL styles.


I'm not following here. Certainly one can specify initialization rules
in a CSL style.

WDYM by "enforced"?


My complain was wrong (besides it is unrelated to my statement that it 
is impossible to just specify another style, be ready to edit citations).


Some journals omit even article titles to get more compact bibliography. 
I do not remember whether I have seen papers where names are not 
shortened to initials. Any description of BST BibTeX language explains 
how to use format.names$ function, so I was surprised when I got the 
following. However after your question I have checked that achicago.bst 
formats entry with full names as well.


#+cite_export: csl 
/usr/share/citation-style-language/styles/chicago-author-date.csl


--- >8 ---
• Default (Schawlow and Townes 1958).

Schawlow, Arthur Leonard, and Charles Hard Townes. 1958. “Infrared and
Optical Masers.” /Phys. Rev./ 112 (December). American Physical
Society:1940–49. .
--- 8< 


Emphasis and bold markers may appear in plain text export. Behavior of styles is
not uniform in respect to adding (unbreakable?) space before citation.


Sorry; not following here again. Isn't the space before a citation
determined by the user?


I was lucky enough to pick a couple of styles having different behavior. 
Notice additional unbreakable space before "[1]" in the second example. 
I have checked a couple of IEEE papers and they have spaces before 
citations, so to switch from IEEE to APS style it is necessary to remove 
spaces before citations.


Examples are generated using text unicode export.

#+cite_export: csl /usr/share/citation-style-language/styles/ieee.csl
- Default: [cite:@schawlow1958iao].

--- >8 ---
• Default: [1].

[1] A. L. Schawlow and C. H. Townes, [“Infrared and optical masers,]”
/Phys. rev./, vol. 112, pp. 1940–1949, Dec. 1958.


[“Infrared and optical masers,]

--- 8< ---


#+cite_export: csl 
/usr/share/citation-style-language/styles/american-physics-society.csl


--- >8 ---
• Default:  [1].

[1] A. L. Schawlow and C. H. Townes, Phys. Rev. *112*, 1940 (1958).
--- 8< ---

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Max Nikulin  writes:

> On 28/03/2022 20:16, Bruce D'Arcus wrote:
>> On Mon, Mar 28, 2022 at 8:37 AM Max Nikulin wrote:
>>>
>>> John, in another message (Sun, 27 Mar 2022 13:00:40 -0400)
>>> https://list.orgmode.org/m24k3jnq0k@andrew.cmu.edu you clearly
>>> stated a technical limitation that is a real reason why org-cite is not
>>> an option for you and for some other users: performance has not been
>>> optimized for large BibTeX databases. It is deserved to be mentioned.
>> 
>> FWIW, Ihor posted a patch related to this a week or so ago.
>
> I am optimistic concerning that patch since a couple of users confirmed 
> improvement, but it is up to John to decide if it is acceptable in 
> comparison to org-ref. I am unsure concerning startup time.

I think this will be a problem that is processor specific. It is
important that org have a reasonable performance here, but I don't think
it was a goal to have the fastest bibtex/json/etc. parser available,
just a reasonable default that works out of the box. That is, it doesn't
take too long to read the database for insertion, and fontification is
not a performance breaker, on say a moderate sized database and
org-file. Getting high performance from large databases and large files
with lots of citations (say a dissertation or big review article) takes
some work.




-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[Denis Maier]

Am 29.03.2022 um 18:14 schrieb Bruce D'Arcus:

On Tue, Mar 29, 2022 at 11:23 AM Max Nikulin  wrote:
...


You even have managed to convince me that, besides adding missing style
names, some existing ones should be adjusted. noauthor/bare for citeyear
example makes for me much more sense ...


This does need some attention, but there are wrinkles here.

Citeyear is specific to author-date styles, while noauthor is intended
to be more general.


Anyway citation style is rather specific for a particular CSL style. I
tried some styles:
https://github.com/citation-style-language/styles/blob/master/ieee.csl
https://github.com/citation-style-language/styles/blob/master/american-physics-society.csl
nature.csl science.csl and for all these styles even "author" is
meaningless since they are numeric styles.


Yes. I think it's more relevant in author-date to note styles. I
believe biblatex has a command relevant here, but Denis knows biblatex
better than I.


I'm not sure I understand the question here. What command should be in 
biblatex? There's \citeauthor if that's what you've meant.





So it is not more general. Switching CSL style means necessity to update
styles in each citations (unless it is possible to specify global or
per-cite mapping).


Not really. Arguably the most important style is "text", which applies
to any output style; author-date, note-based, numeric.

When you start getting into some of the others, the range of styles a
given style may apply to shrinks.

But you might say author-date styles are pretty dependent on such
local citation modification. If those are output to a style that has
no such notions (like a numeric one), then a processor can just ignore
it.


Just to add to this: When Bruce and I have worked on that list of styles 
we found that portability can only be ensured when using high-level 
commands (such as biblatex's autocite), but once you start using 
low-level commands like citeyear etc. you really lose that portability 
to a certain degree.





It seems modifiers are set of boolean flags (positive "year" or negative
"suppress-author") in citeproc.el, set of values in natbib, and a kind
of hierarchy in org-cite. From my point of view, set of constrains
(flags) is the most general variant in this list.


I think that's right, and is how it's represented in a GUI app like
Zotero. But that's not so convenient in a plain text format.

But it's a good way to explain the differences.


I think it's probably a good idea to add "year" to the latex processors too.


I agree. Negations are more confusing when an author needs just year.


Well, negations have the advantage of being more portable. Say you have 
this:


Doe argues X and Y [cite/noauthor:@doe].

It's perfectly clear what this should mean in a author-year, 
author-title or note-based styles, i.e., print the citation without the 
author element. (That's obviously a simplification since some citations 
might not have an author element, but let's just go with it for the moment.)


In a numeric style you can just ignore the noauthor modifier and fall 
back to the default numeric citation.


Now, consider this instead:

Doe argues X and Y [cite/year:@doe].

This might work in author-year styles, but not in author-title, not in 
note-based styles, and not in numeric styles.


Considering the problem that some citations don't have an author element 
I even considered using style names like



[cite/head:@doe]
[cite/tail:@doe]
or
[cite/car:@doe]
[cite/cdr:@doe]
or
[cite/first:@doe]
[cite/rest:@doe]

But that obviously a bit esoteric.

Denis

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Tue, Mar 29, 2022 at 11:23 AM Max Nikulin  wrote:
...

> >> You even have managed to convince me that, besides adding missing style
> >> names, some existing ones should be adjusted. noauthor/bare for citeyear
> >> example makes for me much more sense ...
> >
> > This does need some attention, but there are wrinkles here.
> >
> > Citeyear is specific to author-date styles, while noauthor is intended
> > to be more general.
>
> Anyway citation style is rather specific for a particular CSL style. I
> tried some styles:
> https://github.com/citation-style-language/styles/blob/master/ieee.csl
> https://github.com/citation-style-language/styles/blob/master/american-physics-society.csl
> nature.csl science.csl and for all these styles even "author" is
> meaningless since they are numeric styles.

Yes. I think it's more relevant in author-date to note styles. I
believe biblatex has a command relevant here, but Denis knows biblatex
better than I.

> So it is not more general. Switching CSL style means necessity to update
> styles in each citations (unless it is possible to specify global or
> per-cite mapping).

Not really. Arguably the most important style is "text", which applies
to any output style; author-date, note-based, numeric.

When you start getting into some of the others, the range of styles a
given style may apply to shrinks.

But you might say author-date styles are pretty dependent on such
local citation modification. If those are output to a style that has
no such notions (like a numeric one), then a processor can just ignore
it.

> It seems modifiers are set of boolean flags (positive "year" or negative
> "suppress-author") in citeproc.el, set of values in natbib, and a kind
> of hierarchy in org-cite. From my point of view, set of constrains
> (flags) is the most general variant in this list.

I think that's right, and is how it's represented in a GUI app like
Zotero. But that's not so convenient in a plain text format.

But it's a good way to explain the differences.

> > I think it's probably a good idea to add "year" to the latex processors too.
>
> I agree. Negations are more confusing when an author needs just year.

We might as well do that then, along with bibentry/fullcite.

...

> I am familiar with bst language used by BibTeX and I am surprised that
> initials instead of full names are not enforced by CSL styles.

I'm not following here. Certainly one can specify initialization rules
in a CSL style.

WDYM by "enforced"?

> Emphasis and bold markers may appear in plain text export. Behavior of styles 
> is
> not uniform in respect to adding (unbreakable?) space before citation.

Sorry; not following here again. Isn't the space before a citation
determined by the user?

Bruce

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 28/03/2022 20:16, Bruce D'Arcus wrote:

On Mon, Mar 28, 2022 at 8:37 AM Max Nikulin wrote:


John, in another message (Sun, 27 Mar 2022 13:00:40 -0400)
https://list.orgmode.org/m24k3jnq0k@andrew.cmu.edu you clearly
stated a technical limitation that is a real reason why org-cite is not
an option for you and for some other users: performance has not been
optimized for large BibTeX databases. It is deserved to be mentioned.


FWIW, Ihor posted a patch related to this a week or so ago.


I am optimistic concerning that patch since a couple of users confirmed 
improvement, but it is up to John to decide if it is acceptable in 
comparison to org-ref. I am unsure concerning startup time.



You even have managed to convince me that, besides adding missing style
names, some existing ones should be adjusted. noauthor/bare for citeyear
example makes for me much more sense ...


This does need some attention, but there are wrinkles here.

Citeyear is specific to author-date styles, while noauthor is intended
to be more general.


Anyway citation style is rather specific for a particular CSL style. I 
tried some styles: 
https://github.com/citation-style-language/styles/blob/master/ieee.csl 
https://github.com/citation-style-language/styles/blob/master/american-physics-society.csl 
nature.csl science.csl and for all these styles even "author" is 
meaningless since they are numeric styles.


So it is not more general. Switching CSL style means necessity to update 
styles in each citations (unless it is possible to specify global or 
per-cite mapping).


It seems modifiers are set of boolean flags (positive "year" or negative 
"suppress-author") in citeproc.el, set of values in natbib, and a kind 
of hierarchy in org-cite. From my point of view, set of constrains 
(flags) is the most general variant in this list.



I think it's probably a good idea to add "year" to the latex processors too.


I agree. Negations are more confusing when an author needs just year.


"Year" would be much less portable. That doesn't mean that "year"
couldn't be legitimate, in the sense of "I really need the year here".


I would say that if a citation style has no year than a warning should 
be spat, but year should not appear since it is against the style. 
Unsure if it is useful to have special command or style "query" that can 
insert e.g. year, journal, or any other field disregarding style to 
ensure that the text and the bibliography reference have not diverged, 
but it is rather for descriptive text than for a citation.



But that raises another question: How will citeproc-el know where this
year is coming from, and how the year has to be formatted? That will
have to be hardcoded in the processor as styles contain no information
about this.


A fallback style (for the particular citation, for the whole document or 
in global user preferences) might be specified for formatters missed in 
the main style.


I have tried the following:

obt.bib:
@ARTICLE{schawlow1958iao,
  author = {Schawlow, Arthur Leonard and Townes, Charles Hard},
  title = {Infrared and Optical Masers},
  journal = {Phys. Rev.},
  year = {1958},
  volume = {112},
  pages = {1940--1949},
  month = {Dec},
  doi = {10.1103/PhysRev.112.1940},
  issue = {6},
  publisher = {American Physical Society},
}

obt.org:
#+bibliography: obt.bib
# #+cite_export: csl
# #+cite_export: csl 
/usr/share/citation-style-language/styles/chicago-author-date.csl
# #+cite_export: csl 
/usr/share/citation-style-language/styles/american-physics-society.csl

- Default: [cite:@schawlow1958iao].
- text: [cite/text:@schawlow1958iao].
- noauthor: [cite/noauthor:@schawlow1958iao].
- year/bare: [cite/year/bare:@schawlow1958iao].

#+print_bibliography:

I am familiar with bst language used by BibTeX and I am surprised that 
initials instead of full names are not enforced by CSL styles. Emphasis 
and bold markers may appear in plain text export. Behavior of styles is 
not uniform in respect to adding (unbreakable?) space before citation.

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 28, 2022 at 8:37 AM Max Nikulin  wrote:
>
> On 28/03/2022 02:40, John Kitchin wrote:
> > Max Nikulin writes:
> >> On 21/03/2022 18:51, John Kitchin wrote:
> >
> > Rather than rehash a lot of experiences, I really encourage you to try
> > writing a processor that can support this. Or even, try modifying
> > org-ref-cite to support it. Not as some thought experiment about what
> > should be possible, but an actual experiment that is worked out.
>
> I have some ideas for links, other inline objects and export attributes
> that I should try before.
>
> >> In particular I am worrying concerning https://github.com/jkitchin/org-ref
> >> README (and the same phrase from the earlier message):
> >>
> >>> org-cite does not meet my citation and technical document publishing 
> >>> needs,
> >>> and it was not possible to integrate it into org-ref without compromising
> >>> those.
> >
> > I have taken this out of the readme. I still agree with the sentiment,
> > but my needs are not the same as others (for example, I include in my
> > needs include ease of support and development, which is not a user
> > need), and it is possible to meet some basic needs fully.
>
> John, in another message (Sun, 27 Mar 2022 13:00:40 -0400)
> https://list.orgmode.org/m24k3jnq0k@andrew.cmu.edu you clearly
> stated a technical limitation that is a real reason why org-cite is not
> an option for you and for some other users: performance has not been
> optimized for large BibTeX databases. It is deserved to be mentioned.

FWIW, Ihor posted a patch related to this a week or so ago.

> You even have managed to convince me that, besides adding missing style
> names, some existing ones should be adjusted. noauthor/bare for citeyear
> example makes for me much more sense ...

This does need some attention, but there are wrinkles here.

Citeyear is specific to author-date styles, while noauthor is intended
to be more general.

Hence, initially "noauthor".

This indeed was influenced by CSL implementations like Zotero and
pandoc, which have a notion of "suppress author".

But the names are kind of awkward admittedly, and Andras Simonyi
subsequently added a "year" style in oc-csl.

You can see the conversation about it here.

https://github.com/andras-simonyi/oc-csl-ns/issues/1#issuecomment-895440897

I think it's probably a good idea to add "year" to the latex processors too.

I will reproduce Denis' explanation from that linked comment here:

---
The reasoning behind "noauthor" vs "year" was that our priority was
portability between different classes of citation styles. If we have
this item "John, Doe. /A book/. 2020." the variant "noauthor" can be
used to render this consistently.

In-Text:
a) author-year: "(2020)"
b) author-title: "(/A book/)"

Author-title in footnote: "/A book/. 2020."

Here, it's not so much about what information should be rendered, but
rather what should be ommitted.

"Year" would be much less portable. That doesn't mean that "year"
couldn't be legitimate, in the sense of "I really need the year here".
But that raises another question: How will citeproc-el know where this
year is coming from, and how the year has to be formatted? That will
have to be hardcoded in the processor as styles contain no information
about this.


Bruce


Bruce

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 28/03/2022 02:40, John Kitchin wrote:

Max Nikulin writes:

On 21/03/2022 18:51, John Kitchin wrote:


Rather than rehash a lot of experiences, I really encourage you to try
writing a processor that can support this. Or even, try modifying
org-ref-cite to support it. Not as some thought experiment about what
should be possible, but an actual experiment that is worked out.


I have some ideas for links, other inline objects and export attributes 
that I should try before.



In particular I am worrying concerning https://github.com/jkitchin/org-ref
README (and the same phrase from the earlier message):


org-cite does not meet my citation and technical document publishing needs,
and it was not possible to integrate it into org-ref without compromising
those.


I have taken this out of the readme. I still agree with the sentiment,
but my needs are not the same as others (for example, I include in my
needs include ease of support and development, which is not a user
need), and it is possible to meet some basic needs fully.


John, in another message (Sun, 27 Mar 2022 13:00:40 -0400) 
https://list.orgmode.org/m24k3jnq0k@andrew.cmu.edu you clearly 
stated a technical limitation that is a real reason why org-cite is not 
an option for you and for some other users: performance has not been 
optimized for large BibTeX databases. It is deserved to be mentioned. 
You even have managed to convince me that, besides adding missing style 
names, some existing ones should be adjusted. noauthor/bare for citeyear 
example makes for me much more sense than general words that org-cite is 
too CSL-oriented. So the opinion that the set of styles has not really 
settled yet may be precious for users choosing between org-cite and org-ref.


From my point of view such arguments are more clear than your earlier 
message (Sun, 20 Mar 2022 20:31:29 -0400) 
https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu I had no 
intention to force to remove that phrase before the issues are fixed. I 
was trying to ask you about more details behind your opinion (and now we 
have them).

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 27, 2022 at 12:00 PM John Kitchin  wrote:
>
>
> Nicolas Goaziou  writes:
>
> > Hello,
> >
> > Max Nikulin  writes:
> >
> >> Nicolas, concerning a new thread, I have an impression that you are
> >> busy with over activities since you are participating in discussions
> >> not so frequently. So I am unsure at which moment it is appropriate to
> >> raise such question that otherwise may just be buried in the list
> >> archive.
> >
> > I don't see how my presence (or not) on the list relates to this. If
> > there's an idea worth a discussion, it should not be buried within
> > a thread.
> >
> >> Outside of Org, citations are links.
> >
> > But we're on an Org mailing list so…
> >
> >> I consider fixed set of properties as a problem.
> >
> > I don't think it is a problem for citations.
> >
> >> At first I tried to draw attention to additional link attributes.
> >
> > I think link attributes were discussed a couple of times on this ML
> > already. Nothing was implemented tho.
>
> I don't think any further implementation outside the existing link
> framework is required to do this. The only limitations of the current
> framework are (I think) it is limited to a single line (i.e. not
> multiline), and it would be difficult to have nested links.

I'm glad to hear that.

FWIW, by asking earlier about possible "incremental improvements" for
cross-references, I was including within that possible implementation
improvements: new helper functions or interactive commands, standard
link types, documentation, etc.

I note, for example, that currently local links do export as LaTeX
cross-references, and that if you use cleverref, the results seem
pretty reasonable from my non-expert POV.

But it's not easy from a UX POV to create those cross-references,
particularly with a longer, more complex, document.

And in my coding experiments [1] awhile back, it wasn't that
straightforward to create a UI to make that easier, mostly because it
wasn't unambiguous how to consistently parse common targets (sections,
figures, tables, equations) [2].

Bruce

[1] https://github.com/bdarcus/oxr
[2] 
https://github.com/bdarcus/oxr/blob/28ec1345e9c8d659afea6c8569479b667697eaa2/oxr.el#L108-L128

> Otherwise, you can put most things in the path, and then parse it as you
> see fit. I think it would be interesting to couple this with a recursive
> descent parser one day, and some DSL perhaps.
>
> There is a prototype of this idea at
> https://kitchingroup.cheme.cmu.edu/blog/2015/02/05/Extending-the-org-mode-link-syntax-with-attributes/
>
> I wouldn't claim it is the best way to do that, or the right thing to
> do.
>
> scimax-editmarks is a step further
> (https://github.com/jkitchin/scimax/blob/master/scimax-editmarks.org)
> that doesn't use org-links or any org-syntax for something more like an
> inline object. It mostly addresses the multiline issue, but it also
> doesn't support nested objects, mostly because of my limited knowledge
> of recursive parsing. I would not advocate for putting this in org
> either though.
>
> If you are interested in this kind of thing, you might find
> https://www.emacswiki.org/emacs/linkd.el interesting too. It leverages
> an S-expression approach, which makes the parsing and nesting more
> straightforward.
>
> I know this is a little off-track of the original thread, but I agree
> with Nicolas that it does not seem necessary at this point to add this
> into org.
>
> >
> > I'm not convinced Org should generalize this to any inline object,
> > either, mainly because it sounds messy. Of course, if you have an
> > idea on the subject, please share it.
> >
> > In any case, this is another topic, neither related to citations nor to
> > cross-references.
> >
> >> For citations some values may be passed to specific citation backend
> >> overriding default value derived from style.
> >
> > In that situation, you can define a new style specific to the citation
> > back-end.
> >
> > Regards,
>
>
> --
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
> Pronouns: he/him/his
>

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 27, 2022 at 3:41 PM John Kitchin  wrote:

...

> Regarding that org-cite adds an abstraction layer, how else could one
> interpret this (from
> https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-syntax)
> other than abstraction:
>
> [cite/na/b:@key] or [cite/noauthor/bare:@key] to mean \citeyear{key}?
>
> Why wouldn't it be \citetitle? or \citeurl, or \citedate? or even,
> \citenum?

You mean why shouldn't we privilege natbib, as you have in org-ref?

And let me turn the question around: how would you propose to
translate those natbib-derived commands to biblatex, or CSL, so it
works reliably across users and documents? The mapping has to happen
someplace, after all.

And from a UX POV, how well would that work for users who have no
previous experience with natbib or even latex?

...

> It might be a social problem, and I do not think it is trivial to solve.
> It is not just about having a syntax that works. I wrote and shared a
> whole set of processors for org-cite that did or tried to do all those
> things above. It was fine to use, but it was very difficult code to
> write for me. I don't personally want to support it in part because it
> was so difficult to write.

I think what Nicolas is asking is when you have time, to itemize the
pain points that made development difficult for you, so that we might
figure out how to improve them (perhaps new helper functions, etc.).

As another data point, one of the things I've loved about org-cite is
how easy I found it to develop pretty functional processors for citar
with minimal code. Total LOC for citar is just under 2000, with just
under 400 specific to org.

But I'm deliberately developing a small, focused, modular, package.

...

> Some motivation for org-cite stemmed from at least perceived limitations
> in org-ref, especially related to pre/post notes and CSL support. I
> think it is totally reasonable to learn if those were real limitations
> or not.

The org-cite citation syntax and model, I hope you would agree, is
unambiguously better than natbib or the original org-ref syntax. It's
simpler than biblatex in the sense of no difference between single and
multicite citations, but can easily and losslessly map to and from
either.

Org-ref 3.0 adds essentially a copy of that syntax and model, with a
trivial difference.

To me, that's the biggest problem. Aside from users having
incompatible documents, it forces other developers either to dedicate
additional development and maintenance time to supporting both
syntaxes, or to choose one.

Pandoc only supports org-cite.

In the case of citar, I have also decided to only support org-cite
(though I leave the function to generate citations configurable, so
it's easy enough for a user to configure this themselves; but I don't
include this by default because I have other processor code that
relies on parsed citations).

Org-roam supports all three.

It sounds like the biggest hold up was with reconciling the org-ref
command model with more general approach of the oc style and variants.

But as a first step, you could do as you originally planned to: simply
use the same names for styles. If Nicolas were to allow the mapping in
natbib to be handled via the defcustom, you could even do this without
having to write and maintain your own export processor.

And then you could save the other part (how to map those to other
export processors) for another step, if and when you or your users
need it or want it.

Certainly that would address the most fundamental incompatibility.

I guess to be direct: what value does the v3 syntax provide you, your
users, or the org ecosystem in general?

Bruce

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Max Nikulin  writes:

> On 21/03/2022 18:51, John Kitchin wrote:
>> citenum and bibentry are the only two I am not sure have a CSL analog.
>
> I read your messages once more and I should say that I feel some disagreement 
> of
> this one (I removed most of it) and the earlier and longer one from Sun, 20 
> Mar
> 2022 20:31:29 -0400 https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu
>
> I admit that org-ref is carefully tuned to your workflow. I hope, it is 
> possible
> to left aside decomposition of org-cite into modules for some time.
>
> Let's assume org-cite with natbib backend for citations and org-ref for
> cross-references. It seems, a couple of missed styles currently is not a 
> problem
> due to the defcustom for the mapping.
>
> Are there still any technical limitations that prevent getting in the exported
> LaTeX file the same citation commands as for org-ref?

If all one wants is LaTeX export, there are not technical limitations.
That is not all people want in general though, they also want other
exports, and that they are reasonably similar, or even possible. It is
easy to get \citenum and \bibentry for LaTeX. I promise though, someone
wants this or some other thing for HTML. 

Rather than rehash a lot of experiences, I really encourage you to try
writing a processor that can support this. Or even, try modifying
org-ref-cite to support it. Not as some thought experiment about what
should be possible, but an actual experiment that is worked out.

> In particular I am worrying concerning https://github.com/jkitchin/org-ref
> README (and the same phrase from the earlier message):
>
>> org-cite does not meet my citation and technical document publishing needs,
>> and it was not possible to integrate it into org-ref without compromising
>> those.

I have taken this out of the readme. I still agree with the sentiment,
but my needs are not the same as others (for example, I include in my
needs include ease of support and development, which is not a user
need), and it is possible to meet some basic needs fully.

> Does it refer to exported result or to convenience of working with citations?
> Would it help if it were possible to choose style by its natbib command?
>
> I see that you do not like org-cite styles, but I can not figure out what are
> the real blockers that prevent producing documents having the same
> quality.

If you spend enough time teaching people how to use these things, you
might feel differently about this. It is so easy to just switch away to
Endnote, Papers, Paperpile, Mendeley, Zotero, Overleaf, etc. I guess I
am not on the prevailing side here though.


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Nicolas Goaziou  writes:

> Hello,
>
> John Kitchin  writes:
>
>> I do not think it is productive for the community to say or consider it
>> is a sad situation. Many good things have emerged from these
>> discussions, even if it is not yet consensus on a solution. It is a
>> complex problem, with many years of effort by many people on each side.
>> That is an indication of how ambitious this project is and that there
>> may be more than one solution that is needed.
>
> [...]
>
>> There are more than 8 years of legacy org-ref documents. I have written
>> 40+ scientific papers with it, and countless technical documents with
>> more than 8000 cite links among them. org-ref has exceeded 190K
>> downloads from MELPA, so I feel obligated to maintain org-ref for
>> myself, and those users. org-ref may be heavyweight in bundling a lot of
>> capability together that could be separated into individual packages,
>> but it is also convenient for people who need it, and a GitHUB issue or
>> pull request away from new features. I remain committed to supporting
>> this, and I do it in a way I can manage, hence the monolithic package
>> design.
>
> I think there's a misunderstanding here. Org Cite was never meant as
> a replacement for Org Ref. It was designed from the beginning as
> a framework other libraries could rely on and extend in their own way.
> Org Ref could have been one of them.
>
> It looks like a social problem to me. I remember well asking for
> feedback when implementing the various prototypes, i.e., ways to make
> Org Cite more useful to other libraries. I don't think I got much of it,
> barring the cross-references topic, which is not solved. Maybe I was not
> explicit enough about what I was expecting. For example, this is—please
> correct me if I'm wrong—the first time I read about the "BibLaTeX is not
> fully implemented in Org Cite" and "Org Cite is adding an extra
> abstraction layer on top of BibLaTeX commands" issues, which are both
> trivial to solve, either on the Org Cite or the Org Ref side. But then
> again, it is perfectly fine if Org Cite doesn't provide that, as some
> libraries can extend it if needed.

I don't think it is the first time I have mentioned biblatex is not
fully implemented, but I am not sure. I have mentioned things like
\citenum somewhere, but it is not the main point.

I know it is not that difficult to add support for LaTeX export in
org-cite, e.g. [cite/num:]. I don't think it is all that easy to add
additional backend support though, for something like [cite/num:] in
HTML or other backends. No doubt, it is not impossible, if someone would
just do it, and that might include extending citeproc too, or developing
some pre-processing function to get a citation number, or something
else. Whether cite/num or any other command exists is not the main
point. What is the point is what mechanisms exist to support the
addition, and the exports to various backends.

Regarding that org-cite adds an abstraction layer, how else could one
interpret this (from
https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-syntax)
other than abstraction:

[cite/na/b:@key] or [cite/noauthor/bare:@key] to mean \citeyear{key}?

Why wouldn't it be \citetitle? or \citeurl, or \citedate? or even,
\citenum?

I get it, you can define [cite/noauthor/year:] or even [cite/year:] or
[cite/y:] and even [cite/citeyear:] to get the command in there, and
something for each of those other ones. Maybe even the documented
convention will change to some other potentially mnemonic form. 

I also get they are not all that common perhaps, except for a few people
who use them, and maybe should be responsible for supporting it
themselves, either by defcustom or their own library.

This is just trading a proliferation of links for a proliferation of
styles IMO, and it feels a lot like the XKCD standards issue
https://xkcd.com/927/.

As others have argued, it is just a bit of knowledge, maybe a UI can
compensate to help you insert what you want, then know what it means
later. It is my opinion that this will be a technical debt though. I am
content to agree to disagree on this point.

It might be a social problem, and I do not think it is trivial to solve.
It is not just about having a syntax that works. I wrote and shared a
whole set of processors for org-cite that did or tried to do all those
things above. It was fine to use, but it was very difficult code to
write for me. I don't personally want to support it in part because it
was so difficult to write. I don't even want to support it for my own
use at this time. This should not stop anyone who wants to do that
themselves though. Maybe there is a cleaner approach I missed, or others
may be better programmers, and/or have more time to figure this out. At
this time, I do not have time to make for it.

> On the other hand, there have been much activity on GitHub repositories,
> i.e., out of this mailing list. It seems to me Org Ref project has been
> trying to 

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Nicolas Goaziou  writes:

> Hello,
>
> Max Nikulin  writes:
>
>> Nicolas, concerning a new thread, I have an impression that you are
>> busy with over activities since you are participating in discussions
>> not so frequently. So I am unsure at which moment it is appropriate to
>> raise such question that otherwise may just be buried in the list
>> archive.
>
> I don't see how my presence (or not) on the list relates to this. If
> there's an idea worth a discussion, it should not be buried within
> a thread.
>
>> Outside of Org, citations are links.
>
> But we're on an Org mailing list so…
>
>> I consider fixed set of properties as a problem.
>
> I don't think it is a problem for citations.
>
>> At first I tried to draw attention to additional link attributes.
>
> I think link attributes were discussed a couple of times on this ML
> already. Nothing was implemented tho.

I don't think any further implementation outside the existing link
framework is required to do this. The only limitations of the current
framework are (I think) it is limited to a single line (i.e. not
multiline), and it would be difficult to have nested links.

Otherwise, you can put most things in the path, and then parse it as you
see fit. I think it would be interesting to couple this with a recursive
descent parser one day, and some DSL perhaps.

There is a prototype of this idea at
https://kitchingroup.cheme.cmu.edu/blog/2015/02/05/Extending-the-org-mode-link-syntax-with-attributes/

I wouldn't claim it is the best way to do that, or the right thing to
do.

scimax-editmarks is a step further
(https://github.com/jkitchin/scimax/blob/master/scimax-editmarks.org)
that doesn't use org-links or any org-syntax for something more like an
inline object. It mostly addresses the multiline issue, but it also
doesn't support nested objects, mostly because of my limited knowledge
of recursive parsing. I would not advocate for putting this in org
either though.

If you are interested in this kind of thing, you might find
https://www.emacswiki.org/emacs/linkd.el interesting too. It leverages
an S-expression approach, which makes the parsing and nesting more
straightforward.

I know this is a little off-track of the original thread, but I agree
with Nicolas that it does not seem necessary at this point to add this
into org.

>
> I'm not convinced Org should generalize this to any inline object,
> either, mainly because it sounds messy. Of course, if you have an
> idea on the subject, please share it.
>
> In any case, this is another topic, neither related to citations nor to
> cross-references.
>
>> For citations some values may be passed to specific citation backend
>> overriding default value derived from style.
>
> In that situation, you can define a new style specific to the citation
> back-end.
>
> Regards,


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[Vikas Rawal]

I have updated the readme in org-ref to indicate you can use both
> packages.
>
> Thanks.

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

I have updated the readme in org-ref to indicate you can use both
packages.

Max Nikulin  writes:

> On 21/03/2022 18:51, John Kitchin wrote:
>> Vikas Rawal writes:
>>>
>>>  From the perspective of a user, this was only meant to express a
>>> sentiment that one finds oneself in a situation of having to choose
>>> between two good things, and that we have not been able to find a way
>>> to make both compatible with each other. It was in not meant as a
>>> disrespect in any way.
>> I don't think you have to choose. You can use org-cite for
>> citations, and org-ref for cross-references. The citation syntax is
>> orthogonal, you just should not mix them. You can even wire org-ref to
>> use org-cite-insert like this:
>> (setq org-ref-insert-cite-function (lambda () (org-cite-insert nil)))
>
> I am glad to read this. John, could you, please, update the README file for
> https://github.com/jkitchin/org-ref to clarify that both packages may be used
> within the same document while org-ref is used solely for cross-references (I
> may create a github issue if your prefer)? From my point of view the following
> phrase may be considered as prohibitive in respect to combining the packages 
> for
> any purpose:
>
>> You can use both org-cite and org-ref (although you should not do that in the
>> same document as they are independent citation tools).
> I feel some ambiguity in the starting message in this thread:
>
> Vikas Rawal, Sun, 20 Mar 2022 17:38:30 +0530.
> https://list.orgmode.org/caltzab2bhuldoxaamuzfqh2h453ekb6k7bkohbu-dhpn98a...@mail.gmail.com
>
>> This obviously creates many problems including that two people using
>> different citation systems cannot share org files.
> Accordingly to the org-ref README installed org-ref package does not break
> org-cite, so people are free to share documents. They just should have both
> packages configured and should adhere to a chosen package in each document 
> (for
> *citations*).


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[M. Pger]

On Wednesday, March 23rd, 2022 at 3:39 PM, Bruce D'Arcus  
wrote:

> Have you actually looked at the table of existing mappings? See table 1 here 
> (which it seems we might want to add to the manual?):
>
> https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-syntax

Sorry for the late reaction to this. As an user I completely agree. IMO, having 
these mappings in the doc could clearly convince some org-ref users to give a 
try to org-cite. Some information about the possibility of using .bst 
bibliography style files would also be nice. Finally, some info about cross-ref 
could also be useful, so that users easily get an idea of what would imply a 
switch from org-ref to org-cite.

For what it's worth, I think that Vanilla org becomes more and more amazing, 
thanks for that.

Best,

MP

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

I obviously can't speak for John, but on this:

On Fri, Mar 25, 2022 at 1:11 PM Max Nikulin  wrote:
>
> On 21/03/2022 18:51, John Kitchin wrote:
> >
> > citenum and bibentry are the only two I am not sure have a CSL analog.

As I said in an earlier message, it's no problem to add these. Someone
can always add the bibentry/fullcite style to oc-csl later, if and
when Andras adds the functionality to citeproc-el.

Though it's feasible, I doubt you'll ever see citenum in CSL, since
those sorts of low-level commands don't really fit there.

But that also is not a problem.

> I read your messages once more and I should say that I feel some
> disagreement of this one (I removed most of it) and the earlier and
> longer one from Sun, 20 Mar 2022 20:31:29 -0400
> https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu
>
> I admit that org-ref is carefully tuned to your workflow. I hope, it is
> possible to left aside decomposition of org-cite into modules for some time.
>
> Let's assume org-cite with natbib backend for citations and org-ref for
> cross-references. It seems, a couple of missed styles currently is not a
> problem due to the defcustom for the mapping.

The mapping defcustom is currently only in oc-biblatex. It might be
worth adding it to oc-natbib, and so generalizing how these mappings
are handled in latex?

Bruce

> Are there still any technical limitations that prevent getting in the
> exported LaTeX file the same citation commands as for org-ref?
>
> In particular I am worrying concerning
> https://github.com/jkitchin/org-ref README (and the same phrase from the
> earlier message):
>
> > org-cite does not meet my citation and technical document publishing
> > needs, and it was not possible to integrate it into org-ref without
> > compromising those.
> Does it refer to exported result or to convenience of working with
> citations? Would it help if it were possible to choose style by its
> natbib command?
>
> I see that you do not like org-cite styles, but I can not figure out
> what are the real blockers that prevent producing documents having the
> same quality.
>
>

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 21/03/2022 18:51, John Kitchin wrote:


citenum and bibentry are the only two I am not sure have a CSL analog.


I read your messages once more and I should say that I feel some 
disagreement of this one (I removed most of it) and the earlier and 
longer one from Sun, 20 Mar 2022 20:31:29 -0400 
https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu


I admit that org-ref is carefully tuned to your workflow. I hope, it is 
possible to left aside decomposition of org-cite into modules for some time.


Let's assume org-cite with natbib backend for citations and org-ref for 
cross-references. It seems, a couple of missed styles currently is not a 
problem due to the defcustom for the mapping.


Are there still any technical limitations that prevent getting in the 
exported LaTeX file the same citation commands as for org-ref?


In particular I am worrying concerning 
https://github.com/jkitchin/org-ref README (and the same phrase from the 
earlier message):


org-cite does not meet my citation and technical document publishing 
needs, and it was not possible to integrate it into org-ref without 
compromising those.
Does it refer to exported result or to convenience of working with 
citations? Would it help if it were possible to choose style by its 
natbib command?


I see that you do not like org-cite styles, but I can not figure out 
what are the real blockers that prevent producing documents having the 
same quality.

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 24/03/2022 06:04, Nicolas Goaziou wrote:

Max Nikulin writes:


Nicolas, concerning a new thread, I have an impression that you are
busy with over activities since you are participating in discussions
not so frequently. So I am unsure at which moment it is appropriate to
raise such question that otherwise may just be buried in the list
archive.


I don't see how my presence (or not) on the list relates to this. If
there's an idea worth a discussion, it should not be buried within
a thread.


It was you who implemented org-cite and org-element parser. John chose a 
direct and practical way. He defined multiple link types and got the 
working solution. A more elegant approach would require some extension 
of syntax, so your opinion is really important. For cross-references it 
might be "[ref/style...]" in additional to "[cite/style...]" or 
attributes specific to links.



Outside of Org, citations are links.


But we're on an Org mailing list so…


In respect to citations and cross-references Org is a tool to prepare 
documents for "outside of Org".



I think link attributes were discussed a couple of times on this ML
already. Nothing was implemented tho.


I do not remember such thread during last couple of years. I will try to 
search deeper and maybe will start another thread.



I'm not convinced Org should generalize this to any inline object,
either, mainly because it sounds messy. Of course, if you have an
idea on the subject, please share it.

In any case, this is another topic, neither related to citations nor to
cross-references.


"pageref", "nameref", "eqref", "autoref" (see 
https://github.com/jkitchin/org-ref/blob/master/org-ref.org) may be 
values of the style attribute of a "[[name]]" link instead of link type 
as it is done in org-ref.



For citations some values may be passed to specific citation backend
overriding default value derived from style.


In that situation, you can define a new style specific to the citation
back-end.


Sometimes an ad-hoc adjustment at the particular place is more 
convenient than even file-local property.

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 21/03/2022 18:51, John Kitchin wrote:

Vikas Rawal writes:


 From the perspective of a user, this was only meant to express a
sentiment that one finds oneself in a situation of having to choose
between two good things, and that we have not been able to find a way
to make both compatible with each other. It was in not meant as a
disrespect in any way.


I don't think you have to choose. You can use org-cite for
citations, and org-ref for cross-references. The citation syntax is
orthogonal, you just should not mix them. You can even wire org-ref to
use org-cite-insert like this:

(setq org-ref-insert-cite-function (lambda () (org-cite-insert nil)))


I am glad to read this. John, could you, please, update the README file 
for https://github.com/jkitchin/org-ref to clarify that both packages 
may be used within the same document while org-ref is used solely for 
cross-references (I may create a github issue if your prefer)? From my 
point of view the following phrase may be considered as prohibitive in 
respect to combining the packages for any purpose:


You can use both org-cite and org-ref (although you should not do 
that in the same document as they are independent citation tools).

I feel some ambiguity in the starting message in this thread:

Vikas Rawal, Sun, 20 Mar 2022 17:38:30 +0530. 
https://list.orgmode.org/caltzab2bhuldoxaamuzfqh2h453ekb6k7bkohbu-dhpn98a...@mail.gmail.com



This obviously creates many problems including that two people using
different citation systems cannot share org files.
Accordingly to the org-ref README installed org-ref package does not 
break org-cite, so people are free to share documents. They just should 
have both packages configured and should adhere to a chosen package in 
each document (for *citations*).

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 22/03/2022 00:00, John Kitchin wrote:

"Bruce D'Arcus" writes:


Indeed, the question of how to better support cross-references in org
is an important one.

I don't really use them much, and so am still unsure if this could be
addressed with incremental improvements in existing org link support,
or if it would require more significant enhancements.


No incremental improvement is required IMO, regular links are
sufficient. org-ref has handled these with regular links from the
beginning. It even has an org-ref-refproc now for non-LaTeX exports that
has some support for things like sorting, grouping and cleveref. You can
find an example org file at
https://github.com/jkitchin/org-ref/blob/master/examples/refproc.org,


Interesting package. At first I thought you use keywords around links as 
implicit markup, but later I realized that this file is example of 
export, not the original source file (with different link types).



and see it in action at https://www.youtube.com/watch?v=rRR-5NSpKyE the
video has some rough spots, but you can get the idea.

You can support all kinds of things in these links (for example, I now
support pre/post note text in the cite link paths), really anything you
want to parse out of the path.

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

I think `fullcite' is OK, although it will be a bit verbose:

┌
│ [cite/fullcite:...]
└


Personally, I don’t mind using `full', and so having a duplicate between a style
and a variant.

But, to be honest, anything is fine with me, as long as it is readily available
and documented.

Thank you!
Dominik

“Bruce D’Arcus”  writes:

> On Wed, Mar 23, 2022 at 6:04 PM Nicolas Goaziou  
> wrote:
>>
>> Hello,
>>
>> “Bruce D’Arcus”  writes:
>>
>> > On Wed, Mar 23, 2022 at 5:27 PM Nicolas Goaziou  
>> > wrote:
>>
>> >> I can add it, but “full” is already the name of a variant, so
>> >> [cite/full: …] and [cite/style/full: …] would mean different things.
>> >> Is this a problem, or do you think of a better style name?
>> >
>> > FWIW, Nicolas, biblatex “fullcite” is equivalent to natbib/bibtex 
>> > “bibentry”.
>> >
>> > That might be a reasonable alternative style name?
>> >
>> >> Also, are there possible variants for this style?
>> >
>> > AFAIK, no.
>>
>> Hmm, OK. What about:
>>
>>   (“fullcite” nil “fullcite” nil nil)
>>
>> ?
>
> Seems fine by me, so long as you use the same name for natbib if and
> when you add bibentry support?
>
> Bruce

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

Max Nikulin  writes:

> Nicolas, concerning a new thread, I have an impression that you are
> busy with over activities since you are participating in discussions
> not so frequently. So I am unsure at which moment it is appropriate to
> raise such question that otherwise may just be buried in the list
> archive.

I don't see how my presence (or not) on the list relates to this. If
there's an idea worth a discussion, it should not be buried within
a thread.

> Outside of Org, citations are links.

But we're on an Org mailing list so…

> I consider fixed set of properties as a problem.

I don't think it is a problem for citations.

> At first I tried to draw attention to additional link attributes.

I think link attributes were discussed a couple of times on this ML
already. Nothing was implemented tho.

I'm not convinced Org should generalize this to any inline object,
either, mainly because it sounds messy. Of course, if you have an
idea on the subject, please share it.

In any case, this is another topic, neither related to citations nor to
cross-references.

> For citations some values may be passed to specific citation backend
> overriding default value derived from style.

In that situation, you can define a new style specific to the citation
back-end.

Regards,
-- 
Nicolas Goaziou

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Wed, Mar 23, 2022 at 1:19 PM Max Nikulin  wrote:

> My point was that it should not be unconsciously ignored. Since the
> message was long enough, this particular complain may remain unnoticed.
> I can not say that I fully agree with your decision, but I respect it. I
> had no intention to upset you and I beg your pardon if it happened.

No personal offense taken. I was just a bit frustrated with the
direction this discussion seemed to be going.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Wed, Mar 23, 2022 at 6:04 PM Nicolas Goaziou  wrote:
>
> Hello,
>
> "Bruce D'Arcus"  writes:
>
> > On Wed, Mar 23, 2022 at 5:27 PM Nicolas Goaziou  
> > wrote:
>
> >> I can add it, but "full" is already the name of a variant, so
> >> [cite/full: ...] and [cite/style/full: ...] would mean different things.
> >> Is this a problem, or do you think of a better style name?
> >
> > FWIW, Nicolas, biblatex "fullcite" is equivalent to natbib/bibtex 
> > "bibentry".
> >
> > That might be a reasonable alternative style name?
> >
> >> Also, are there possible variants for this style?
> >
> > AFAIK, no.
>
> Hmm, OK. What about:
>
>   (“fullcite” nil “fullcite” nil nil)
>
> ?

Seems fine by me, so long as you use the same name for natbib if and
when you add bibentry support?

Bruce

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus"  writes:

> On Wed, Mar 23, 2022 at 5:27 PM Nicolas Goaziou  
> wrote:

>> I can add it, but "full" is already the name of a variant, so
>> [cite/full: ...] and [cite/style/full: ...] would mean different things.
>> Is this a problem, or do you think of a better style name?
>
> FWIW, Nicolas, biblatex "fullcite" is equivalent to natbib/bibtex "bibentry".
>
> That might be a reasonable alternative style name?
>
>> Also, are there possible variants for this style?
>
> AFAIK, no.

Hmm, OK. What about:

  (“fullcite” nil “fullcite” nil nil)

?

Regards,
-- 
Nicolas Goaziou

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Wed, Mar 23, 2022 at 5:27 PM Nicolas Goaziou  wrote:
>
> Hello,
>
> Dominik Schrempf  writes:
>
> > (add-to-list ’org-cite-biblatex-styles ’(“full” nil “fullcite” nil nil))
> >
> > (This or something similar should be added upstream).
>
> I can add it, but "full" is already the name of a variant, so
> [cite/full: ...] and [cite/style/full: ...] would mean different things.
> Is this a problem, or do you think of a better style name?

FWIW, Nicolas, biblatex "fullcite" is equivalent to natbib/bibtex "bibentry".

That might be a reasonable alternative style name?

> Also, are there possible variants for this style?

AFAIK, no.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

Dominik Schrempf  writes:

> (add-to-list ’org-cite-biblatex-styles ’(“full” nil “fullcite” nil nil))
>
> (This or something similar should be added upstream).

I can add it, but "full" is already the name of a variant, so
[cite/full: ...] and [cite/style/full: ...] would mean different things.
Is this a problem, or do you think of a better style name? Also, are
there possible variants for this style?

Regards,
-- 
Nicolas Goaziou

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

John Kitchin  writes:

> I do not think it is productive for the community to say or consider it
> is a sad situation. Many good things have emerged from these
> discussions, even if it is not yet consensus on a solution. It is a
> complex problem, with many years of effort by many people on each side.
> That is an indication of how ambitious this project is and that there
> may be more than one solution that is needed.

[...]

> There are more than 8 years of legacy org-ref documents. I have written
> 40+ scientific papers with it, and countless technical documents with
> more than 8000 cite links among them. org-ref has exceeded 190K
> downloads from MELPA, so I feel obligated to maintain org-ref for
> myself, and those users. org-ref may be heavyweight in bundling a lot of
> capability together that could be separated into individual packages,
> but it is also convenient for people who need it, and a GitHUB issue or
> pull request away from new features. I remain committed to supporting
> this, and I do it in a way I can manage, hence the monolithic package
> design.

I think there's a misunderstanding here. Org Cite was never meant as
a replacement for Org Ref. It was designed from the beginning as
a framework other libraries could rely on and extend in their own way.
Org Ref could have been one of them.

It looks like a social problem to me. I remember well asking for
feedback when implementing the various prototypes, i.e., ways to make
Org Cite more useful to other libraries. I don't think I got much of it,
barring the cross-references topic, which is not solved. Maybe I was not
explicit enough about what I was expecting. For example, this is—please
correct me if I'm wrong—the first time I read about the "BibLaTeX is not
fully implemented in Org Cite" and "Org Cite is adding an extra
abstraction layer on top of BibLaTeX commands" issues, which are both
trivial to solve, either on the Org Cite or the Org Ref side. But then
again, it is perfectly fine if Org Cite doesn't provide that, as some
libraries can extend it if needed.

On the other hand, there have been much activity on GitHub repositories,
i.e., out of this mailing list. It seems to me Org Ref project has been
trying to work around possible blockers in Org Cite project the whole
time without the latter knowing about them, and having an opportunity to
lift them.

> Both projects have benefited from this discussion a lot. org has
> org-cite now, and org-ref now handles pre/post notes like org-cite does,
> it supports CSL much better, and is even a little more modular, lighter
> weight, and more easily integrated with other completion backends than
> ivy or helm. That should broadly be viewed as a win-win situation.

But it is not. There are now two, more or less official, citations
syntax. Interoperability is the big loss. Features do not count; it is
perfectly fine to have different packages targeting different needs, as
long as they share the same syntax.

Hopefully, at some point, we'll be able to build a list of blockers that
prevent Org Ref being built on top of Org Cite, and proceed.

Regards,
-- 
Nicolas Goaziou

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 23/03/2022 21:39, Bruce D'Arcus wrote:


Finding and analyzing existing papers again raises the question of
which ones; citation practices look VERY different in chemistry than
in art history or sociology. It also raises the question of who will
do this work, and whether it's the most efficient use of their time.
And finally, your suggestion seems to assume we didn't put a lot of
research and thought into the existing mappings.


I think that a working solution (Eric just have confirmed it) is a great 
result. I am aware that even in different areas of physics citation 
traditions vary. I am not trying to dispute that you invest a lot of 
efforts. I just consider as normal when some polishing is required after 
the initial release. Really exhaustive research is usually prohibitively 
expensive. I had a hope that it was clear from the beginning that I 
considered a missed style as a minor issue.


My point was that it should not be unconsciously ignored. Since the 
message was long enough, this particular complain may remain unnoticed. 
I can not say that I fully agree with your decision, but I respect it. I 
had no intention to upset you and I beg your pardon if it happened.



Have you actually looked at the table of existing mappings? See table
1 here (which it seems we might want to add to the manual?):

https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-syntax


I have seen this post and I have even checked the org-cite code that the 
example from John's message is not covered by the existing mapping.

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 23/03/2022 06:52, Nicolas Goaziou wrote:

Max Nikulin writes:


Another point is more serious. Besides citations there are internal
cross-references. Org supports them but only in a rudimentary form.
Actually cross-references are similar to citations in the sense that
they can have style, prefixes and suffixes, and their appearance
depends on target properties. Another feature is grouping. However
cross-references should not be handled by citation backends, they
require different handlers. Unfortunately there is no way to define
custom "citation" type e.g. "[ref:...]" in addition to "[cite:...]".


There is too little information here for me to understand what
cross-references (or grouping) are, or why they would require different
handlers. In any case, if such thing are deemed necessary in Org Cite,
one can always start a new thread.


Frankly speaking, for me it was enough just to use available LaTeX 
commands. I never thought about corner cases, but feature request 
requires such analysis.


By grouping I mean the existing feature of org-cite: several keys with 
common prefix and suffix, items within such group may be sorted.


Citation handlers obtain information necessary for formatting from a 
bibliography database. For cross-references such additional info should 
be pulled from other parts of the same document. There is no point to 
mix the code for both cases withing the same handler, it is better to 
combine independent handlers and to allow each type to have alternative 
implementations. I agree with those who say that cross-references and 
citations are rather independent when implementation is considered.


Judging from the description, org-ref (with org-refproc as an optional 
addition) does the job taking advantage of multiple custom link types 
for the same actual standard internal link target.


Sphinx has a feature that is an interesting border case between internal 
cross-references and external links (almost citations). Readthedocs 
sites host mapping tables for python packages. It is possible to fetch a 
file that associates e.g. anchors and section names. In generated docs 
anchors (similar to custom_id values) are replaced by section names and 
link target points to particular location in the external file.


Nicolas, concerning a new thread, I have an impression that you are busy 
with over activities since you are participating in discussions not so 
frequently. So I am unsure at which moment it is appropriate to raise 
such question that otherwise may just be buried in the list archive. 
Sometimes I am in doubts if an idea deserves a dedicated thread or 
initial feedback may be received from a related rather generic topic.



To assign additional properties, info "(org) Links in HTML export"
https://orgmode.org/manual/Links-in-HTML-export.html recommends usage
of "#+ATTR_HTML", but such technique has several issues:


This is a different issue. Citations are not link, and have a fixed set
of properties.


Outside of Org, citations are links. Along with cross-references they 
worked before appearance of the web. To be recognizable on paper they 
may have a bit special form. An author may choose what will appear in 
the text: page number, section number, or section text. For HTML links 
page number option is not meaningful.


Within Org citations look like generalized links: a citation may have 
multiple targets, they have more properties: prefixes, suffixes 
(including common ones), style, and locators. However there is no 
description.


Internal links in Org (built-in ones without additional packages) are 
more limited than full-fledged cross-references. When exported they have 
the same form.


I consider fixed set of properties as a problem. At first I tried to 
draw attention to additional link attributes. Then I realized that 
block-level elements may have arbitrary attributes. It would be a great 
feature to have some syntax construct to assign attributes for 
particular inline object. People actively use link types as an 
additional property, but it gives just one degree of freedom. Sometimes 
more parameters are required and abusing link types means combinatorics 
explosion. Encoding extra properties inside link targets sometimes is 
enough (as in org-ref) but some times it is inconvenient.


Advantages of arbitrary attributes for inline objects for links:
- Within the same paragraph only part of links may be marked as 
"nofollow noopener" during exporting to HTML.

- Each link may have its own title.
- Link types similar to org-ref cross-references ("pageref", "nameref", 
"eqref", etc.) becomes an extra attribute while link type and link 
target remains the standard one for Org (heading text, custom id, name 
attribute).


For citations some values may be passed to specific citation backend 
overriding default value derived from style. It should be similar to 
babel header arguments specific to particular language or export options 
for particular format.


Leaving 

Re: citations: org-cite vs org-ref 3.0

[Eric S Fraga]

On Wednesday, 23 Mar 2022 at 10:39, Bruce D'Arcus wrote:
> Those mappings merely generalize existing systems (bibtex, natbib,
> biblatex, csl) used by millions of users (if you include Zotero,
> etc.), and already incorporate the feedback of those users.

I just want to jump in to say that the new cite functionality works very
well, at least for this LaTeX user.  I'm writing a book at the moment
using [cite:...].  No issues encountered and everything I need to do is
there, out-of-the-box!

My thanks to all that put in so much effort into this.

-- 
: Eric S Fraga, with org release_9.5.2-407-gc9c0b9 in Emacs 29.0.50

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Wed, Mar 23, 2022 at 8:52 AM Max Nikulin  wrote:
>
> On 23/03/2022 00:20, Bruce D'Arcus wrote:
> > On Tue, Mar 22, 2022 at 10:42 AM Max Nikulin wrote:
> >>
> >> John Kitchin, this thread, Sun, 20 Mar 2022 20:31:29 -0400.
> >> https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu:
> >>
> >>> I don't know the equivalent of \citenum in CSL.
> >
> > Right; so John or someone else should send a message to the list
> > requesting it specifically?
>
>  From my point of view he has already requested support of \citenum by
> that message.

I just mean here, in general. If people complain about missing
mappings or performance issues silently, or elsewhere, without ever
raising them on the list, then it's hard to address them.

> I have an impression that the ball is on the side of the org-cite, and
> next steps may be to ask for real documents (e.g. open access papers)
> that are prepared with such format and to discuss most suitable style
> for CSL.

Suffice to say I disagree :-)

Sorry if the below gets wordy, but it's complicated.

Finding and analyzing existing papers again raises the question of
which ones; citation practices look VERY different in chemistry than
in art history or sociology. It also raises the question of who will
do this work, and whether it's the most efficient use of their time.
And finally, your suggestion seems to assume we didn't put a lot of
research and thought into the existing mappings.

Have you actually looked at the table of existing mappings? See table
1 here (which it seems we might want to add to the manual?):

https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-syntax

Those mappings merely generalize existing systems (bibtex, natbib,
biblatex, csl) used by millions of users (if you include Zotero,
etc.), and already incorporate the feedback of those users.

So the styles included now ARE a sound starting point, that we can
iterate based on feedback.

We can and should add "number" and "entry" (for "bibentry" and
biblatex "fullcite") styles to those mappings, however, at least for
natbib and biblatex. But the first is one of those lower-level types
of commands, and probably why it's not there now.

On that last point, I do want people to understand that there are
places where you can't easily generalize across those systems, because
the logic of them varies in places. Biblatex, most notably, has IIRC
more than 50 commands, which also vary by style used. And some of
those commands (like autocite) are high-level and appropriate for this
sort of mapping, and others (like footcite) are low-level, and
probably not. Adding every option may make a small number of power
users happy, but at the expense of raising complexity for others.
Which is why the new defcustom is a good compromise.

CSL is different here than the LaTeX alternatives, as Andras can just
add support for some feature to citeproc-el, and add a style for it to
oc-csl. But CSL also has a different design than the LaTeX options,
and so will always be simpler. For example, in effect, all citations
in a CSL systems work like biblatex's "auto" commands. If one uses a
note-based citation style, citations get automatically footnoted, for
example, and so one can seamlessly change between note-based and other
kinds of styles, without having to modify the document source.

So missing mappings aren't necessarily an oversight; it might just be
that how to implement them was unclear, or whether users would need or
want them.



Bruce

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 23/03/2022 00:20, Bruce D'Arcus wrote:

On Tue, Mar 22, 2022 at 10:42 AM Max Nikulin wrote:


John Kitchin, this thread, Sun, 20 Mar 2022 20:31:29 -0400.
https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu:


I don't know the equivalent of \citenum in CSL.


Right; so John or someone else should send a message to the list
requesting it specifically?


From my point of view he has already requested support of \citenum by 
that message. I am taking into account that he is the developer org-ref 
and he is happy with this package. John made a lot to adopt org-cite 
syntax to ensure feature parity. Bruce, you should know it better than 
me since you participated in discussions while I am merely and observer. 
John does not really need \citenum namely in org-cite since his tool is 
org-ref, but his words may affect perception of org-cite.


I have an impression that the ball is on the side of the org-cite, and 
next steps may be to ask for real documents (e.g. open access papers) 
that are prepared with such format and to discuss most suitable style 
for CSL.



One possible idea to consider is to allow two systems in each of
LaTeX-oriented processors: what we might call a default "org-cite"
one, and an optional "literal" one. So if you only use oc-natbib, and
you want the natbib commands directly, you might change a variable to
get that.


Maybe I will share my idea in response to the message from Nicolas
https://list.orgmode.org/871qytlf49@nicolasgoaziou.fr
Wed, 23 Mar 2022 00:52:22 +0100, this thread.

Since John explicitly confirmed that org-ref for cross-references may be 
combined with org-cite for citations, it may be great that Org has 2 
independent implementations for citations. When a user meets some corner 
case their has an opportunity to try another package.

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

Max Nikulin  writes:

> Nicolas, I think, a part of problem is that you are not an org-cite
> user. The packages require some *polishing*, but it have to be 
> *user-driven*.

Just to be clear: the development was _user-driven_ from the start.
I read years of discussions about citation, suggested a syntax, waited
for input, amended the syntax… Then I did the same for the whole
library, implementing proof of concepts, waiting for feedback, etc.

> Another point is more serious. Besides citations there are internal
> cross-references. Org supports them but only in a rudimentary form. 
> Actually cross-references are similar to citations in the sense that
> they can have style, prefixes and suffixes, and their appearance
> depends on target properties. Another feature is grouping. However 
> cross-references should not be handled by citation backends, they
> require different handlers. Unfortunately there is no way to define 
> custom "citation" type e.g. "[ref:...]" in addition to "[cite:...]".

There is too little information here for me to understand what
cross-references (or grouping) are, or why they would require different
handlers. In any case, if such thing are deemed necessary in Org Cite,
one can always start a new thread.

The goal of Org Cite is not to do everything, however. Its raison d'être
is to provide building blocks around citations that external packages
can use and extend. Therefore cross-references might belong to an
extension instead.

> To assign additional properties, info "(org) Links in HTML export"
> https://orgmode.org/manual/Links-in-HTML-export.html recommends usage
> of "#+ATTR_HTML", but such technique has several issues:

This is a different issue. Citations are not link, and have a fixed set
of properties.

Regards,
-- 
Nicolas Goaziou

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Tue, Mar 22, 2022 at 10:42 AM Max Nikulin  wrote:
>
> On 21/03/2022 22:19, Bruce D'Arcus wrote:
> > On Mon, Mar 21, 2022 at 10:41 AM Max Nikulin wrote:
> >
> >> A bit of routine work will alleviate some user issues:
> >> - add missed styles
> >
> > The initial list of style-command mappings was pretty comprehensive,
> > but we left out some of the more obscure biblatex commands because
> > unsure if they were needed, or how best to add them (conceptually
> > there's a mix of different kinds of commands in biblatex, which are
> > hard to fit into a more general style system, for example).
> >
> > Since then:
> >
> > - people have occasionally asked to add new mappings, and Nicolas has added 
> > them
> > - he's also added the styles defcustoms for biblatex, so users can do
> > this themselves
> >
> > In short, I think we're good on this actually.
>
> John Kitchin, this thread, Sun, 20 Mar 2022 20:31:29 -0400.
> https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu:
>
> > I don't know the equivalent of \citenum in CSL.

Right; so John or someone else should send a message to the list
requesting it specifically?

That's how we've done it so far anyway. Not sure what the practical
alternative is.

I will say that much of this discussion is not about org-cite per se,
but rather implementation decisions in the specific bundled
processors. I fear these conversations get hard to track if we aren't
precise about what we're talking about; citation syntax, included
processor functionality or style mappings, documentation, etc.

>  From my point of view it may be a reason to add a new style to
> defaults. It is important whether a tool works out of the box.

But the question here becomes "works for whom?" Citations practices
vary a fair bit by field.

Primarily LaTeX/natbib users like John and other science folks,
biblatex users primarily coming from the humanities as near as I can
tell? Or people coming from markdown and pandoc (which has an
excellent citation system that clearly inspired org-cite) and needing
Word-ready documents for journals etc?

The style system in the bundled processors is designed to work for all
of them, and not privilege one or another. It does this by creating a
new, more general, org-cite style and variant system, which then maps
to different targets.

That approach has obvious advantages: citations are (mostly) decoupled
from particular export processors. One can write a document with these
styles, and get PDF citation and bibliographic output via biblatex,
for example, similar to the HTML or ODT output from the CSL export
processor.

And that also has advantages beyond org. For example, pandoc recently
added support for org-cite, and that includes mapping between the two
style/command systems. So in theory you could convert an org file to
docx, letting pandoc process the citations along the way, and the
result would more than acceptable.

> Custom variables make a document less portable unless they are specified as
> file-local ones.

Indeed!

Or more specifically here, it ties the citations to the export processor.

But maybe for some or many people that's perfectly fine.

One possible idea to consider is to allow two systems in each of
LaTeX-oriented processors: what we might call a default "org-cite"
one, and an optional "literal" one. So if you only use oc-natbib, and
you want the natbib commands directly, you might change a variable to
get that.

I really don't know if that's a good idea or not, but I raise it to
emphasize there's a lot of flexibility with the org-cite design, and
so it's just a question of how we make use of it.

> I think, the goal may be formulated as "John can not
> say the following any more" (at least in respect to citations leaving
> aside cross-references):
>
> > I simply cannot
> > compromise on the capability org-ref provides me, or wait for an
> > alternative complete solution in org-mode.

I don't really see why these are the two choices.

> On the other hand I do not consider the following argument as a strong one
>
> > I do not like the abstraction away from LaTeX cite commands in org-cite.
> > This is an example of a compromise between LaTeX and CSL.
>
> despite I believe that convenience and habits are important. Mapping of
> styles to commands is just a piece of knowledge.

Yes.

> I have no particular opinion if enough efforts should be invested from
> both sides to allow mixing on both citation syntax constructs (org-cite
> and org-ref) in the same document. Bruce, you made a lot for support of
> CSL in org-cite, so I will stressed another direction of feature
> comparison since Bib(La)TeX users should feel themselves first-class
> citizens.

Absolutely! In fact, Denis Meier, who has extensive biblatex
knowledge, helped a lot with the initial mapping there, and some other
biblatex users have contributed feedback along the way.


Bruce

> The choice between org-ref and org-cite, when the former can do its job,
> should be matter of taste and 

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

On 21/03/2022 22:19, Bruce D'Arcus wrote:

On Mon, Mar 21, 2022 at 10:41 AM Max Nikulin wrote:


A bit of routine work will alleviate some user issues:
- add missed styles


The initial list of style-command mappings was pretty comprehensive,
but we left out some of the more obscure biblatex commands because
unsure if they were needed, or how best to add them (conceptually
there's a mix of different kinds of commands in biblatex, which are
hard to fit into a more general style system, for example).

Since then:

- people have occasionally asked to add new mappings, and Nicolas has added them
- he's also added the styles defcustoms for biblatex, so users can do
this themselves

In short, I think we're good on this actually.


John Kitchin, this thread, Sun, 20 Mar 2022 20:31:29 -0400. 
https://list.orgmode.org/m2sfrc149c@andrew.cmu.edu:



I don't know the equivalent of \citenum in CSL.


From my point of view it may be a reason to add a new style to 
defaults. It is important whether a tool works out of the box. Custom 
variables make a document less portable unless they are specified as 
file-local ones. I think, the goal may be formulated as "John can not 
say the following any more" (at least in respect to citations leaving 
aside cross-references):



I simply cannot
compromise on the capability org-ref provides me, or wait for an
alternative complete solution in org-mode.


On the other hand I do not consider the following argument as a strong one


I do not like the abstraction away from LaTeX cite commands in org-cite.
This is an example of a compromise between LaTeX and CSL.


despite I believe that convenience and habits are important. Mapping of 
styles to commands is just a piece of knowledge.


I have no particular opinion if enough efforts should be invested from 
both sides to allow mixing on both citation syntax constructs (org-cite 
and org-ref) in the same document. Bruce, you made a lot for support of 
CSL in org-cite, so I will stressed another direction of feature 
comparison since Bib(La)TeX users should feel themselves first-class 
citizens.


The choice between org-ref and org-cite, when the former can do its job, 
should be matter of taste and personal preferences unrelated to 
technical limitations.

Re: citations: org-cite vs org-ref 3.0

[indieterminacy]

Hi Juh,

juh  writes:

> [[PGP Signed Part:Undecided]]
> Am Sun, Mar 20, 2022 at 08:31:29PM -0400 schrieb John Kitchin:
>> For those who need high fidelity LaTeX export like I do, I think
>> org-ref is still a superior solution. For everyone else, and
>> especially if you do not need sophisticated cross-references and don't
>> want the dependencies of org-ref, org-cite is likely the better
>> solution. 
>
> Thank you very much for the clarification. 
>
> I am new to both and I need something that works together with ConTeXt
> export. Currently I am happy with org-cite. I used org-ref before but
> never got good results in ConTeXt export. So I think that I will stick
> wit org-cite. 
>

I used to use ConTeXt and have a healthy respect for it.
That feedback is good to hear.

> Org-mode is not LaTeX but I wonder if it can be a better replacement for
> my Markdown-->Pandoc-->EPUB/ConTeXt single source publication process.
> Better as I could make more use of org-roam in my writing process.
>

May I recommend the mininimal Markdown format, GemText?
It has a very simple syntax which is easily convertable into Orgmode
syntax.

It could be useful for projects where the flow starts with simple
written prose.

> Though I am a long time emacs user putting all these blocks together is
> a challenge, still.
>
> With your words I now see much much clearer.
>
> juh


Jonathan

Re: citations: org-cite vs org-ref 3.0

[Vikas Rawal]

>
>
>
> I don't think you have to choose. You can use org-cite for
> citations, and org-ref for cross-references. The citation syntax is
> orthogonal, you just should not mix them.


That is excellent then.

Thanks for clarifications.

Best,

Vikas

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

"Bruce D'Arcus"  writes:

> On Mon, Mar 21, 2022 at 10:41 AM Max Nikulin  wrote:
>
> ...
>
>> A bit of routine work will alleviate some user issues:
>> - add missed styles
>
> The initial list of style-command mappings was pretty comprehensive,
> but we left out some of the more obscure biblatex commands because
> unsure if they were needed, or how best to add them (conceptually
> there's a mix of different kinds of commands in biblatex, which are
> hard to fit into a more general style system, for example).
>
> Since then:
>
> - people have occasionally asked to add new mappings, and Nicolas has added 
> them
> - he's also added the styles defcustoms for biblatex, so users can do
> this themselves
>
> In short, I think we're good on this actually.
>
>> - improve documentation, e.g. to make backend choice more conscious.
>
> This is the bigger user-facing issue that could use attention.
>
>> Another point is more serious. Besides citations there are internal
>> cross-references. Org supports them but only in a rudimentary form.
>
> Indeed, the question of how to better support cross-references in org
> is an important one.
>
> I don't really use them much, and so am still unsure if this could be
> addressed with incremental improvements in existing org link support,
> or if it would require more significant enhancements.

No incremental improvement is required IMO, regular links are
sufficient. org-ref has handled these with regular links from the
beginning. It even has an org-ref-refproc now for non-LaTeX exports that
has some support for things like sorting, grouping and cleveref. You can
find an example org file at
https://github.com/jkitchin/org-ref/blob/master/examples/refproc.org,
and see it in action at https://www.youtube.com/watch?v=rRR-5NSpKyE the
video has some rough spots, but you can get the idea.

You can support all kinds of things in these links (for example, I now
support pre/post note text in the cite link paths), really anything you
want to parse out of the path. 


>
> Perhaps we need the community to itemize what the gaps and limitations
> are there?
>
>
> Bruce
>
>> Actually cross-references are similar to citations in the sense that
>> they can have style, prefixes and suffixes, and their appearance depends
>> on target properties. Another feature is grouping. However
>> cross-references should not be handled by citation backends, they
>> require different handlers. Unfortunately there is no way to define
>> custom "citation" type e.g. "[ref:...]" in addition to "[cite:...]".
>>
>> I can not judge if uniform UI issues are really severe and if it would
>> be convenient if depending on prefix argument either org-cite or org-ref
>> command would be called for a citation or for a reference.
>>
>> Actually "[cite:...]" construct is a kind of link with additional
>> flexibility missed for regular links. Anything besides target and
>> description requires some workarounds. Usual approach is proliferation
>> of link types. E.g. inline source blocks allows almost arbitrary extra
>> parameters. Citation syntax is rather domain specific, it allows more
>> than regular links, but for convenience the set of properties is fixed:
>> style, prefixes, locators, suffixes. It is impossible to add extra one.
>>
>> To assign additional properties, info "(org) Links in HTML export"
>> https://orgmode.org/manual/Links-in-HTML-export.html recommends usage of
>> "#+ATTR_HTML", but such technique has several issues:
>> - attributes becomes specific to the export backend
>> - the same attributes are added to the enclosing paragraph
>>https://linevi.ch/en/org-link-extra-attrs.html
>> - a paragraph may have more than one link.
>> It is possible to use link target similar to form values encoded into
>> URI, but it hardly can be considered as convenient for editing.
>>
>> Custom citation types may alleviate the issue with cross-references. It
>> would be great to have more flexible links with arbitrary properties
>> (and it would allow to consider citations and cross-references as
>> special cases of links), but it does not fit into the Org syntax.
>>
>> P.S. John has a valid complain but it hardly relates to the "cite vs.
>> cross-reference" topic. When some package is not loaded and link type is
>> undefined then the link becomes a fuzzy one leading to user confusion.
>>
>>


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[juh]

Am Sun, Mar 20, 2022 at 08:31:29PM -0400 schrieb John Kitchin:
> For those who need high fidelity LaTeX export like I do, I think
> org-ref is still a superior solution. For everyone else, and
> especially if you do not need sophisticated cross-references and don't
> want the dependencies of org-ref, org-cite is likely the better
> solution. 

Thank you very much for the clarification. 

I am new to both and I need something that works together with ConTeXt
export. Currently I am happy with org-cite. I used org-ref before but
never got good results in ConTeXt export. So I think that I will stick
wit org-cite. 

Org-mode is not LaTeX but I wonder if it can be a better replacement for
my Markdown-->Pandoc-->EPUB/ConTeXt single source publication process.
Better as I could make more use of org-roam in my writing process.

Though I am a long time emacs user putting all these blocks together is
a challenge, still.

With your words I now see much much clearer.

juh

-- 
Autoren-Homepage: . http://literatur.hasecke.com
Satiren & Essays: . http://www.sudelbuch.de
Privater Blog:  http://www.hasecke.eu
Netzliteratur-Projekt:  http://www.generationenprojekt.de




signature.asc
Description: PGP signature

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 10:41 AM Max Nikulin  wrote:

...

> A bit of routine work will alleviate some user issues:
> - add missed styles

The initial list of style-command mappings was pretty comprehensive,
but we left out some of the more obscure biblatex commands because
unsure if they were needed, or how best to add them (conceptually
there's a mix of different kinds of commands in biblatex, which are
hard to fit into a more general style system, for example).

Since then:

- people have occasionally asked to add new mappings, and Nicolas has added them
- he's also added the styles defcustoms for biblatex, so users can do
this themselves

In short, I think we're good on this actually.

> - improve documentation, e.g. to make backend choice more conscious.

This is the bigger user-facing issue that could use attention.

> Another point is more serious. Besides citations there are internal
> cross-references. Org supports them but only in a rudimentary form.

Indeed, the question of how to better support cross-references in org
is an important one.

I don't really use them much, and so am still unsure if this could be
addressed with incremental improvements in existing org link support,
or if it would require more significant enhancements.

Perhaps we need the community to itemize what the gaps and limitations
are there?


Bruce

> Actually cross-references are similar to citations in the sense that
> they can have style, prefixes and suffixes, and their appearance depends
> on target properties. Another feature is grouping. However
> cross-references should not be handled by citation backends, they
> require different handlers. Unfortunately there is no way to define
> custom "citation" type e.g. "[ref:...]" in addition to "[cite:...]".
>
> I can not judge if uniform UI issues are really severe and if it would
> be convenient if depending on prefix argument either org-cite or org-ref
> command would be called for a citation or for a reference.
>
> Actually "[cite:...]" construct is a kind of link with additional
> flexibility missed for regular links. Anything besides target and
> description requires some workarounds. Usual approach is proliferation
> of link types. E.g. inline source blocks allows almost arbitrary extra
> parameters. Citation syntax is rather domain specific, it allows more
> than regular links, but for convenience the set of properties is fixed:
> style, prefixes, locators, suffixes. It is impossible to add extra one.
>
> To assign additional properties, info "(org) Links in HTML export"
> https://orgmode.org/manual/Links-in-HTML-export.html recommends usage of
> "#+ATTR_HTML", but such technique has several issues:
> - attributes becomes specific to the export backend
> - the same attributes are added to the enclosing paragraph
>https://linevi.ch/en/org-link-extra-attrs.html
> - a paragraph may have more than one link.
> It is possible to use link target similar to form values encoded into
> URI, but it hardly can be considered as convenient for editing.
>
> Custom citation types may alleviate the issue with cross-references. It
> would be great to have more flexible links with arbitrary properties
> (and it would allow to consider citations and cross-references as
> special cases of links), but it does not fit into the Org syntax.
>
> P.S. John has a valid complain but it hardly relates to the "cite vs.
> cross-reference" topic. When some package is not loaded and link type is
> undefined then the link becomes a fuzzy one leading to user confusion.
>
>

Re: citations: org-cite vs org-ref 3.0

[Max Nikulin]

Disclaimer: I am neither org-cite no org-ref user. In the past I used 
LaTeX and BibTeX directly though, and it is a reason why I am reading 
the discussions.


On 20/03/2022 20:19, Nicolas Goaziou wrote:

Vikas Rawal writes:


This obviously creates many problems including that two people using
different citation systems cannot share org files.


Indeed.


What is the general view of the community about this?


I don't know about the general view of the community, but, as a data
point, I find it very sad.


Nicolas, I think, a part of problem is that you are not an org-cite 
user. The packages require some *polishing*, but it have to be 
*user-driven*. Even if the feature were perfect, it would face some 
tension from people who already have a working solution.


A bit of routine work will alleviate some user issues:
- add missed styles
- improve documentation, e.g. to make backend choice more conscious.

Another point is more serious. Besides citations there are internal 
cross-references. Org supports them but only in a rudimentary form. 
Actually cross-references are similar to citations in the sense that 
they can have style, prefixes and suffixes, and their appearance depends 
on target properties. Another feature is grouping. However 
cross-references should not be handled by citation backends, they 
require different handlers. Unfortunately there is no way to define 
custom "citation" type e.g. "[ref:...]" in addition to "[cite:...]".


I can not judge if uniform UI issues are really severe and if it would 
be convenient if depending on prefix argument either org-cite or org-ref 
command would be called for a citation or for a reference.


Actually "[cite:...]" construct is a kind of link with additional 
flexibility missed for regular links. Anything besides target and 
description requires some workarounds. Usual approach is proliferation 
of link types. E.g. inline source blocks allows almost arbitrary extra 
parameters. Citation syntax is rather domain specific, it allows more 
than regular links, but for convenience the set of properties is fixed: 
style, prefixes, locators, suffixes. It is impossible to add extra one.


To assign additional properties, info "(org) Links in HTML export" 
https://orgmode.org/manual/Links-in-HTML-export.html recommends usage of 
"#+ATTR_HTML", but such technique has several issues:

- attributes becomes specific to the export backend
- the same attributes are added to the enclosing paragraph
  https://linevi.ch/en/org-link-extra-attrs.html
- a paragraph may have more than one link.
It is possible to use link target similar to form values encoded into 
URI, but it hardly can be considered as convenient for editing.


Custom citation types may alleviate the issue with cross-references. It 
would be great to have more flexible links with arbitrary properties 
(and it would allow to consider citations and cross-references as 
special cases of links), but it does not fit into the Org syntax.


P.S. John has a valid complain but it hardly relates to the "cite vs. 
cross-reference" topic. When some package is not loaded and link type is 
undefined then the link becomes a fuzzy one leading to user confusion.

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

chris  writes:

> On Sunday, 20 March 2022 20:44:50 CET Bruce D'Arcus wrote:
>> On Sun, Mar 20, 2022 at 9:42 AM Ihor Radchenko  wrote:
>> > For citar, why not simply using ivy-bibtex? It supports org-cite, AFAIK.
>> 
>> Not really; or rather minimally.
>
> I use `org-cite` with a very minimal configuration, and it works very well. I 
> don't use `ivy` at all. I don't use `helm` at all. Those are very large 
> framework and should not be forced to people. (I don't use `org-ref`.)

These days org-ref does not require helm or ivy, and it does not install
them. You can use the core library with any completion backend you want.
It uses vanilla completing-read.

org-ref does rely exclusively on bibtex-completion at this time. I have
not abstracted that out to allow pluggable alternatives like citar.

>
> I only use `citar` with minimal configuration. I use `vertico` for the 
> completion. `citar` is simple enough for me to be able to read and understand 
> a large part of it.
>
> IMO the more layers of code there are, the more difficult it is to have 
> things 
> work right. And similarly with the size of the code.
>
> Chris
>
>> 
>> Ivy-bibtex supports, for example, inserting of org-cite citations, but
>> not via org-cite-insert.
>
> And I have `org-cite-insert` working straight out of the box.
>
>> 
>> So there are currently no org-cite processors for ivy-bibtex etc.
>> 
>> Bruce


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

"Bruce D'Arcus"  writes:

> On Sun, Mar 20, 2022 at 9:42 AM Ihor Radchenko  wrote:
>
>> For citar, why not simply using ivy-bibtex? It supports org-cite, AFAIK.
>
> Not really; or rather minimally.
>
> Ivy-bibtex supports, for example, inserting of org-cite citations, but
> not via org-cite-insert.
>
> So there are currently no org-cite processors for ivy-bibtex etc.

I think this one in
https://github.com/jkitchin/org-ref-cite/blob/main/org-ref-cite-insert.el

provides an insert processor based on bibtex-completion, and that
provides ivy-bibtex actions. The commentary in this file also discusses some
challenges in relying exclusively on completing-read. My opinion of
course.

I don't think you need a special processor for ivy-bibtex. It could be
wired to insert org-cites on its own.

>
> Bruce


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Bruce and I looked into this UI approach in
https://github.com/jkitchin/org-ref-cite/issues/9. Bruce and I discussed
and worked on this for almost two weeks. There are 70 comments in this
issue.

There are opportunities now to annotate completion targets, which you can
see in the link above. The annotations are not selectable though during
completion, and this implementation was not too fast as I recall. You
probably should also augment the tooltips like this because you have to be
able to tell what a citation format is in the future too, not just at
insert time, e.g. suppose you are reading work of a collaborator. It was
hard to write, and ambiguous in ways, e.g. what is the export backend you
want? The annotations should differ for LaTeX (where you want to see the
latex command) vs HTML (where you probably want to see the formatted CSL
cite)...

We did not surmount these challenges at the time. Maybe others will succeed
in this another day.

John

---
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Mon, Mar 21, 2022 at 8:42 AM Bruce D'Arcus  wrote:

> On Mon, Mar 21, 2022 at 8:23 AM John Kitchin 
> wrote:
>
> >> A package could be created, say `org-cite-literal-biblatex' which is
> just a copy
> >> of `oc-biblatex.el' with a different default `org-cite-biblatex-styles'
> and
> >> `org-cite-biblatex-style-shortcuts' (or just sets those variables in
> >> `org-cite-biblatex'). As far as I can tell this would provide exactly
> the
> >> functionality you say org-cite can’t provide but org-ref does.
> >
> >
> > I wrote this package you suggest in org-ref-cite. In discussions during
> that development, it was clear the preference was on the more abstracted,
> and uniform syntax across backends cite commands in org-cite, and not this
> kind of variant. Of course one can do this. It is not that org-cite can't
> provide it, it is that it doesn't at this time.
>
> Just for some broader context on this particular issue.
>
> The advantage of the org-cite style/variant design reflected in the
> included export processors (natbib, biblatex, csl) is that the same
> styles will mostly generate the same final output.
>
> But that portability will only work with those styles and variants.
>
> With the new org-cite-biblatex-styles defcustom, however, one can
> augment or completely replace all those. But if you care about that
> portability, you'd want to be aware of this, and think about it.
>
> So per Timothy's point, you actually don't even need a new processor
> for biblatex if you want to include all the extensive list of biblatex
> commands.
>
> Natbib AFAIK is already fully covered.
>
> There's another POV on this though:
>
> If one doesn't like to see the org-cite styles, because of familiarity
> with LaTeX commands etc., I would argue that can be addressed in the
> style part of an insert processor and/or in an activate processor.
> E.g. I would argue this is a UI issue; not fundamentally about the
> styles names.
>
> Bruce
>

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

Vikas Rawal  writes:

> Dear John,
>
> Thanks very much for taking time to write a detailed reply.
>
>  I do not think it is productive for the community to say or consider it
>  is a sad situation. 
>
> From the perspective of a user, this was only meant to express a
> sentiment that one finds oneself in a situation of having to choose
> between two good things, and that we have not been able to find a way
> to make both compatible with each other. It was in not meant as a
> disrespect in any way.

I don't think you have to choose. You can use org-cite for
citations, and org-ref for cross-references. The citation syntax is
orthogonal, you just should not mix them. You can even wire org-ref to
use org-cite-insert like this:

(setq org-ref-insert-cite-function (lambda () (org-cite-insert nil)))

or use your own key binding if you need the prefix arg functionality of
org-cite-insert.

org-ref is more configurable and modular than it is given credit for.
This is not an either/or choice.

>
> My use case is very similar to yours and I have been an org-ref user
> for a long time (I was surely one of the earliest beneficiaries of
> your work), having written two books and innumerable research papers
> with org-ref citation syntax. Being able to export to LaTeX has been
> my primary use but the fact that citations were not exported easily to
> other formats thus far was a problem I had to struggle with every now
> and then.

with org-ref-3, this is much better, but org-cite also works well,
probably even better if you use footnote styles. Either way, it is not
trivial to get the same output from LaTeX and CSL for arbitrary styles.
They are often close though.

>  There are more than 8 years of legacy org-ref documents. I have written
>  40+ scientific papers with it, and countless technical documents with
>  more than 8000 cite links among them. org-ref has exceeded 190K
>  downloads from MELPA, so I feel obligated to maintain org-ref for
>  myself, and those users.
>
> Given that it is not very difficult to convert a document from old
> org-ref citation syntax to the org-cite syntax, at least as far as
> citation is concerned, this should not be a big problem. Do these
> documents use citation commands that are not available in org-cite?
> Can those not be added to org-cite?

It is possible to convert, I wouldn't say not very difficult. In
principle one maps over the links and citations, and then in reverse
order replace the one format with the converted other format in a
buffer. That basically means change & to @ (easy), and provide the
mapping between org-ref link type and org-cite style. org-ref has 70ish
cite commands that it supports because those are in bib(la)tex. If you
only use the basic types, it "should be easy". If you use others, you
will have to settle for approximately correct in some cases.

>  I think org-ref and org-cite have different priorities, they solve
>  different problems with different approaches, and they have different
>  pros and cons. 
>
> It might be useful to discuss specific gaps (such as citenum) that
> need to be plugged in org-cite for it to be usable. In fact, making
> org-cite usable for a heavyweight user like you is a useful goalpost.

citenum and bibentry are the only two I am not sure have a CSL analog. I
guess that things like citedate and citetitle are not fully implemented
in citeproc. That is fine, org-cite does not need to fully support all
of bib(la)tex, no single user needs that. Lots of users need one part
or another though, and collectively bib(la)tex exists for them.


>  
> I understand that you do not particularly like the modularity and
> complexity of org-cite way of specifying styles and variants. But if
> one is able to make the two compatible, filling the gaps, they could
> have a friendly co-existence with some way of being able to convert a
> document between the two styles. And if there are some
> incompatibilities that cannot be resolved, it would be good to know
> exactly what all those are. If somebody was to write functions to
> convert from one format to the other, they could choose how they want
> to deal with those incompatibilities.

I guess you won't know what all the incompatibilities are until you try
this. In my experience with org-ref, there is a years long tail of
discovery as people try different things and have corner cases of need.


>  
>  Cross-references are critical for me; without them, there is no path
>  forward for me with org-cite. I did work on a cross-reference approach
>  that leveraged org-cite syntax
>  (https://github.com/jkitchin/org-ref-cite/issues/16), but there was not
>  much appetite for the approach so I abandoned that. 
>
> What org-ref seems to do with cross-references is very nice.
> Unfortunately this would not be available if a user chooses to use
> org-cite. Do the capabilities of cross-referencing have to be wedded
> to the citation system?

They don't have to be integrated. Someone could separate out 

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 9:13 AM Dominik Schrempf
 wrote:
>
> “Bruce D’Arcus”  writes:
>
> > On Mon, Mar 21, 2022 at 8:41 AM Dominik Schrempf
> >  wrote:
> >>
> >> Thank you, I can use `citar-insert-edit` to perform this action.
> >
> > You can actually just use org-cite-insert, which is context aware.
> >
> >> Now, I failed to create a full citation in the text. This corresponds to 
> >> the
> >> `\fullcite{}’ command in LaTeX. Is there an option for this?
> >
> > No, but we should add one, as natbib has a similar command.
> >
> > And, of course, per Timothy’s point, you can customize
> > org-cite-biblatex-styles to include it in the meantime. If you want to
> > experiment with this, contact me off-list, as there’s some nuances
> > with citar.
>
> Great, that was easy, thank you:
>
> (add-to-list ’org-cite-biblatex-styles ’(“full” nil “fullcite” nil nil))

On citar (the "nuance"), until I make the style UI smarter, you'll
want to add this command and a preview to
citar-org-style-preview-alist.

You might want to even completely redefine it to preview using the
biblatex command.

> (This or something similar should be added upstream).

I agree.

We just need to decide what the style name should be (and also add a
mapping from it for bibentry in natbib, and hopefully later csl). We
already have a "full" variant for "author", so another "full" might be
a little confusing.

Bruce

> >> In summary, it is a bit painful to use `org-cite` compared to using 
> >> `org-ref`.
> >> In my opinion, the main reasons are:
> >>
> >> • Documentation is missing for `org-cite`.
> >> • The citation commands seem to be incomplete (or at least not documented).
> >
> > The documentation is admittedly thin. Thoughts on what we should add?
> >
> > The styles, and how to customize them with oc-biblatex?
>
> I know writing documentation is a lot of work. It is easy to complain, but 
> hard
> to contribute. I think we need:
> • A complete description of styles and variants (for the actual citation
>   command, and also for `#+cite_export: ...') or at least a link to an
>   up-to-date table.
> • An improved description of the capabilities of `org-cite-insert' (i.e., that
>   it can change the citation style and variant).
>
> >
> > Bruce

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

“Bruce D’Arcus”  writes:

> On Mon, Mar 21, 2022 at 8:41 AM Dominik Schrempf
>  wrote:
>>
>> Thank you, I can use `citar-insert-edit` to perform this action.
>
> You can actually just use org-cite-insert, which is context aware.
>
>> Now, I failed to create a full citation in the text. This corresponds to the
>> `\fullcite{}’ command in LaTeX. Is there an option for this?
>
> No, but we should add one, as natbib has a similar command.
>
> And, of course, per Timothy’s point, you can customize
> org-cite-biblatex-styles to include it in the meantime. If you want to
> experiment with this, contact me off-list, as there’s some nuances
> with citar.

Great, that was easy, thank you:

(add-to-list ’org-cite-biblatex-styles ’(“full” nil “fullcite” nil nil))

(This or something similar should be added upstream).

>
>> In summary, it is a bit painful to use `org-cite` compared to using 
>> `org-ref`.
>> In my opinion, the main reasons are:
>>
>> • Documentation is missing for `org-cite`.
>> • The citation commands seem to be incomplete (or at least not documented).
>
> The documentation is admittedly thin. Thoughts on what we should add?
>
> The styles, and how to customize them with oc-biblatex?

I know writing documentation is a lot of work. It is easy to complain, but hard
to contribute. I think we need:
• A complete description of styles and variants (for the actual citation
  command, and also for `#+cite_export: ...') or at least a link to an
  up-to-date table.
• An improved description of the capabilities of `org-cite-insert' (i.e., that
  it can change the citation style and variant).

>
> Bruce

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 8:41 AM Dominik Schrempf
 wrote:
>
> Thank you, I can use `citar-insert-edit` to perform this action.

You can actually just use org-cite-insert, which is context aware.

> Now, I failed to create a full citation in the text. This corresponds to the
> `\fullcite{}' command in LaTeX. Is there an option for this?

No, but we should add one, as natbib has a similar command.

And, of course, per Timothy's point, you can customize
org-cite-biblatex-styles to include it in the meantime. If you want to
experiment with this, contact me off-list, as there's some nuances
with citar.

> In summary, it is a bit painful to use `org-cite` compared to using `org-ref`.
> In my opinion, the main reasons are:
>
> • Documentation is missing for `org-cite`.
> • The citation commands seem to be incomplete (or at least not documented).

The documentation is admittedly thin. Thoughts on what we should add?

The styles, and how to customize them with oc-biblatex?

Bruce

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 8:23 AM John Kitchin  wrote:

>> A package could be created, say `org-cite-literal-biblatex' which is just a 
>> copy
>> of `oc-biblatex.el' with a different default `org-cite-biblatex-styles' and
>> `org-cite-biblatex-style-shortcuts' (or just sets those variables in
>> `org-cite-biblatex'). As far as I can tell this would provide exactly the
>> functionality you say org-cite can’t provide but org-ref does.
>
>
> I wrote this package you suggest in org-ref-cite. In discussions during that 
> development, it was clear the preference was on the more abstracted, and 
> uniform syntax across backends cite commands in org-cite, and not this kind 
> of variant. Of course one can do this. It is not that org-cite can't provide 
> it, it is that it doesn't at this time.

Just for some broader context on this particular issue.

The advantage of the org-cite style/variant design reflected in the
included export processors (natbib, biblatex, csl) is that the same
styles will mostly generate the same final output.

But that portability will only work with those styles and variants.

With the new org-cite-biblatex-styles defcustom, however, one can
augment or completely replace all those. But if you care about that
portability, you'd want to be aware of this, and think about it.

So per Timothy's point, you actually don't even need a new processor
for biblatex if you want to include all the extensive list of biblatex
commands.

Natbib AFAIK is already fully covered.

There's another POV on this though:

If one doesn't like to see the org-cite styles, because of familiarity
with LaTeX commands etc., I would argue that can be addressed in the
style part of an insert processor and/or in an activate processor.
E.g. I would argue this is a UI issue; not fundamentally about the
styles names.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

Thank you, I can use `citar-insert-edit` to perform this action.

Now, I failed to create a full citation in the text. This corresponds to the
`\fullcite{}' command in LaTeX. Is there an option for this?

In summary, it is a bit painful to use `org-cite` compared to using `org-ref`.
In my opinion, the main reasons are:

• Documentation is missing for `org-cite`.
• The citation commands seem to be incomplete (or at least not documented).

I am aware that all this is a lot of work, and I am really grateful that we have
`org-cite` and `org-ref` available!

Thank you,
Dominik

“Bruce D’Arcus”  writes:

> On Mon, Mar 21, 2022 at 4:24 AM Dominik Schrempf
>  wrote:
>
>> I am trying out `org-cite’ right now. It works much better than the last 
>> time (I
>> am using the `biblatex’ backend right now). However, I can not find any
>> documentation about the available /styles/.
>>
>> They are mentioned here: 
>>
>> But no styles are provided.
>
> I have come to believe the best approach to handling available styles
> and previews is in the style selection UI. So that one can see what
> one is waiting when selecting.
>
> In citar, those styles and preview are currently hard-coded in a
> defcustom, but I’d like to figure out how to do that dynamically,
> using org-cite-supported-styles etc.
>
> In the meantime, Timothy’s article should help (see the table on the styles):
>
> 
>
>> For example, I need citations in the text (showing up as “Author(s) (Year)”).
>> With BibTeX, this would be `\citet{}’ (or `\textcite{}’ with BibLaTeX). But
>> there is a lot more. See: 
>> ).
>
> You want the “text” style; (NO_ITEM_DATA:doe20), or with the shortcut
> (/t: NO_ITEM_DATA:doe20).
>
> Bruce

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

On Sun, Mar 20, 2022 at 9:57 PM Timothy  wrote:

> Hi John,
>
> Thanks for your considered response.
>
> When you contrast org-cite and org-ref, you say:
>
> > With org-ref, bib(la)tex export is almost fully supported, and is easy,
>
> I find this odd as org-cite supports bib(la)tex export, and rather easily.
>
> ┌
> │ #+bibliography: references.bib
> │ #+cite_export: biblatex authortitle/authortitle-ibid
> │
> │ [cite:@key] etc.
> │
> │ #+print_bibliography:
> └
>
> The limitation which I think is on your mind is that not all bib(la)tex
> commands
> are supported, and not in the “usual” form. For instance, to get
> `pnotecite' one
> would use `[cite/locators:]'. However, to get a 1-to-1 name mapping, you
> can just
> customise `org-cite-biblatex-styles'. For instance, `parencite' is not
> currently
> available, but if I just add `("parencite" nil "parencite" nil nil)' I can
> then do
> `[cite/parencite:]' or if I replace the first `"parencite"' with
> `"paren"', just
> `[cite/paren:]'.
>

Yes, of course I mean the citation commands.


>
> A package could be created, say `org-cite-literal-biblatex' which is just
> a copy
> of `oc-biblatex.el' with a different default `org-cite-biblatex-styles' and
> `org-cite-biblatex-style-shortcuts' (or just sets those variables in
> `org-cite-biblatex'). As far as I can tell this would provide exactly the
> functionality you say org-cite can’t provide but org-ref does.
>

I wrote this package you suggest in org-ref-cite. In discussions during
that development, it was clear the preference was on the more abstracted,
and uniform syntax across backends cite commands in org-cite, and not this
kind of variant. Of course one can do this. It is not that org-cite can't
provide it, it is that it doesn't at this time. Even so, it is only a
partial solution to deprecating org-ref.


>
> You can already use `.bib' files, and so frankly I cannot myself see the
> point in
> org-ref’s existence beyond bifurcating the community on this. At this
> point the
> only remaining motivation I see is old documents and current users, and
> for this
> a migration tool seems more appropriate.


> I don’t mean to be overly critical, however this is my current honest
> assessment
> of the situation.
>
> –
> All the best,
> Timothy
>
> John Kitchin  writes:
>
> > I do not think it is productive for the community to say or consider it
> > is a sad situation. Many good things have emerged from these
> > discussions, even if it is not yet consensus on a solution. It is a
> > complex problem, with many years of effort by many people on each side.
> > That is an indication of how ambitious this project is and that there
> > may be more than one solution that is needed. It pains me quite a bit
> > there is a sentiment of fractionation, and that I may be contributing to
> > it.
> >
> > My regular job workload the past few years has been crushing, and I have
> > not had the time to participate in this that I wish I had. I am not sure
> > I can add much here without sounding or feeling defensive about org-ref,
> > and my decision to continue supporting and developing it. I have thought
> > about this for most of the day, and in the (very long, apologies in
> > advance) response that follows I will do my best to provide a balanced
> > perspective (from my point of view) on the situation.
> >
> > Some specific context that is important to me is that I wrote org-ref
> > long ago to solve a specific problem for me in the preparation of
> > scientific publications that are destined for LaTeX export. I intended
> > it to provide nearly equivalent bib(la)tex citation export, and as
> > reasonable an export as possible for everything else. I use org-ref
> > professionally, and it is a complete solution for me. I simply cannot
> > compromise on the capability org-ref provides me, or wait for an
> > alternative complete solution in org-mode. I have work I have to do now,
> > and org-ref lets me do it. This alone is reason enough for me to
> > continue using, developing and supporting org-ref. I understand org is
> > not intended to be a substitute for writing LaTeX, but it is a fact of
> > my job that I have to do that.
> >
> > There are more than 8 years of legacy org-ref documents. I have written
> > 40+ scientific papers with it, and countless technical documents with
> > more than 8000 cite links among them. org-ref has exceeded 190K
> > downloads from MELPA, so I feel obligated to maintain org-ref for
> > myself, and those users. org-ref may be heavyweight in bundling a lot of
> > capability together that could be separated into individual packages,
> > but it is also convenient for people who need it, and a GitHUB issue or
> > pull request away from new features. I remain committed to supporting
> > this, and I do it in a way I can manage, hence the monolithic package
> > design.
> >
> > org-cite was also developed to solve some specific citation problems for
> > others that org-ref did not address well 

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Mon, Mar 21, 2022 at 4:24 AM Dominik Schrempf
 wrote:

> I am trying out `org-cite' right now. It works much better than the last time 
> (I
> am using the `biblatex' backend right now). However, I can not find any
> documentation about the available /styles/.
>
> They are mentioned here: 
>
> But no styles are provided.

I have come to believe the best approach to handling available styles
and previews is in the style selection UI. So that one can see what
one is waiting when selecting.

In citar, those styles and preview are currently hard-coded in a
defcustom, but I'd like to figure out how to do that dynamically,
using org-cite-supported-styles etc.

In the meantime, Timothy's article should help (see the table on the styles):

https://blog.tecosaur.com/tmio/2021-07-31-citations.html

> For example, I need citations in the text (showing up as “Author(s) (Year)”).
> With BibTeX, this would be `\citet{}' (or `\textcite{}' with BibLaTeX). But
> there is a lot more. See: 
> ).

You want the "text" style; [cite/text:@doe20], or with the shortcut
[cite:/t:@doe20].

Bruce

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

Hi,

I am trying out `org-cite' right now. It works much better than the last time (I
am using the `biblatex' backend right now). However, I can not find any
documentation about the available /styles/.

They are mentioned here: 

But no styles are provided.

For example, I need citations in the text (showing up as “Author(s) (Year)”).
With BibTeX, this would be `\citet{}' (or `\textcite{}' with BibLaTeX). But
there is a lot more. See: 
).

Thank you for working on `org-cite', and `org-ref'!
Dominik

Timothy  writes:

> Hi John,
>
> Thanks for your considered response.
>
> When you contrast org-cite and org-ref, you say:
>
>> With org-ref, bib(la)tex export is almost fully supported, and is easy,
>
> I find this odd as org-cite supports bib(la)tex export, and rather easily.
>
> ┌
> │ #+bibliography: references.bib
> │ #+cite_export: biblatex authortitle/authortitle-ibid
> │
> │ (NO_ITEM_DATA:key) etc.
> │
> │ #+print_bibliography:
> └
>
> The limitation which I think is on your mind is that not all bib(la)tex 
> commands
> are supported, and not in the “usual” form. For instance, to get `pnotecite’ 
> one
> would use `[cite/locators:]’. However, to get a 1-to-1 name mapping, you can 
> just
> customise `org-cite-biblatex-styles’. For instance, `parencite’ is not 
> currently
> available, but if I just add `(“parencite” nil “parencite” nil nil)’ I can 
> then do
> `[cite/parencite:]’ or if I replace the first `“parencite”’ with `“paren”’, 
> just
> `[cite/paren:]’.
>
> A package could be created, say `org-cite-literal-biblatex’ which is just a 
> copy
> of `oc-biblatex.el’ with a different default `org-cite-biblatex-styles’ and
> `org-cite-biblatex-style-shortcuts’ (or just sets those variables in
> `org-cite-biblatex’). As far as I can tell this would provide exactly the
> functionality you say org-cite can’t provide but org-ref does.
>
> You can already use `.bib’ files, and so frankly I cannot myself see the 
> point in
> org-ref’s existence beyond bifurcating the community on this. At this point 
> the
> only remaining motivation I see is old documents and current users, and for 
> this
> a migration tool seems more appropriate.
>
> I don’t mean to be overly critical, however this is my current honest 
> assessment
> of the situation.
>
> –
> All the best,
> Timothy
>
> John Kitchin  writes:
>
>> I do not think it is productive for the community to say or consider it
>> is a sad situation. Many good things have emerged from these
>> discussions, even if it is not yet consensus on a solution. It is a
>> complex problem, with many years of effort by many people on each side.
>> That is an indication of how ambitious this project is and that there
>> may be more than one solution that is needed. It pains me quite a bit
>> there is a sentiment of fractionation, and that I may be contributing to
>> it.
>>
>> My regular job workload the past few years has been crushing, and I have
>> not had the time to participate in this that I wish I had. I am not sure
>> I can add much here without sounding or feeling defensive about org-ref,
>> and my decision to continue supporting and developing it. I have thought
>> about this for most of the day, and in the (very long, apologies in
>> advance) response that follows I will do my best to provide a balanced
>> perspective (from my point of view) on the situation.
>>
>> Some specific context that is important to me is that I wrote org-ref
>> long ago to solve a specific problem for me in the preparation of
>> scientific publications that are destined for LaTeX export. I intended
>> it to provide nearly equivalent bib(la)tex citation export, and as
>> reasonable an export as possible for everything else. I use org-ref
>> professionally, and it is a complete solution for me. I simply cannot
>> compromise on the capability org-ref provides me, or wait for an
>> alternative complete solution in org-mode. I have work I have to do now,
>> and org-ref lets me do it. This alone is reason enough for me to
>> continue using, developing and supporting org-ref. I understand org is
>> not intended to be a substitute for writing LaTeX, but it is a fact of
>> my job that I have to do that.
>>
>> There are more than 8 years of legacy org-ref documents. I have written
>> 40+ scientific papers with it, and countless technical documents with
>> more than 8000 cite links among them. org-ref has exceeded 190K
>> downloads from MELPA, so I feel obligated to maintain org-ref for
>> myself, and those users. org-ref may be heavyweight in bundling a lot of
>> capability together that could be separated into individual packages,
>> but it is also convenient for people who need it, and a GitHUB issue or
>> pull request away from new features. I remain committed to supporting
>> this, and I do it in a way I can manage, hence the monolithic package
>> design.
>>
>> org-cite was 

Re: citations: org-cite vs org-ref 3.0

[Vikas Rawal]

Dear John,

Thanks very much for taking time to write a detailed reply.

I do not think it is productive for the community to say or consider it
> is a sad situation.


>From the perspective of a user, this was only meant to express a sentiment
that one finds oneself in a situation of having to choose between two good
things, and that we have not been able to find a way to make both
compatible with each other. It was in not meant as a disrespect in any way.

My use case is very similar to yours and I have been an org-ref user for a
long time (I was surely one of the earliest beneficiaries of your work),
having written two books and innumerable research papers with org-ref
citation syntax. Being able to export to LaTeX has been my primary use but
the fact that citations were not exported easily to other formats thus far
was a problem I had to struggle with every now and then.


> There are more than 8 years of legacy org-ref documents. I have written
> 40+ scientific papers with it, and countless technical documents with
> more than 8000 cite links among them. org-ref has exceeded 190K
> downloads from MELPA, so I feel obligated to maintain org-ref for
> myself, and those users.


Given that it is not very difficult to convert a document from old org-ref
citation syntax to the org-cite syntax, at least as far as citation is
concerned, this should not be a big problem. Do these documents use
citation commands that are not available in org-cite? Can those not be
added to org-cite?


> I think org-ref and org-cite have different priorities, they solve
> different problems with different approaches, and they have different
> pros and cons.


It might be useful to discuss specific gaps (such as citenum) that need to
be plugged in org-cite for it to be usable. In fact, making org-cite usable
for a heavyweight user like you is a useful goalpost.

I understand that you do not particularly like the modularity and
complexity of org-cite way of specifying styles and variants. But if one is
able to make the two compatible, filling the gaps, they could have a
friendly co-existence with some way of being able to convert a document
between the two styles. And if there are some incompatibilities that cannot
be resolved, it would be good to know exactly what all those are. If
somebody was to write functions to convert from one format to the other,
they could choose how they want to deal with those incompatibilities.


> Cross-references are critical for me; without them, there is no path
> forward for me with org-cite. I did work on a cross-reference approach
> that leveraged org-cite syntax
> (https://github.com/jkitchin/org-ref-cite/issues/16), but there was not
> much appetite for the approach so I abandoned that.


What org-ref seems to do with cross-references is very nice. Unfortunately
this would not be available if a user chooses to use org-cite. Do the
capabilities of cross-referencing have to be wedded to the citation system?

Can this not be resolved?


> I am content to agree to disagree on these points and move forward with
> both packages because they solve different problems, are suitable for
> different communities, and they continue to benefit each other.


Friendly co-existence should be our goal. But can that be a situation in
which one is able to choose between the best of both and, as far as
possible, switch from one to the other.

Thanks again for your time and effort,

Vikas

Re: citations: org-cite vs org-ref 3.0

[Timothy]

Hi John,

Thanks for your considered response.

When you contrast org-cite and org-ref, you say:

> With org-ref, bib(la)tex export is almost fully supported, and is easy,

I find this odd as org-cite supports bib(la)tex export, and rather easily.

┌
│ #+bibliography: references.bib
│ #+cite_export: biblatex authortitle/authortitle-ibid
│ 
│ [cite:@key] etc.
│ 
│ #+print_bibliography:
└

The limitation which I think is on your mind is that not all bib(la)tex commands
are supported, and not in the “usual” form. For instance, to get `pnotecite' one
would use `[cite/locators:]'. However, to get a 1-to-1 name mapping, you can 
just
customise `org-cite-biblatex-styles'. For instance, `parencite' is not currently
available, but if I just add `("parencite" nil "parencite" nil nil)' I can then 
do
`[cite/parencite:]' or if I replace the first `"parencite"' with `"paren"', just
`[cite/paren:]'.

A package could be created, say `org-cite-literal-biblatex' which is just a copy
of `oc-biblatex.el' with a different default `org-cite-biblatex-styles' and
`org-cite-biblatex-style-shortcuts' (or just sets those variables in
`org-cite-biblatex'). As far as I can tell this would provide exactly the
functionality you say org-cite can’t provide but org-ref does.

You can already use `.bib' files, and so frankly I cannot myself see the point 
in
org-ref’s existence beyond bifurcating the community on this. At this point the
only remaining motivation I see is old documents and current users, and for this
a migration tool seems more appropriate.

I don’t mean to be overly critical, however this is my current honest assessment
of the situation.

–
All the best,
Timothy

John Kitchin  writes:

> I do not think it is productive for the community to say or consider it
> is a sad situation. Many good things have emerged from these
> discussions, even if it is not yet consensus on a solution. It is a
> complex problem, with many years of effort by many people on each side.
> That is an indication of how ambitious this project is and that there
> may be more than one solution that is needed. It pains me quite a bit
> there is a sentiment of fractionation, and that I may be contributing to
> it.
>
> My regular job workload the past few years has been crushing, and I have
> not had the time to participate in this that I wish I had. I am not sure
> I can add much here without sounding or feeling defensive about org-ref,
> and my decision to continue supporting and developing it. I have thought
> about this for most of the day, and in the (very long, apologies in
> advance) response that follows I will do my best to provide a balanced
> perspective (from my point of view) on the situation.
>
> Some specific context that is important to me is that I wrote org-ref
> long ago to solve a specific problem for me in the preparation of
> scientific publications that are destined for LaTeX export. I intended
> it to provide nearly equivalent bib(la)tex citation export, and as
> reasonable an export as possible for everything else. I use org-ref
> professionally, and it is a complete solution for me. I simply cannot
> compromise on the capability org-ref provides me, or wait for an
> alternative complete solution in org-mode. I have work I have to do now,
> and org-ref lets me do it. This alone is reason enough for me to
> continue using, developing and supporting org-ref. I understand org is
> not intended to be a substitute for writing LaTeX, but it is a fact of
> my job that I have to do that.
>
> There are more than 8 years of legacy org-ref documents. I have written
> 40+ scientific papers with it, and countless technical documents with
> more than 8000 cite links among them. org-ref has exceeded 190K
> downloads from MELPA, so I feel obligated to maintain org-ref for
> myself, and those users. org-ref may be heavyweight in bundling a lot of
> capability together that could be separated into individual packages,
> but it is also convenient for people who need it, and a GitHUB issue or
> pull request away from new features. I remain committed to supporting
> this, and I do it in a way I can manage, hence the monolithic package
> design.
>
> org-cite was also developed to solve some specific citation problems for
> others that org-ref did not address well at the time it was started. I
> believe those were issues like better pre/post note support, and
> integration with CSL.
>
> I think org-ref and org-cite have different priorities, they solve
> different problems with different approaches, and they have different
> pros and cons. I believe there are mutually incompatible compromises one
> must make here because the specific choices you make determine what is
> easy and what is possible. There is not a direct mapping of bib(la)tex
> and CSL as a citation processor. They are different programs and they
> don’t share citation commands, or style information. It is inevitable
> that something will be lost in the translation between these 

Re: citations: org-cite vs org-ref 3.0

[John Kitchin]

I do not think it is productive for the community to say or consider it
is a sad situation. Many good things have emerged from these
discussions, even if it is not yet consensus on a solution. It is a
complex problem, with many years of effort by many people on each side.
That is an indication of how ambitious this project is and that there
may be more than one solution that is needed. It pains me quite a bit
there is a sentiment of fractionation, and that I may be contributing to
it.

My regular job workload the past few years has been crushing, and I have
not had the time to participate in this that I wish I had. I am not sure
I can add much here without sounding or feeling defensive about org-ref,
and my decision to continue supporting and developing it. I have thought
about this for most of the day, and in the (very long, apologies in
advance) response that follows I will do my best to provide a balanced
perspective (from my point of view) on the situation.

Some specific context that is important to me is that I wrote org-ref
long ago to solve a specific problem for me in the preparation of
scientific publications that are destined for LaTeX export. I intended
it to provide nearly equivalent bib(la)tex citation export, and as
reasonable an export as possible for everything else. I use org-ref
professionally, and it is a complete solution for me. I simply cannot
compromise on the capability org-ref provides me, or wait for an
alternative complete solution in org-mode. I have work I have to do now,
and org-ref lets me do it. This alone is reason enough for me to
continue using, developing and supporting org-ref. I understand org is
not intended to be a substitute for writing LaTeX, but it is a fact of
my job that I have to do that.

There are more than 8 years of legacy org-ref documents. I have written
40+ scientific papers with it, and countless technical documents with
more than 8000 cite links among them. org-ref has exceeded 190K
downloads from MELPA, so I feel obligated to maintain org-ref for
myself, and those users. org-ref may be heavyweight in bundling a lot of
capability together that could be separated into individual packages,
but it is also convenient for people who need it, and a GitHUB issue or
pull request away from new features. I remain committed to supporting
this, and I do it in a way I can manage, hence the monolithic package
design.

org-cite was also developed to solve some specific citation problems for
others that org-ref did not address well at the time it was started. I
believe those were issues like better pre/post note support, and
integration with CSL.

I think org-ref and org-cite have different priorities, they solve
different problems with different approaches, and they have different
pros and cons. I believe there are mutually incompatible compromises one
must make here because the specific choices you make determine what is
easy and what is possible. There is not a direct mapping of bib(la)tex
and CSL as a citation processor. They are different programs and they
don't share citation commands, or style information. It is inevitable
that something will be lost in the translation between these from a
single source like org, and whether that loss is acceptable depends on
what you need. For many things, close enough is ok for me, but for
manuscripts and proposals, they must be perfect, and in bib(la)tex form
for the journals I publish in. It is not that one thing is possible in
one and not the other; it is that you have to compromise one to do the
other no matter what you choose and that typically makes one thing or
the other easier to do. I am not even sure it is possible to do
everything one can do in bib(la)tex with CSL, for example, I don't know
the equivalent of \citenum in CSL. org-ref sides on making bib(la)tex
easy, and CSL is possible.

With org-cite, CSL can be readily used across export backends, more
bibliography database formats are supported, and it is also possible to
get LaTeX export, although there is no goal to fully support all of
bib(la)tex. A pure org approach (e.g. org-files as bibliography
database) can be used, there are a lot of CSL styles to work with, and
you can develop or adapt your own style if needed. With reasonable
defaults this should be a straightforward introduction to using
citations for new users that works out of the box. I support that.

With org-ref, bib(la)tex export is almost fully supported, and is easy,
and other exports via CSL are possible. Of course, you must have a
working LaTeX installation, only bibtex databases are supported, and
working knowledge of bib(la)tex is required to leverage this. Whether
you want it or not, you get a lot of extra utilities for getting bibtex
entries from a DOI, and support for cross-references, indexes,
glossaries and acronyms. This is a lot to ask of people who don't need
it, and convenient for those who do. I support this.

Which one of these approaches is right depends on what you want to 

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

“Bruce D’Arcus”  writes:

> On Sun, Mar 20, 2022 at 4:21 PM Dominik Schrempf
>  wrote:
>
>> For what it’s worth, I use `org-ref` because fine-grained citation export 
>> with
>> LaTeX (using BibTeX or BibLaTeX) only works with `org-ref`, and not with
>> `org-cite`.
>
> Part of the challenge is this isn’t an apples-apples comparison;
> org-cite a framework, and it’s the processors that really provide the
> functionality.
>
>> If I remember correctly, `org-cite` exports the formatted citations into the
>> LaTeX file …
>
> This is not true.

Thanks for correcting me on that. I remember having LaTeX files with full
citations in them, but maybe I messed something up.

>
> Or rather, it’s only true if you use the oc-csl export processor; you
> can use other export processors (like oc-biblatex) to get whatever
> LaTeX output you want, and …
>
>>  …. it is really difficult to use different style files, etc. With
>> `org-ref`, I can use BibLaTeX with specific style files. (Journal require
>> different style files). I can even use my personal LaTeX classes (with 
>> specific
>> bibliography styles) for Org export.
>
> … I believe Nicolas recently added support for customizing the
> export styles for biblatex.
>
> Bruce

When I tried setting `org-cite` up, it didn’t work. Maybe I should give it
another try at some point. For now, I am happy with `org-ref`.

Re: citations: org-cite vs org-ref 3.0

[chris]

On Sunday, 20 March 2022 20:44:50 CET Bruce D'Arcus wrote:
> On Sun, Mar 20, 2022 at 9:42 AM Ihor Radchenko  wrote:
> > For citar, why not simply using ivy-bibtex? It supports org-cite, AFAIK.
> 
> Not really; or rather minimally.

I use `org-cite` with a very minimal configuration, and it works very well. I 
don't use `ivy` at all. I don't use `helm` at all. Those are very large 
framework and should not be forced to people. (I don't use `org-ref`.)

I only use `citar` with minimal configuration. I use `vertico` for the 
completion. `citar` is simple enough for me to be able to read and understand 
a large part of it.

IMO the more layers of code there are, the more difficult it is to have things 
work right. And similarly with the size of the code.

Chris

> 
> Ivy-bibtex supports, for example, inserting of org-cite citations, but
> not via org-cite-insert.

And I have `org-cite-insert` working straight out of the box.

> 
> So there are currently no org-cite processors for ivy-bibtex etc.
> 
> Bruce

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 20, 2022 at 4:21 PM Dominik Schrempf
 wrote:

> For what it’s worth, I use `org-ref` because fine-grained citation export with
> LaTeX (using BibTeX or BibLaTeX) only works with `org-ref`, and not with
> `org-cite`.

Part of the challenge is this isn't an apples-apples comparison;
org-cite a framework, and it's the processors that really provide the
functionality.

> If I remember correctly, `org-cite` exports the formatted citations into the
> LaTeX file ...

This is not true.

Or rather, it's only true if you use the oc-csl export processor; you
can use other export processors (like oc-biblatex) to get whatever
LaTeX output you want, and ...

>   it is really difficult to use different style files, etc. With
> `org-ref`, I can use BibLaTeX with specific style files. (Journal require
> different style files). I can even use my personal LaTeX classes (with 
> specific
> bibliography styles) for Org export.

... I believe Nicolas recently added support for customizing the
export styles for biblatex.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Vikas Rawal]

>
>
>
> For what it’s worth, I use `org-ref` because fine-grained citation export
> with
> LaTeX (using BibTeX or BibLaTeX) only works with `org-ref`, and not with
> `org-cite`.
>

 It would help if you can explain what you mean.

If I remember correctly, `org-cite` exports the formatted citations into the
> LaTeX file, which is a no-go for me (let me know if I am wrong).


This is not correct. org-cite exports citation commands to LaTeX, and LaTeX
does rest of the processing. I do not think there is any difference in the
approach between org-cite and org-ref.


> Then, for
> example, it is really difficult to use different style files, etc. With
> `org-ref`, I can use BibLaTeX with specific style files. (Journal require
> different style files). I can even use my personal LaTeX classes (with
> specific
> bibliography styles) for Org export.
>
>
Surely org-cite allows you to do all of this.

Vikas

Re: citations: org-cite vs org-ref 3.0

[Dominik Schrempf]

“Thomas S. Dye”  writes:

> Ihor Radchenko  writes:
>
>> Vikas Rawal  writes:
>>
>>> What is the general view of the community about this? Is there a
>>> comprehensive discussion of pros and cons of each?
>>
>> Prof. Kitchin himself provided a summary on why he decided to give up on
>> using org-cite. See 
>>
> This claim in Kitchin’s summary surprised me:
>
> “It comes down to org-ref using bibtex/biblatex as the predominant citation
> processor, and org-cite using CSL. These two processors have different 
> syntaxes,
> and I don’t think it is possible to have a single approach that works for both
> of them without making compromises in capability.”
>
> I wonder what are the “compromises in capability” required to support both?
>
> I haven’t found them yet, though I’m just now finishing old projects started
> prior to the release of org-cite and haven’t worked through a LaTeX export
> project with it.  My first, and so far only, project with org-cite was for a
> simple html-based presentation using CSL, and I was pleased with the results 
> and
> relative ease of setup.
>
> All the best,
> Tom

For what it’s worth, I use `org-ref` because fine-grained citation export with
LaTeX (using BibTeX or BibLaTeX) only works with `org-ref`, and not with
`org-cite`.

If I remember correctly, `org-cite` exports the formatted citations into the
LaTeX file, which is a no-go for me (let me know if I am wrong). Then, for
example, it is really difficult to use different style files, etc. With
`org-ref`, I can use BibLaTeX with specific style files. (Journal require
different style files). I can even use my personal LaTeX classes (with specific
bibliography styles) for Org export.

Not sure if this helps…, but this is maybe what John Kitchin is referring to.

Dominik

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 20, 2022 at 9:42 AM Ihor Radchenko  wrote:

> For citar, why not simply using ivy-bibtex? It supports org-cite, AFAIK.

Not really; or rather minimally.

Ivy-bibtex supports, for example, inserting of org-cite citations, but
not via org-cite-insert.

So there are currently no org-cite processors for ivy-bibtex etc.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Thomas S. Dye]

Ihor Radchenko  writes:


Vikas Rawal  writes:

What is the general view of the community about this? Is there 
a

comprehensive discussion of pros and cons of each?


Prof. Kitchin himself provided a summary on why he decided to 
give up on
using org-cite. See 
https://github.com/jkitchin/org-ref/issues/892



This claim in Kitchin's summary surprised me:

"It comes down to org-ref using bibtex/biblatex as the predominant 
citation processor, and org-cite using CSL. These two processors 
have different syntaxes, and I don't think it is possible to have 
a single approach that works for both of them without making 
compromises in capability."


I wonder what are the "compromises in capability" required to 
support both?


I haven't found them yet, though I'm just now finishing old 
projects started prior to the release of org-cite and haven't 
worked through a LaTeX export project with it.  My first, and so 
far only, project with org-cite was for a simple html-based 
presentation using CSL, and I was pleased with the results and 
relative ease of setup.


All the best,
Tom

--
Thomas S. Dye
https://tsdye.online/tsdye

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 20, 2022 at 10:08 AM Vikas Rawal  wrote:
>>
>> > What is the general view of the community about this?
>>
>> I don't know about the general view of the community, but, as a data
>> point, I find it very sad.
>
>
> Exactly how I feel. Particularly so because org-cite was indeed inspired from 
> org-ref and all the great work John has done on this over the years. I do not 
> see any substantive discussion of the limitations that John has pointed out 
> with org-cite in the archives of the mailing list. We should have a 
> conversation about those.
>
> If I understand correctly, the main limitation according to him is that 
> org-cite does not handle cross-references. Is that the only problem? What do 
> others think about it? Do we want the citation system to be able to handle 
> cross-references? If not, do we need a separate system that handles 
> cross-references beyond what is available in org-mode already? Can that 
> system not be created in a manner that would not break the org-cite syntax?
>
> I think we should have a conversation and collectively think of what is the 
> best way forward.

Yes, and no.

Conversation on org-cite took place on this list for years, with
diverse people, with diverse expertise and POV, cycling in and out
over that time.

John raised the cross-reference question towards the very end, during
the testing period, when it was almost merged.

https://www.mail-archive.com/emacs-orgmode@gnu.org/msg138349.html

I still think the path forward on that is TBD, though as I said, I
don't think it should be in scope for org-cite per se, but org more
generally yes.

If he has some other issues (which I suspect he does; maybe related to
fontification?) that might be addressed, I'm sure he'll raise them.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Vikas Rawal]

>
> > What is the general view of the community about this?
>
> I don't know about the general view of the community, but, as a data
> point, I find it very sad.
>

Exactly how I feel. Particularly so because org-cite was indeed inspired
from org-ref and all the great work John has done on this over the years. I
do not see any substantive discussion of the limitations that John has
pointed out with org-cite in the archives of the mailing list. We should
have a conversation about those.

If I understand correctly, the main limitation according to him is that
org-cite does not handle cross-references. Is that the only problem? What
do others think about it? Do we want the citation system to be able to
handle cross-references? If not, do we need a separate system that handles
cross-references beyond what is available in org-mode already? Can that
system not be created in a manner that would not break the org-cite syntax?

I think we should have a conversation and collectively think of what is the
best way forward.

Vikas

Re: citations: org-cite vs org-ref 3.0

[Ihor Radchenko]

Vikas Rawal  writes:

> What is the general view of the community about this? Is there a
> comprehensive discussion of pros and cons of each?

Prof. Kitchin himself provided a summary on why he decided to give up on
using org-cite. See https://github.com/jkitchin/org-ref/issues/892

> What is everyone doing? I was an org-ref user for long before I switched to
> org-cite. I can now shift to citar but since I am an ivy user, the switch
> is not trivial. Also, I like many helper functions that John has created,
> and would have to miss those if I do not use org-ref/org-ref-cite.

I personally used org-ref for a long time simply because no
alternatives. I dislike org-ref because Prof. Kitchin prefers to do
things in very specific way and it is not always possible to customise
them. It is hard to use only a part of org-ref without using everything
and getting surprising behaviour.

Now, I am trying to switch to org-cite + .org files as bibliography. But
the whole thing is an ad-hoc mess.

For citar, why not simply using ivy-bibtex? It supports org-cite, AFAIK.

Best,
Ihor

Re: citations: org-cite vs org-ref 3.0

[Bruce D'Arcus]

On Sun, Mar 20, 2022 at 8:09 AM Vikas Rawal  wrote:

> What is the general view of the community about this? Is there a 
> comprehensive discussion of pros and cons of each?

Not really, but there's John's summary here:

https://github.com/jkitchin/org-ref#what-about-org-cite

The high-level discussion at the beginning, including the enumerated
points, seems right to me.

The following point that "org-cite does not meet my citation and
technical document publishing needs, and it was not possible to
integrate it into org-ref without compromising those" is more subject
to debate, particularly the first clause.

I don't see any practical advantage to the org-ref syntax and model,
unless you include cross-references and such.

But that's because I just don't think cross-references and indexes
should be handled in org-cite; I think if we need improvements in
existing cross-references etc support, we should add those there.

I suspect he's also meaning the different ways that citation
commands/styles are handled in the two systems.

Here's an example from org-ref 3:

[[citealp:See  page 2]]

Note that first piece: "citealp". That's the command, which org-ref
will output directly to LaTeX/Natbib as \citealp. Note that command
only makes sense for natbib.

Org-cite has a different abstraction here: the style/substyle system.
So the above would be ...

[cite/bare:See @kitchin-2015-examp page 2]

... and then whatever export processor would map that "bare" style to
the appropriate output.

So that leaves the last bit; the "not possible" point, which I can't
address. It would obviously be nice if org-ref supported org-cite in
the future.

> What is everyone doing? I was an org-ref user for long before I switched to 
> org-cite. I can now shift to citar but since I am an ivy user, the switch is 
> not trivial. Also, I like many helper functions that John has created, and 
> would have to miss those if I do not use org-ref/org-ref-cite.

A month or so ago, John offered to turn over maintenance of
org-ref-cite to someone else, which might be one good option for
someone who uses ivy, in particular, and interested in developing it
further.

More generally, the modular design of org-cite should result, in time,
with diverse components, including for different completion
frameworks. For example, I think it'd be pretty easy to create an
insert process for helm-bibtex or ivy-bibtex, and a follow processor
for bibtex-completion.

Or alternatively, as citar now no longer requires bibtex-completion,
someone could write a small citar front-end for ivy as an insert
processor.

The whole point of the org-ctite design is it should be easy for users
to mix-and-match different pieces, and for documents to remain
compatible across users and export backends. It also allows developers
to focus on those small components.

Bruce

Re: citations: org-cite vs org-ref 3.0

[Nicolas Goaziou]

Hello,

Vikas Rawal  writes:

> This obviously creates many problems including that two people using
> different citation systems cannot share org files.

Indeed.
>
> What is the general view of the community about this?

I don't know about the general view of the community, but, as a data
point, I find it very sad.

That's not helpful, tho.

Regards,
-- 
Nicolas Goaziou