RE: state of the art in org-mode tables e.g. join, etc

[Cook, Malcolm]

John,

Checkout what R sqldf package makes easy:

** aggregation example

Examples from https://github.com/tbanel/orgaggregate


#+NAME: original
| Day   | Color | Level | Quantity |
|---+---+---+--|
| Monday| Red   |30 |   11 |
| Monday| Blue  |25 |3 |
| Tuesday   | Red   |51 |   12 |
| Tuesday   | Red   |45 |   15 |
| Tuesday   | Blue  |33 |   18 |
| Wednesday | Red   |27 |   23 |
| Wednesday | Blue  |12 |   16 |
| Wednesday | Blue  |15 |   15 |
| Thursday  | Red   |39 |   24 |
| Thursday  | Red   |41 |   29 |
| Thursday  | Red   |49 |   30 |
| Friday| Blue  | 7 |5 |
| Friday| Blue  | 6 |8 |
| Friday| Blue  |11 |9 |

#+PROPERTY: header-args:R  :session *R*

#+begin_src R :results none
library(sqldf)
#+end_src


#+begin_src R :var original=original :colnames yes
sqldf('select Color, count(*) from original group by Color;')
#+end_src

#+RESULTS:
| Color | count(*) |
|---+--|
| Blue  |7 |
| Red   |7 |



** join example

Example from https://github.com/tbanel/orgtbljoin

#+name: nutrition
| type | Fiber | Sugar | Protein | Carb |
|--+---+---+-+--|
| eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| onion|   1.3 |   4.4 | 1.3 |  9.0 |
| egg  | 0 |  18.3 |31.9 | 18.3 |
| rice |   0.2 | 0 | 1.5 | 16.0 |
| bread|   0.7 |   0.7 | 3.3 | 16.0 |
| orange   |   3.1 |  11.9 | 1.3 | 17.6 |
| banana   |   2.1 |   9.9 | 0.9 | 18.5 |
| tofu |   0.7 |   0.5 | 6.6 |  1.4 |
| nut  |   2.6 |   1.3 | 4.9 |  7.2 |
| corn |   4.7 |   1.8 | 2.8 | 21.3 |


#+name: recipe
| type | quty |
|--+--|
| onion|   70 |
| tomatoe  |  120 |
| eggplant |  300 |
| tofu |  100 |


#+begin_src R :var recipe=recipe :var nutrition=nutrition :colnames yes
sqldf('select * from recipe, nutrition where recipe.type=nutrition.type')
#+end_src

#+RESULTS:
| type | quty | type | Fiber | Sugar | Protein | Carb |
|--+--+--+---+---+-+--|
| onion|   70 | onion|   1.3 |   4.4 | 1.3 |9 |
| tomatoe  |  120 | tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| eggplant |  300 | eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tofu |  100 | tofu |   0.7 |   0.5 | 6.6 |  1.4 |



This should also be possible but I cannot get it to work now:

#+begin_src R :var recipe=recipe :var nutrition=nutrition :colnames yes 
:prologue sqldf(' :epilogue ')
select * from recipe, nutrition where recipe.type=nutrition.type
#+end_src





From: Emacs-orgmode  On Behalf 
Of John Kitchin
Sent: Sunday, February 21, 2021 10:24
To: Tim Cross 
Cc: org-mode-email 
Subject: Re: state of the art in org-mode tables e.g. join, etc

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.

For fun, here is the sqlite equivalent of the Pandas example using the same 
tables as before


** aggregation example

Examples from https://github.com/tbanel/orgaggregate


#+NAME: original
| Day   | Color | Level | Quantity |
|---+---+---+--|
| Monday| Red   |30 |   11 |
| Monday| Blue  |25 |3 |
| Tuesday   | Red   |51 |   12 |
| Tuesday   | Red   |45 |   15 |
| Tuesday   | Blue  |33 |   18 |
| Wednesday | Red   |27 |   23 |
| Wednesday | Blue  |12 |   16 |
| Wednesday | Blue  |15 |   15 |
| Thursday  | Red   |39 |   24 |
| Thursday  | Red   |41 |   29 |
| Thursday  | Red   |49 |   30 |
| Friday| Blue  | 7 |5 |
| Friday| Blue  | 6 |8 |
| Friday| Blue  |11 |9 |


#+begin_src sqlite :db ":memory:" :var orgtable=original :colnames yes
drop table if exists testtable;
create table testtable(Day str, Color str, Level int, Quantity int);
.mode csv testtable
.import $orgtable testtable
select Color, count(*) from testtable group by Color;
#+end_src

#+RESULTS:
| Color | count(*) |
|---+--|
| Blue  |7 |
| Red   |7 |

** join example

Example from https://github.com/tbanel/orgtbljoin

#+name: nutrition
| type | Fiber | Sugar | Protein | Carb |
|--+---+---+-+--|
| eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| onion|   1.3 |   4.4 | 1.3 |  9.0 |
| egg  | 0 |  18.3 |31.9 | 18.3 |
| rice |   0.2 | 0 | 1.5 | 16.0 |
| bread|   0.7 |   0.7 | 3.3 | 16.0 |
| orange   |   3.1 |  11.9 | 1.3 | 17.6 |
| banana   |   2.1 |   9.9 | 0.9 | 18.5 |
| tofu |   0.7 |   0.5 | 6.6 |  1.4 |
| nut  |   2.6 |   1.3 | 4.9 |  7.2 |
| corn |   4.7 |   1.8 | 2.8 | 21.3 |


#+name: re

Re: 'false' list item

[Greg Minshall]

Tim,

> There is no plans to change anything as far as I know. What I wrote was
> mainly to show why we have the situation and that any proposed solution
> has its own drawbacks.

thanks.  (i assumed that, but ...)

> Bottom line, we cannot easily prevent the 'false' list item issue
> without introducing either other issues or adding some additional syntax
> to indicate list items, which defeats the 'plain' aspects which many
> appreciate in org. Even the proposed 'solutions' still suffer from false
> positives.

yeah.  i sort of wonder if one can roughly categorize our typical uses
of org-mode into two types:

- random scribbles, agendas, etc., where the consumer is "ourselves" for
  the most part
- more formal presentations (web sites, papers), where the consumer is
  the broader world

and, to the extent this is true, i wonder if some markup issues have a
tension between those two uses.  e.g., for a formal document, i might be
willing to have "heavier" markup, to eliminate (...) false
positives/negatives.  (another example is in-line math markup, with
dollar signs or backslah-parens.)

cheers, Greg

Re: 'false' list item

[Tim Cross]

Greg Minshall  writes:

> Tim,
>
>> If a line starts with a number, period and space, but that line is
>> within a paragraph (i.e. no blank line above), then I don't think it
>> should be interpreted as an enumerated list item. If this is what the OP
>> is referring to, I would argue it is a bug. If it is a 'paragraph'
>> starting with a number, period and space, then being interpreted as a
>> list item would be 'normal'.
>
> i know this isn't precisely the discussion, but i use *unnumbered*
> lists, and i most often have no blank line separating the list from the
> paragraph above (or, indeed, below).  i could get used to a modified
> requirement, to reduce the false positives, but, would want heads up
> (or, maybe, sigh, an option?).
>

There is no plans to change anything as far as I know. What I wrote was
mainly to show why we have the situation and that any proposed solution
has its own drawbacks.

Bottom line, we cannot easily prevent the 'false' list item issue
without introducing either other issues or adding some additional syntax
to indicate list items, which defeats the 'plain' aspects which many
appreciate in org. Even the proposed 'solutions' still suffer from false
positives.

The question is probably, how can we 'flag' possible false positives in
a convenient manner. For example, it has been suggested that some form
of highlighting for list elements might make it easier to spot a false
element lurking in a paragraph of text. this could possibly be useful,
but I also worry about adding more font-locking and highlighting due to
the potential to degrade performance further. We already have people
running into issue with performance in large org files.



--
Tim Cross

Re: 'false' list item

[Greg Minshall]

Tim,

> If a line starts with a number, period and space, but that line is
> within a paragraph (i.e. no blank line above), then I don't think it
> should be interpreted as an enumerated list item. If this is what the OP
> is referring to, I would argue it is a bug. If it is a 'paragraph'
> starting with a number, period and space, then being interpreted as a
> list item would be 'normal'.

i know this isn't precisely the discussion, but i use *unnumbered*
lists, and i most often have no blank line separating the list from the
paragraph above (or, indeed, below).  i could get used to a modified
requirement, to reduce the false positives, but, would want heads up
(or, maybe, sigh, an option?).

cheers, Greg

Re: 'false' list item

[Juan Manuel Macías]

Tim Cross  writes:

> The issue is, if we decide this is a bug, can we fix it? Is it a bug or
> is it just an accepted limitation of the org document format and what we
> have to do is ensure either no line starts with a '1. ' or we use
> something like a unicode character so that it doesn't look like a
> number+period+space and therefore not a list item?

Interesting all the points you comment, and this paragraph is important.
I'm thinking that without a clear and explicit environment (a begin/end
marks), any possible solution will always have its exceptions. However,
IMHO, I would not find desirable two explicit marks environment in the
style of LaTeX, since this would go against the virtue of legibility and
lightness in Org. I don't know if Markdown (on what it has in common
with Org as a lightweight markup language) also has similar problems in
lists formatting, since I hardly use Markdown and am not aware of its
many implementations...

I still think that highlighting the items would at least be a preventive
solution for Org.

Best regards,

Juan Manuel 

Re: 'false' list item

[Tim Cross]

Samuel Wales  writes:

> perhaps if there is no blank line above or below, then it could be not a list.
>

Initially, that was my thought as well. Unfortunately, it isn't quite
that straight forward. Consider the list

 1. Item 1
 2. Item 2
 3. Item 3

The second item has no blank line above or below, but it is a list item.
With the blank line approach, the list would have to be

 1. item 1

 2. item 2

 3. item 3

which I doubt many would like. What I guess is really needed is a list
'environment' - once you see the first list item, following a blank
line, your in a list environment until the next blank line. However,
then, if you have list items which are multi-line items, you cannot have
a blank line between items as that would signal a new list environment.
Personally, if the items are longer, visually, I do like a blank line
between the items. So now either we need to add some sort of list
'marker' or we need something else, such as saying a list must be
indented further than the surrounding paragraphs i.e. if you had (using
loren ipsum for text)

-
Aliquam erat volutpat. Nunc eleifend leo vitae magna. In id erat non
orci commodo lobortis. Proin neque massa, cursus ut, gravida ut,
lobortis eget, lacus. Sed diam. Praesent fermentum tempor tellus. Nullam
tempus. Mauris ac felis vel velit tristique imperdiet. Donec at pede.
Etiam vel neque nec dui dignissim bibendum. Vivamus id enim. Phasellus
neque orci, porta a, aliquet quis, semper a, massa. Phasellus purus.
Pellentesque tristique imperdiet tortor. Nam euismod tellus id erat.

1. Mauris ac felis vel velit tristique imperdiet.

Pellentesque dapibus suscipit ligula. Donec posuere augue in quam. Etiam
vel tortor sodales tellus ultricies commodo. Suspendisse potenti. Aenean
in sem ac leo mollis blandit. Donec neque quam, dignissim in, mollis
nec, sagittis eu, wisi. Phasellus lacus. Etiam laoreet quam sed arcu.
Phasellus at dui in ligula mollis ultricies. Integer placerat tristique
nisl. Praesent augue. Fusce commodo. Vestibulum convallis, lorem a
tempus semper, dui dui euismod elit, vitae placerat urna tortor vitae
lacus. Nullam libero mauris, consequat quis, varius et, dictum id, arcu.
Mauris mollis tincidunt felis. Aliquam feugiat tellus ut neque. Nulla
facilisis, risus a rhoncus fermentum, tellus tellus lacinia purus, et
dictum nunc justo sit amet elit.
-

the 1. line is not a list, but

-
Aliquam erat volutpat. Nunc eleifend leo vitae magna. In id erat non
orci commodo lobortis. Proin neque massa, cursus ut, gravida ut,
lobortis eget, lacus. Sed diam. Praesent fermentum tempor tellus. Nullam
tempus. Mauris ac felis vel velit tristique imperdiet. Donec at pede.
Etiam vel neque nec dui dignissim bibendum. Vivamus id enim. Phasellus
neque orci, porta a, aliquet quis, semper a, massa. Phasellus purus.
Pellentesque tristique imperdiet tortor. Nam euismod tellus id erat.

1. Mauris ac felis vel velit tristique imperdiet.

Pellentesque dapibus suscipit ligula. Donec posuere augue in quam. Etiam
vel tortor sodales tellus ultricies commodo. Suspendisse potenti. Aenean
in sem ac leo mollis blandit. Donec neque quam, dignissim in, mollis
nec, sagittis eu, wisi. Phasellus lacus. Etiam laoreet quam sed arcu.
Phasellus at dui in ligula mollis ultricies. Integer placerat tristique
nisl. Praesent augue. Fusce commodo. Vestibulum convallis, lorem a
tempus semper, dui dui euismod elit, vitae placerat urna tortor vitae
lacus. Nullam libero mauris, consequat quis, varius et, dictum id, arcu.
Mauris mollis tincidunt felis. Aliquam feugiat tellus ut neque. Nulla
facilisis, risus a rhoncus fermentum, tellus tellus lacinia purus, et
dictum nunc justo sit amet elit.
---

the 1. is a list item i.e. being indented compared to surrounding text makes it
a list item. Any further lines indented to that level or more (for sub
lists) are also considered part of the list, until the next blank line.

The problem with this of course is that if you have a sub-heading which
only has a list of items, there is no surrounding text within that
section, so how would you identify it was a list?

The issue is, if we decide this is a bug, can we fix it? Is it a bug or
is it just an accepted limitation of the org document format and what we
have to do is ensure either no line starts with a '1. ' or we use
something like a unicode character so that it doesn't look like a
number+period+space and therefore not a list item?

--
Tim Cross

Re: 'false' list item

[Kyle Meyer]

Juan Manuel Macías writes:

> Kyle Meyer  writes:
>
>> It seems that your approach would do a good job of helping you catch
>> cases that you don't want to be treated as lists.  I'm not aware of any
>> related functionality in Org, so I don't think you're missing something
>> there.
>>
>> Once you know that there is a particular spot that you want to prevent
>> from being interpreted as a list, you could add a zero-width space in
>> front of it:
>>
>> (info "(org)Escape Character")
>>
>> I'm not sure if that's the sort of solution you're asking for, though.
>
> Thanks for your advice, Kyle. Adding the U+200B char. works fine to
> avoid false positives. Anyway, like Tim Cross says, that situation
> maybe should be considered as a bug. I think the ideal behavior would
> be for Org to consider a list only when there is a blank line above.

You can find some previous discussion of this longstanding and known
behavior at .

Re: 'false' list item

[Kyle Meyer]

Diego Zamboni writes:

> Juan Manuel,
>
> YMMV depending on your needs and habits, but another workaround for this
> problem would be to use visual-line-mode instead of filling paragraphs.

Note that filling paragraphs in Org should already guard against
inadvertently creating a list item.  (Some related discussion is at
.)

Re: 'false' list item

[Juan Manuel Macías]

Samuel Wales  writes:

> perhaps if there is no blank line above or below, then it could be not a list.

I think this solution would be fine, although plain paragraphs starting
with digit/letter + period + space would be false positives. Anyway,
some kind of highlighting in the items, a defined face as in
markdown-mode (markdown-list-face), would be also helpful.

Best regards,

Juan Manuel 

Re: 'false' list item

[Samuel Wales]

perhaps if there is no blank line above or below, then it could be not a list.


On 2/21/21, Diego Zamboni  wrote:
> Juan Manuel,
>
> YMMV depending on your needs and habits, but another workaround for this
> problem would be to use visual-line-mode instead of filling paragraphs.
>
> --Diego
>
>
>
>
> On Sun, Feb 21, 2021 at 8:34 PM Juan Manuel Macías 
> wrote:
>
>> Kyle Meyer  writes:
>>
>> > It seems that your approach would do a good job of helping you catch
>> > cases that you don't want to be treated as lists.  I'm not aware of any
>> > related functionality in Org, so I don't think you're missing something
>> > there.
>> >
>> > Once you know that there is a particular spot that you want to prevent
>> > from being interpreted as a list, you could add a zero-width space in
>> > front of it:
>> >
>> > (info "(org)Escape Character")
>> >
>> > I'm not sure if that's the sort of solution you're asking for, though.
>>
>> Thanks for your advice, Kyle. Adding the U+200B char. works fine to
>> avoid false positives. Anyway, like Tim Cross says, that situation
>> maybe should be considered as a bug. I think the ideal behavior would
>> be for Org to consider a list only when there is a blank line above.
>> But, well thought out, I am afraid that it would not prevent false
>> positives, as one may want perfectly write a list at the beginning of
>> the document, or start a plain paragraph with (for example) a digit + a
>> period + a space...
>>
>> Best regards,
>>
>> Juan Manuel
>>
>>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: 'false' list item

[Diego Zamboni]

Juan Manuel,

YMMV depending on your needs and habits, but another workaround for this
problem would be to use visual-line-mode instead of filling paragraphs.

--Diego




On Sun, Feb 21, 2021 at 8:34 PM Juan Manuel Macías 
wrote:

> Kyle Meyer  writes:
>
> > It seems that your approach would do a good job of helping you catch
> > cases that you don't want to be treated as lists.  I'm not aware of any
> > related functionality in Org, so I don't think you're missing something
> > there.
> >
> > Once you know that there is a particular spot that you want to prevent
> > from being interpreted as a list, you could add a zero-width space in
> > front of it:
> >
> > (info "(org)Escape Character")
> >
> > I'm not sure if that's the sort of solution you're asking for, though.
>
> Thanks for your advice, Kyle. Adding the U+200B char. works fine to
> avoid false positives. Anyway, like Tim Cross says, that situation
> maybe should be considered as a bug. I think the ideal behavior would
> be for Org to consider a list only when there is a blank line above.
> But, well thought out, I am afraid that it would not prevent false
> positives, as one may want perfectly write a list at the beginning of
> the document, or start a plain paragraph with (for example) a digit + a
> period + a space...
>
> Best regards,
>
> Juan Manuel
>
>

Re: 'false' list item

[Juan Manuel Macías]

Kyle Meyer  writes:

> It seems that your approach would do a good job of helping you catch
> cases that you don't want to be treated as lists.  I'm not aware of any
> related functionality in Org, so I don't think you're missing something
> there.
>
> Once you know that there is a particular spot that you want to prevent
> from being interpreted as a list, you could add a zero-width space in
> front of it:
>
> (info "(org)Escape Character")
>
> I'm not sure if that's the sort of solution you're asking for, though.

Thanks for your advice, Kyle. Adding the U+200B char. works fine to
avoid false positives. Anyway, like Tim Cross says, that situation
maybe should be considered as a bug. I think the ideal behavior would
be for Org to consider a list only when there is a blank line above.
But, well thought out, I am afraid that it would not prevent false
positives, as one may want perfectly write a list at the beginning of
the document, or start a plain paragraph with (for example) a digit + a
period + a space...

Best regards,

Juan Manuel 

Re: Citations with page numbers using helm-bibtex and org-ref

[Adam Sneller]

Thanks John!

I think you have just given me my next homework assignment for "Adam's list of 
things to noodle around with in eLisp" :)

Adam

> On 21 Feb 2021, at 17:40, John Kitchin  wrote:
>
> It seems like some ideas are getting mixed up in your description. A cite 
> link in org-ref is related to a bibtex entry in a bibtex file, not to an org 
> heading in an org-file. In other words in your example, I would expecta 
> bibtex entry with the key bradley1973es to exist in one of the default 
> bibliography files you use (or in the one you define in a bibliography link). 
> The notes are just for your purposes.
>
> the headings/links in your notes file will not show up in any completion 
> backend in org-ref for citation selection, as only the bibtex entries are 
> used to construct those.
>
>  If you are looking for a way to select one of those headings from your 
> notes, and then insert the appropriate link, you would have to use something 
> different than org-ref. there is not presently a way to map an annotated cite 
> link to the specific note. I am not even sure you can write a function that 
> does that, as the functions only take a key for looking up the note file, and 
> not the description too. It certainly is possible to write a new function 
> that would work on the link at point to do that, and to call it 
> interactively, or add it as an action though. You would still get the key to 
> open the note file, and then use the link description if it exists to somehow 
> search forward for the relevant heading or text, failing gracefully if you, 
> for example, make a cite to a page you did not make a note on.
>
> When it comes time to authoring a paper, I think the workflow is you would 
> have to open the notes you made, find the section you want to use in your 
> paper, and copy the link you put in your notes to your new document. There 
> are some variations you might consider, but none of them would really be 
> integrated into the org-ref completion mechanisms that are generated from the 
> bibtex entries.
>
> For example you  might store the link or parts in a property like this:
>
> * The Accelerator-Multiplier Model
>   :PROPERTIES:
>   :key:  bradley1973es
>   :page: p200
>   :cite: [[cite:bradley1973es][p200]]
>   :END:
>
>
> and then write a small function you use interactively to copy it, e.g.
>
> #+BEGIN_SRC emacs-lisp
> (defun get-link ()
>   (interactive)
>   (kill-new (org-entry-get (point) "cite")))
> #+END_SRC
>
> and you might bind that to a key if you use it a lot. Alternatively you might 
> put the key in file-level property, and only store the page, and use property 
> inheritance, to build the link. There are a lot of options to choose from. 
> But, simply copying and pasting a link might also be the simplest.
>
> It might be possible to use the org-store/insert-link machinery for this too, 
> but I have found that to be trickier than I thought it should be in the past.
>
> John
>
> ---
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu 
>
>
>
> On Sun, Feb 21, 2021 at 12:13 PM Adam Sneller  > wrote:
> Hi Bruce/John,
>
> Thanks for getting back to me. So I guess your notes file would look 
> something like this?
>
>
> #+TITLE: Bradley, J. (1973): Essential Mathematics For Economists
>
> * Dynamic models: the consumption function
> [[cite:bradley1973es][p164]]
>
> * Changes in Capital Stock
> [[cite:bradley1973es][p188]]
>
> * The Accelerator-Multiplier Model
> [[cite:bradley1973es][p200]]
>
>
> So when when it comes time to author your paper, if you run org-store-link on 
> any of these, the description gets stripped off the link, so that only 
> cite:bradley1973es is stored (which obviously defeats the purpose). And if 
> you copy the link over by hand, it maps back to the document bradley197es.org 
>  (not the actual note).
>
> Am I missing anything?
>
> Adam
>
>> On 21 Feb 2021, at 12:21, Bruce D'Arcus > > wrote:
>>
>> On Sat, Feb 20, 2021 at 10:31 PM Adam Sneller > > wrote:
>>
>>> I currently use org-ref and helm-bibtex to manage my database of academic 
>>> sources, with one notes file per source. A lot of my sources are books. So 
>>> note typically grow over time, as I add multiple headers (each pertaining 
>>> to a chapter or topic/note taken from that source).
>>>
>>> But now I want to produce a citation that references the page numbers where 
>>> I captured that note...
>>>
>>> What is the recommended way to handle this? Are you breaking notes into 
>>> individual files, each with their own @inbook citation?
>>
>> Generally speaking, referencing page numbers and sections of a cited
>> source is not handled by dedicated citations

Re: Citations with page numbers using helm-bibtex and org-ref

[John Kitchin]

It seems like some ideas are getting mixed up in your description. A cite
link in org-ref is related to a bibtex entry in a bibtex file, not to an
org heading in an org-file. In other words in your example, I would expecta
bibtex entry with the key bradley1973es to exist in one of the default
bibliography files you use (or in the one you define in a bibliography
link). The notes are just for your purposes.

the headings/links in your notes file will not show up in any completion
backend in org-ref for citation selection, as only the bibtex entries are
used to construct those.

 If you are looking for a way to select one of those headings from your
notes, and then insert the appropriate link, you would have to use
something different than org-ref. there is not presently a way to map an
annotated cite link to the specific note. I am not even sure you can write
a function that does that, as the functions only take a key for looking up
the note file, and not the description too. It certainly is possible to
write a new function that would work on the link at point to do that, and
to call it interactively, or add it as an action though. You would still
get the key to open the note file, and then use the link description if it
exists to somehow search forward for the relevant heading or text, failing
gracefully if you, for example, make a cite to a page you did not make a
note on.

When it comes time to authoring a paper, I think the workflow is you would
have to open the notes you made, find the section you want to use in your
paper, and copy the link you put in your notes to your new document. There
are some variations you might consider, but none of them would really be
integrated into the org-ref completion mechanisms that are generated from
the bibtex entries.

For example you  might store the link or parts in a property like this:

* The Accelerator-Multiplier Model
  :PROPERTIES:
  :key:  bradley1973es
  :page: p200
  :cite: [[cite:bradley1973es][p200]]
  :END:


and then write a small function you use interactively to copy it, e.g.

#+BEGIN_SRC emacs-lisp
(defun get-link ()
  (interactive)
  (kill-new (org-entry-get (point) "cite")))
#+END_SRC

and you might bind that to a key if you use it a lot. Alternatively you
might put the key in file-level property, and only store the page, and use
property inheritance, to build the link. There are a lot of options to
choose from. But, simply copying and pasting a link might also be the
simplest.

It might be possible to use the org-store/insert-link machinery for this
too, but I have found that to be trickier than I thought it should be in
the past.

John

---
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Feb 21, 2021 at 12:13 PM Adam Sneller  wrote:

> Hi Bruce/John,
>
> Thanks for getting back to me. So I guess your notes file would look
> something like this?
>
>
> #+TITLE: Bradley, J. (1973): Essential Mathematics For Economists
>
> * Dynamic models: the consumption function
> [[cite:bradley1973es][p164]]
>
> * Changes in Capital Stock
> [[cite:bradley1973es][p188]]
>
> * The Accelerator-Multiplier Model
> [[cite:bradley1973es][p200]]
>
>
> So when when it comes time to author your paper, if you run org-store-link
> on any of these, the description gets stripped off the link, so that only
> cite:bradley1973es is stored (which obviously defeats the purpose). And if
> you copy the link over by hand, it maps back to the document
> bradley197es.org (not the actual note).
>
> Am I missing anything?
>
> Adam
>
> On 21 Feb 2021, at 12:21, Bruce D'Arcus  wrote:
>
> On Sat, Feb 20, 2021 at 10:31 PM Adam Sneller 
> wrote:
>
> I currently use org-ref and helm-bibtex to manage my database of academic
> sources, with one notes file per source. A lot of my sources are books. So
> note typically grow over time, as I add multiple headers (each pertaining
> to a chapter or topic/note taken from that source).
>
> But now I want to produce a citation that references the page numbers
> where I captured that note...
>
> What is the recommended way to handle this? Are you breaking notes into
> individual files, each with their own @inbook citation?
>
>
> Generally speaking, referencing page numbers and sections of a cited
> source is not handled by dedicated citations, but rather by
> annotations on the containing citation (book etc.).
>
> So in the pandoc syntax, for example, [@book, p23].
>
> I do the same with notes, and just included the specific citation with
> the note if I need to maintain the specific source page.
>
> Bruce
>
>
>

Re: Citations with page numbers using helm-bibtex and org-ref

[Adam Sneller]

Hi Bruce/John,

Thanks for getting back to me. So I guess your notes file would look something 
like this?


#+TITLE: Bradley, J. (1973): Essential Mathematics For Economists

* Dynamic models: the consumption function
[[cite:bradley1973es][p164]]

* Changes in Capital Stock
[[cite:bradley1973es][p188]]

* The Accelerator-Multiplier Model
[[cite:bradley1973es][p200]]


So when when it comes time to author your paper, if you run org-store-link on 
any of these, the description gets stripped off the link, so that only 
cite:bradley1973es is stored (which obviously defeats the purpose). And if you 
copy the link over by hand, it maps back to the document bradley197es.org 
 (not the actual note).

Am I missing anything?

Adam

> On 21 Feb 2021, at 12:21, Bruce D'Arcus  wrote:
>
> On Sat, Feb 20, 2021 at 10:31 PM Adam Sneller  
> wrote:
>
>> I currently use org-ref and helm-bibtex to manage my database of academic 
>> sources, with one notes file per source. A lot of my sources are books. So 
>> note typically grow over time, as I add multiple headers (each pertaining to 
>> a chapter or topic/note taken from that source).
>>
>> But now I want to produce a citation that references the page numbers where 
>> I captured that note...
>>
>> What is the recommended way to handle this? Are you breaking notes into 
>> individual files, each with their own @inbook citation?
>
> Generally speaking, referencing page numbers and sections of a cited
> source is not handled by dedicated citations, but rather by
> annotations on the containing citation (book etc.).
>
> So in the pandoc syntax, for example, [@book, p23].
>
> I do the same with notes, and just included the specific citation with
> the note if I need to maintain the specific source page.
>
> Bruce
>

Re: state of the art in org-mode tables e.g. join, etc

[John Kitchin]

For fun, here is the sqlite equivalent of the Pandas example using the same
tables as before


** aggregation example

Examples from https://github.com/tbanel/orgaggregate


#+NAME: original
| Day   | Color | Level | Quantity |
|---+---+---+--|
| Monday| Red   |30 |   11 |
| Monday| Blue  |25 |3 |
| Tuesday   | Red   |51 |   12 |
| Tuesday   | Red   |45 |   15 |
| Tuesday   | Blue  |33 |   18 |
| Wednesday | Red   |27 |   23 |
| Wednesday | Blue  |12 |   16 |
| Wednesday | Blue  |15 |   15 |
| Thursday  | Red   |39 |   24 |
| Thursday  | Red   |41 |   29 |
| Thursday  | Red   |49 |   30 |
| Friday| Blue  | 7 |5 |
| Friday| Blue  | 6 |8 |
| Friday| Blue  |11 |9 |


#+begin_src sqlite :db ":memory:" :var orgtable=original :colnames yes
drop table if exists testtable;
create table testtable(Day str, Color str, Level int, Quantity int);
.mode csv testtable
.import $orgtable testtable
select Color, count(*) from testtable group by Color;
#+end_src

#+RESULTS:
| Color | count(*) |
|---+--|
| Blue  |7 |
| Red   |7 |

** join example

Example from https://github.com/tbanel/orgtbljoin

#+name: nutrition
| type | Fiber | Sugar | Protein | Carb |
|--+---+---+-+--|
| eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| onion|   1.3 |   4.4 | 1.3 |  9.0 |
| egg  | 0 |  18.3 |31.9 | 18.3 |
| rice |   0.2 | 0 | 1.5 | 16.0 |
| bread|   0.7 |   0.7 | 3.3 | 16.0 |
| orange   |   3.1 |  11.9 | 1.3 | 17.6 |
| banana   |   2.1 |   9.9 | 0.9 | 18.5 |
| tofu |   0.7 |   0.5 | 6.6 |  1.4 |
| nut  |   2.6 |   1.3 | 4.9 |  7.2 |
| corn |   4.7 |   1.8 | 2.8 | 21.3 |


#+name: recipe
| type | quty |
|--+--|
| onion|   70 |
| tomatoe  |  120 |
| eggplant |  300 |
| tofu |  100 |


#+begin_src sqlite :db ":memory:" :var nut=nutrition rec=recipe :colnames
yes
drop table if exists nutrition;
drop table if exists recipe;
create table nutrition(type str, Fiber float, Sugar float, Protein float,
Carb float);
create table recipe(type str, quty int);

.mode csv nutrition
.import $nut nutrition

.mode csv recipe
.import $rec recipe

select * from recipe, nutrition where recipe.type=nutrition.type;
#+end_src

#+RESULTS:
| type | quty | type | Fiber | Sugar | Protein | Carb |
|--+--+--+---+---+-+--|
| onion|   70 | onion|   1.3 |   4.4 | 1.3 |  9.0 |
| tomatoe  |  120 | tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| eggplant |  300 | eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tofu |  100 | tofu |   0.7 |   0.5 | 6.6 |  1.4 |


John

---
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Feb 21, 2021 at 10:03 AM John Kitchin 
wrote:

> Thanks Tim and Greg. I had mostly come to the same conclusions that it is
> probably best to outsource this. I worked out some examples from
> the orgtbljoin and orgaggregate packages with Pandas below, in case anyone
> is interested in seeing how it works. A key point is using the ":colnames
> no" header args to get the column names for Pandas. It seems like a pretty
> good approach.
>
> * org-mode tables with Pandas
> ** Aggregating from a table
>
> Examples from https://github.com/tbanel/orgaggregate
>
>
> #+NAME: original
> | Day   | Color | Level | Quantity |
> |---+---+---+--|
> | Monday| Red   |30 |   11 |
> | Monday| Blue  |25 |3 |
> | Tuesday   | Red   |51 |   12 |
> | Tuesday   | Red   |45 |   15 |
> | Tuesday   | Blue  |33 |   18 |
> | Wednesday | Red   |27 |   23 |
> | Wednesday | Blue  |12 |   16 |
> | Wednesday | Blue  |15 |   15 |
> | Thursday  | Red   |39 |   24 |
> | Thursday  | Red   |41 |   29 |
> | Thursday  | Red   |49 |   30 |
> | Friday| Blue  | 7 |5 |
> | Friday| Blue  | 6 |8 |
> | Friday| Blue  |11 |9 |
>
>
> #+BEGIN_SRC ipython :var data=original :colnames no
> import pandas as pd
>
> pd.DataFrame(data[1:], columns=data[0]).groupby('Color').size()
> #+END_SRC
>
> #+RESULTS:
> :results:
> # Out [1]:
> # text/plain
> : Color
> : Blue7
> : Red 7
> : dtype: int64
> :end:
>
> The categorical stuff here is just to get the days sorted the same way as
> the example. It is otherwise not needed. I feel there should be a more
> clever way to do this, but didn't think of it.
>
> #+BEGIN_SRC ipython :var data=original :colnames no
> df = pd.DataFrame(data[1:], columns=data[0])
> days = ['Monday', 'Tuesday', 'We

Re: org-agenda for a day different than today

[Alan Schmitt]

On 2021-02-21 01:23, Kyle Meyer  writes:

> Okay, thanks for checking.  Added a basic test and applied (3a522ad53).

Great, thanks a lot!

Best,

Alan


signature.asc
Description: PGP signature

Re: state of the art in org-mode tables e.g. join, etc

[John Kitchin]

Thanks Tim and Greg. I had mostly come to the same conclusions that it is
probably best to outsource this. I worked out some examples from
the orgtbljoin and orgaggregate packages with Pandas below, in case anyone
is interested in seeing how it works. A key point is using the ":colnames
no" header args to get the column names for Pandas. It seems like a pretty
good approach.

* org-mode tables with Pandas
** Aggregating from a table

Examples from https://github.com/tbanel/orgaggregate


#+NAME: original
| Day   | Color | Level | Quantity |
|---+---+---+--|
| Monday| Red   |30 |   11 |
| Monday| Blue  |25 |3 |
| Tuesday   | Red   |51 |   12 |
| Tuesday   | Red   |45 |   15 |
| Tuesday   | Blue  |33 |   18 |
| Wednesday | Red   |27 |   23 |
| Wednesday | Blue  |12 |   16 |
| Wednesday | Blue  |15 |   15 |
| Thursday  | Red   |39 |   24 |
| Thursday  | Red   |41 |   29 |
| Thursday  | Red   |49 |   30 |
| Friday| Blue  | 7 |5 |
| Friday| Blue  | 6 |8 |
| Friday| Blue  |11 |9 |


#+BEGIN_SRC ipython :var data=original :colnames no
import pandas as pd

pd.DataFrame(data[1:], columns=data[0]).groupby('Color').size()
#+END_SRC

#+RESULTS:
:results:
# Out [1]:
# text/plain
: Color
: Blue7
: Red 7
: dtype: int64
:end:

The categorical stuff here is just to get the days sorted the same way as
the example. It is otherwise not needed. I feel there should be a more
clever way to do this, but didn't think of it.

#+BEGIN_SRC ipython :var data=original :colnames no
df = pd.DataFrame(data[1:], columns=data[0])
days = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday',
'Sunday']
df['Day'] = pd.Categorical(df['Day'], categories=days, ordered=True)

(df
 .groupby('Day')
 .agg({'Level': 'mean',
   'Quantity': 'sum'})
 .sort_values('Day'))
#+END_SRC

#+RESULTS:
:results:
# Out [2]:
# text/plain
:Level  Quantity
: Day
: Monday  27.514
: Tuesday 43.045
: Wednesday   18.054
: Thursday43.083
: Friday   8.022
: Saturday NaN 0
: Sunday   NaN 0

[[file:/var/folders/3q/ht_2mtk52hl7ydxrcr87z2grgn/T/ob-ipython-htmlMnDA9a.html]]
:end:

** Joining tables

Example from https://github.com/tbanel/orgtbljoin

#+name: nutrition
| type | Fiber | Sugar | Protein | Carb |
|--+---+---+-+--|
| eggplant |   2.5 |   3.2 | 0.8 |  8.6 |
| tomatoe  |   0.6 |   2.1 | 0.8 |  3.4 |
| onion|   1.3 |   4.4 | 1.3 |  9.0 |
| egg  | 0 |  18.3 |31.9 | 18.3 |
| rice |   0.2 | 0 | 1.5 | 16.0 |
| bread|   0.7 |   0.7 | 3.3 | 16.0 |
| orange   |   3.1 |  11.9 | 1.3 | 17.6 |
| banana   |   2.1 |   9.9 | 0.9 | 18.5 |
| tofu |   0.7 |   0.5 | 6.6 |  1.4 |
| nut  |   2.6 |   1.3 | 4.9 |  7.2 |
| corn |   4.7 |   1.8 | 2.8 | 21.3 |


#+name: recipe
| type | quty |
|--+--|
| onion|   70 |
| tomatoe  |  120 |
| eggplant |  300 |
| tofu |  100 |


#+BEGIN_SRC ipython :var nut=nutrition recipe=recipe :colnames no
nutrition = pd.DataFrame(nut[1:], columns=nut[0])
rec = pd.DataFrame(recipe[1:], columns=recipe[0])

pd.merge(rec, nutrition, on='type')
#+END_SRC

#+RESULTS:
:results:
# Out [4]:
# text/plain
:type  quty  Fiber  Sugar  Protein  Carb
: 0 onion701.34.4  1.3   9.0
: 1   tomatoe   1200.62.1  0.8   3.4
: 2  eggplant   3002.53.2  0.8   8.6
: 3  tofu   1000.70.5  6.6   1.4
:end:


John

---
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Feb 21, 2021 at 1:54 AM Tim Cross  wrote:

>
> Greg Minshall  writes:
>
> > John,
> >
> >> Is there a state of the art in using org-tables as little databases
> >> with joins and stuff?
> >
> > i have to admit i do all that with an R code source block.  (the dplyr
> > package has the relevant joins, e.g. dplyr::inner_join().)  and, in R,
> > ":colnames yes" as a header argument gives you header lines on results.
> > (maybe that's ?now? for "all" languages?)
> >
>
> For really complex joins and ad hoc queries, I would do similar or put
> the data into sqlite. For more simple ones, I just define a table which
> uses table formulas to extract the values from the other tables - the
> downside being the tables need to have the same data ordering or the
> formulas need to be somewhat complex. Provided the tables have the same
> number of records in the same order, table formulas are usually fairly
> easy.
>
> I did think about writing some elisp functions to use in my table
> formulas to make things easier, but then decided I was just re-inventing
> and well defined datab

Re: 'false' list item

[Juan Manuel Macías]

Hi,

Tim Cross  writes:

> If a line starts with a number, period and space, but that line is
> within a paragraph (i.e. no blank line above), then I don't think it
> should be interpreted as an enumerated list item. If this is what the OP
> is referring to, I would argue it is a bug. If it is a 'paragraph'
> starting with a number, period and space, then being interpreted as a
> list item would be 'normal'.

The problem occurs on the lines within a paragraph (no blank line
above). I have uploaded this screenshot:

https://gnutas.juanmanuelmacias.com/images/false-list-item.webm

It also happens with a clean startup.

Best regards,

Juan Manuel

Re: Citations with page numbers using helm-bibtex and org-ref

[John Kitchin]

In org-ref you also handle this by annotations of the cite links, e.g.
[[cite:book-key][p23]] and then bibtex or biblatex does the formatting.

John

---
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Feb 21, 2021 at 7:22 AM Bruce D'Arcus  wrote:

> On Sat, Feb 20, 2021 at 10:31 PM Adam Sneller 
> wrote:
>
> > I currently use org-ref and helm-bibtex to manage my database of
> academic sources, with one notes file per source. A lot of my sources are
> books. So note typically grow over time, as I add multiple headers (each
> pertaining to a chapter or topic/note taken from that source).
> >
> > But now I want to produce a citation that references the page numbers
> where I captured that note...
> >
> > What is the recommended way to handle this? Are you breaking notes into
> individual files, each with their own @inbook citation?
>
> Generally speaking, referencing page numbers and sections of a cited
> source is not handled by dedicated citations, but rather by
> annotations on the containing citation (book etc.).
>
> So in the pandoc syntax, for example, [@book, p23].
>
> I do the same with notes, and just included the specific citation with
> the note if I need to maintain the specific source page.
>
> Bruce
>
>

Re: Bug: exporting a subtree to icalendar omits body of subtree [9.3 (release_9.3 @ /usr/share/emacs/27.1/lisp/org/)]

[Stephen Eglen]

> When exporting a subtree, the top-level outline becomes the title of the
> document, i.e., the document becomes morally equivalent to:
>
> #+title: test
> <2021-03-18 Thu 15:00-16:00>
> body of text
>
> There is indeed no event to export in this case. Only level 2 outline
> will become events. You may want to export region instead.

Thank you very much Nicolas for explaining; this was my mistake.

My solution for now is to use org-narrow-to-subtree and then export it.

Stephen

Re: Citations with page numbers using helm-bibtex and org-ref

[Bruce D'Arcus]

On Sat, Feb 20, 2021 at 10:31 PM Adam Sneller  wrote:

> I currently use org-ref and helm-bibtex to manage my database of academic 
> sources, with one notes file per source. A lot of my sources are books. So 
> note typically grow over time, as I add multiple headers (each pertaining to 
> a chapter or topic/note taken from that source).
>
> But now I want to produce a citation that references the page numbers where I 
> captured that note...
>
> What is the recommended way to handle this? Are you breaking notes into 
> individual files, each with their own @inbook citation?

Generally speaking, referencing page numbers and sections of a cited
source is not handled by dedicated citations, but rather by
annotations on the containing citation (book etc.).

So in the pandoc syntax, for example, [@book, p23].

I do the same with notes, and just included the specific citation with
the note if I need to maintain the specific source page.

Bruce