Re: [PATCH] Accept more :tangle-mode specification forms

2021-11-18 Thread Tom Gillespie
Hi Timothy,
The confusion with 755 and "755" could lead to security issues in
cases like 600 vs "600" vs #o600. The need to protect against the 600
case is fairly important, however I don't think there is anything we
can do about it, because someone might want to enter their modes as
base 10 integers.

If we were to prepend every integer with #o (or setting the radix to 8
when reading this particular field) before passing it to
org-babel-parse-header-arguments then it would be impossible to use
base 10 integers unless they were provided in the #10r600 form (Emacs
doesn't support #d600 notation).

I think the best bet is to change the radix for bare integers to 8
when reading that particular header, however I don't know how complex
that would be to implement.

If we don't want to change the radix to 8 then here are some suggestions.

If #o0600 already parses correctly, then I suggest we leave things as
is. Adding complexity just to drop the leading # seems wasteful.

We may want to warn or raise an error if someone uses a value such as
the base 10 integer 600 which does not map to the usual expected octac
codes so that they don't silently get bad file modes that could leave
files readable to the world.

Best,
Tom



Re: how to org-babel-detangle with nested noweb?

2021-10-18 Thread Tom Gillespie
Hi Edgar,
Degangling of nested noweb blocks tangled using
:comments noweb is broken at the moment. There are
some deep bugs that need to be worked out, and last
time I looked at the code I think my conclusion that it
was better to do a complete rewrite starting from a new
specification of the behavior along with some gnarly test
cases to ensure that everything works as expected.
Best!
Tom



Re: Org lint and named source blocks

2021-10-04 Thread Tom Gillespie
Thanks for the pointer! The actual point of contact seems to be
https://github.com/milisims/tree-sitter-org. Good to find another
group that is working on this. Best,
Tom



Re: Org lint and named source blocks

2021-10-04 Thread Tom Gillespie
> By the way, wouldn't it be better to use tree-sitter rather than
> something else for the format grammar?

Not really since we are going to need more than one implementation
using a parser generator to avoid baking implementation specific
details into the spec by accident. This is true for more than just
the grammar as well. The complexity of tokenization, parsing,
expanding, etc, for Org means that we are going to need multiple
implementations to nail the behavior for any formal spec.

That said, we definitely want a TS implementation at some point.
See https://github.com/tgbugs/laundry/issues/1 for a recent
discussion about ways forward.

The implementation I'm working on should translate to TS without
too much work since both brag and tree sitter describe LR variants.
There may be some subtle differences, but nothing fundamental.

The issue for me is that I don't have the bandwidth to get started
with a full tree sitter implementation, especially because it is going
to need a custom scanner, and because you're effectively on your
own when it comes to reconstructing the output of the AST into the
actual internal representation of an Org file. I also have no idea how
to deal with nested parsers in tree sitter. I have some ideas about
how it might be done, but nothing concrete (see the linked issue
for more on that).

Best,
Tom



Re: [PATCH] Don't fill displayed equations

2021-10-04 Thread Tom Gillespie
> Does anybody have any other thoughts?

>From time to time I encounter random patterns that I don't want to be
reformatted during a fill operation. Maybe a custom variable like
org-fill-paragraph-skip-regexp or similar that could be set by the user?
For Timothy's use case he would set it to the regexp provided in the
original patch? Not sure how much of the implementation in the patch
is dependent on that particular regexp, but a general solution that
could even be set per org file might be a very useful new feature.

Best!
Tom



Re: [PATCH] Don't fill displayed equations

2021-10-03 Thread Tom Gillespie
Some thoughts.

> Maybe you are right and Tom was actually assuming \begin{equation*}, not
> #+begin_export latex.

Correct. My bad on that one.

> Just as Timothy, I believe that \begin{equation*} is unnecessary verbose
> when \[ works *mostly* in a similar way.

\begin{equation*} is absolutely required if you want to be able to include
newlines because \[ and \begin are not similar at all as far as parsing
is concerned.

>From the spec: https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Environments
> CONTENTS can contain anything but the “\end{NAME}” string.
The spec is not completely accurate since latex environments can't
contain a new heading, but the point is that latex environments are
elements, whereas \[ \] is an object.

> If I understand correctly, making \[ \] available outside paragraph
> would mean that it becomes a new element (currently \[\] is a
> latex-fragment object).

Correct. Promoting \[ to an element would mean every \ in an org file
becomes a stop word. Also, Since full fledged latex environments
already exist to serve this purpose I find it hard to justify, especially
given that Org tries to give clear indication of when a block structure
is starting and ending.

> Isn't the whole point of the \[ ... \], \( ... \), $ ... $, $$ ... $$,
> and \begin{env} ... \end{env} and constructs in Org to be consistent
> with LaTeX?

For \begin and \end yes. For the others no. In general it would be to
make it possible to express things using latex-like syntax that would
otherwise require Org to come up with some new and different syntax.
These are values that may be translated to latex, but they exist inside
a larger syntax that is decidedly not latex, and thus they only have
meaningful translation to latex if they exist as well formed Org.

As a side note, the $ syntax is slated to be deprecated and removed.
https://orgmode.org/worg/dev/org-syntax.html#Entities_and_LaTeX_Fragments
> It would introduce incompatibilities with previous Org versions, but
> support for $...$ (and for symmetry, $$...$$) constructs ought to be removed.

> Indeed, it will be a breaking change.

I'm actually fairly certain that such a change should never be made
due to the recent changes in org link syntax. Specifically given how
\[ is used for escapes in links. https://orgmode.org/manual/Link-Format.html
This means that the only place you could reliably use \[ is at the start of a
new line preceded only by whitespace. However, if this were to happen then
pretty much every org document that uses \[ \] is at risk for being broken
because something that was once a single paragraph will now be multiple
paragraphs.

If you need multiline use \begin \end, that is what they are there for, and they
fit better with org's general extensible approach to blocks. I would dearly love
to be able to have a single shorthand for src blocks that worked inline and
standalone, but the complexity that it would induce is just not worth it. Same
thing for \[ \]. It seems simple until you get down to account for all the edge
that it would induce in the grammar.

Consider the case where you have something like

\[ something something

more content
more content [[www.example.com/\]oops][evil link]] \]

I've seen enough cases that are similar to this in the existing implementation
that have inconsistent behavior that I can safely say that this one would too.
Not to mention that I can think of at least 3 different cases that will all have
slightly different behavior that is inexplicable to users at best and
infuriating
at worst.

\[ a

b \]

\[
a
b
\]

a \[ b

c \] d

etc. There are plenty more variants that would all be subtly different depending
on the exact way such a thing were implemented.

In short. Just not worth it.



Re: [PATCH] Don't fill displayed equations

2021-10-02 Thread Tom Gillespie
Hi Timothy,

> │ \[
> │   not part of a paragraph
> │ \]

My point is that that parses first as a paragraph (check org-element-at-point).
\[ and \] would be meaningless if it did not first parse as a paragraph.

> I also don’t see how footnotes are analogous, as footnotes are placed in the
> middle of a line of text.

Inline footnotes [fn::
can span
multiple lines] but can't contain empty lines because the empty line ends the
paragraph that they are contained in.

> org-latex-preview :)

But surely #+begin_export latex works with org-latex-preview? If not then
that would be a feature request to org-latex-preview yes?

Best!
Tom



Re: Comments break up a paragraph when writing one-setence-per-line

2021-10-02 Thread Tom Gillespie
A general comment (heh) here. This is not a bug and not easily fixed.
Line comments are their own top level element distinct from
paragraphs. If you need something that fits in a paragraph you can use
@@comment:@@ at the start of a line.

I agree that it is annoying, but Org line comment syntax also only
works if it starts the line, so the behavior diverges from traditional
code comments. It may make sense to update the docs to call them "line
comments" instead of just comments.

One area where we could almost certainly do better is in how line
comments break up the flow of text. I'm not sure there will ultimately
be much we can do about it, but it is worth investigating.

Best,
Tom



Re: [PATCH] Don't fill displayed equations

2021-10-02 Thread Tom Gillespie
> do not see a reason for idiosyncrasy that markup intended to add LaTeX
> snippet that looks like exactly as LaTeX commands for this purpose and
> even actually preserved during export to LaTeX should have different
> semantics for Org parser.

The answer is that \[ \] can only occur inside paragraphs. The issues
here are exactly the same as the issues for inline footnotes. Org gives
us a bit more power, but not the full power because Org is Org, not
Latex. Making \[ \] available outside of a paragraph would be a massive
breaking change.

In Timothy's original example he is narrowly skirting the syntax to
allow that all to remain a single paragraph, but stick in a newline
anywhere and boom, no more paragraph, no more equation.

I guess one thing I'm missing/not understanding is when/why people
want to use \[ \] instead of full #+begin_export latex block?

Best,
Tom



Re: [PATCH] Accept more :tangle-mode specification forms

2021-10-01 Thread Tom Gillespie
> I'd like to understand these objections better. Aren't you overstating
what is at issue?

Yes, after hitting send I realized I overstated my position a bit.
In the meantime the comments in this thread are encouraging,
however I have finally figured out what I was really trying to say.

tl;dr file permission modes are not universal and should thus not
be part of the Org implementation, Org itself knows nothing about
files or permissions, it is the system that Org is running in/on.
Therefore, so long as we make it abundantly clear that the
value for :tangle-mode is not expected to be portable and that
it is always up to the user to ensure correct behavior, then we
are ok. I'm not happy about this conclusion from a security
perspective, but it isn't really worse than the situation we have
right now.


As many have pointed out, the grammar itself will not be affected.
However, other parts of the spec will. In general my objective is to
try to reduce the number of special cases that an org implementation
has to know about and delegate them to something else.

However in this case it is a bit tricky because of the security implications
and due to the fact that octal modes for file permissions are NOT universal
and should not be expected to be universal!

I actually think that my gut reaction was correct, but was expressed
in the wrong way.

Unix file modes are not universal and should thus not be encoded as
part of a portable document format. This means that it is up to the
user to know what representation is suitable.

Right now that representation is delegated to Emacs, because Emacs
handles file permissions for Org, and Emac's language for modes is
octal.

There are some octal modes that do not translate on Windows, and cannot
be correctly set. There will (hopefully) be some happy day in the future
where there is an operating system that will run Org babel where octal
file modes do not exist at all!

Therefore I suggest that we do not enshrine a particularly obscure way
of expressing file modes into Org itself. Right now Org is confined to
Emacs' representations, which in a sense protects Org from becoming
too ossified by bad designs of the past --- Emacs can keep all that
for us!

If we want a more user friendly syntax for this I would suggest that we do
something like what has been done for Org babel :results, i.e. like
:tangle-mode read write execute, unfortunately that does not compose
well at all with user, group, and other and becomes exceedingly verbose.


Final conclusion, after all that rambling, is that I'd actually be ok with
any of the solutions proposed, so long as it is clear that :tangle-mode
will always be implementation dependent, and may or may not be
meaningful depending on which operating system you are using.
Unfortunate for security, but I don't see any way around tha. The
best we could do for security would be for implementations to
test the file modes after tangling to ensure that they match,
which is more important I think.

That said, reducing the number of forms as Eric suggests would
be a happy medium.

Best!
Tom



Re: [PATCH] Accept more :tangle-mode specification forms

2021-09-30 Thread Tom Gillespie
I strongly oppose this patch. It adds far too much complexity to the
org grammar. Representation of numbers is an extremely nasty part of
nearly every language, and I suggest that org steer well clear of
trying to formalize this. With an eye to future portability I suggest
that no special cases be given to something as important for security
as tangle mode without very careful consideration. Emacs lisp closures
have clear semantics in Org and the number syntax is clear. If users
are concerned about the verbosity of (identity #o0600) they could go
with the sorter (or #o0600). Best,
Tom



Re: Empty headline titles unsupported: Bug?

2021-09-26 Thread Tom Gillespie
Hi Bastien,
I am strongly in favor of this change. It simplifies the grammar
significantly, and from my work on the laundry lexer and parser, I'm
99% certain that the current behavior is a bug that is the result of
gobbling the space after the stars in the headline. The correct
implementation peeks 1 char ahead for the space, and then starts
parsing again starting with the space. This is because tags MUST be
preceded by a space, so if you incorrectly gobble the space after the
stars then that space cannot be used as the start for tags. Best,
Tom



Re: [PATCH] lisp/ox-html.el: Restore org-svg class

2021-09-21 Thread Tom Gillespie
Bumping this patch for 9.5.

On Fri, Jul 30, 2021 at 8:59 PM Tom Gillespie  wrote:
>
> Hi,
>This patch restores the addition of class="org-svg" to svg images
> during html export. Best!
> Tom



Re: Org lint and named source blocks

2021-09-21 Thread Tom Gillespie
> Should we allow syntax like #+KEYWORD:value to be correct or do we
> require a whitespace/space after colon all the time?

The spec as written is ambiguous/silent on this issue. In my work on
laundry tokenizer and grammar I have found keyword syntax to be a
thorny issue, and I strongly suggest that for the time being we either
make no ruling on this or we state that the colon that ends the
keyword should be followed by a space as a precautionary measure.
The safe thing to do is to always require whitespace after the colon
because it guarantees correct interpretation.

Requiring whitespace after the colon simplifies the grammar, however
it means that you can't compact keyword lines, and it induces an
annoying failure mode where missing spaces are no longer keywords.

However, it is technically possible to make keywords work without the
whitespace, so long as there is at least one whitespace prior to the
next colon (but not contained in square brackets, e.g. #+key:lol[ a b
c ]:value is a well formed keyword under a slighly generalized
grammar). The problem is that we would like to make keyword syntax
fully closed, and I need a bit more time to get that worked out before
any definitive conclusions are drawn.

The complexity of the generalized keyword syntax can be seen here
https://github.com/tgbugs/laundry/blob/5a396bef98d9a3cd9ee929f21cd47612dd6cb1ac/laundry/lex-abbrev.rkt#L107-L249

Best,
Tom



Re: [org-cite] citations in property drawers?

2021-09-16 Thread Tom Gillespie
> I understand the problem, but the solution should not be: "let's pretend
> export does not exist".

>From my perspective any org object that is not in a section that
allows org objects could in principle be parsed as such, but it would
not be in the core of the grammar, and it also would have to parse to
something that did not trigger side effects related to export.

Allowing org objects to appear at arbitrary places in the grammar is
definitely not a good idea because in many senses they cannot actually
be those objects. Maybe the syntax could be the same, but they would
have to be "shadow objects" or something like that?

Best,
Tom



Re: [org-cite] citations in property drawers?

2021-09-15 Thread Tom Gillespie
> That would be a terrible idea. Exporters are not required to handle all
> data contained in properties drawers, so this may introduce errors,
> e.g., when trying to number citations.

I agree completely. You can't export something that has no anchor in
text that would be rendered. Maybe I misunderstood the original
question, because there is no way that a citation or footnote could be
exported from there, so I think in your conception text that follows
the format of the citations or footnotes isn't actually a citation or
footnote unless it exports as such.

Best,
Tom



Re: [org-cite] citations in property drawers?

2021-09-14 Thread Tom Gillespie
Hi Bruce,
I could certainly imagine using it, and I don't see any issue with
doing it from the point of view of the grammar. Footnotes can appear
in a property drawer without issue, though obviously they don't
export. One question though since I may have missed this in the other
threads is cite: allowed without the square brackets? Either way, org
element just parses the value to a string and it is up to any
consuming application to parse the node property further. Best!
Tom

On Thu, Sep 9, 2021 at 11:45 AM Bruce D'Arcus  wrote:
>
> Just bumping this.
>
> Another question about where to allow cite elements.
>
> On Fri, Aug 20, 2021 at 4:18 PM Bruce D'Arcus  wrote:
> >
> > So this is a tentative request/question; I'm not really sure the best
> > approach here.
> >
> > This is based on discussion with one of the org-roam-bibtex developers
> > about what the proper way to indicate an org-roam note is a
> > bibliographic note; e.g. a note about a bibliographic source.
> >
> > Traditionally in org-roam, that is in a property drawer; like:
> >
> > :ROAM_REFS: cite:wallace-wells2019
> >
> > That is using org-ref syntax there.
> >
> > So the obvious question is should one just put an org-cite citation
> > there to do the same thing?
> >
> > Right now, the answer is clearly no, since they aren't allowed in
> > property drawers.
> >
> > But perhaps they should be, just as any link can be?
> >
> > Except if they are, I recognize, they need to be treated as special
> > cases; e.g ignored for the purposes of export and such.
> >
> > WDYT?
> >
> > Bruce
>



Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

2021-08-10 Thread Tom Gillespie
In general I like John's suggestion. Org link syntax can be made to do
nearly anything because it is possible to bind link actions to
arbitrary elisp functions (I have used them to create buttons that run
source blocks for some of my non-technical colleagues). The grouping
of cross references under org-cite seems reasonable to me, and I would
love it if they could handle arbitrary references, e.g. to hypothesis
web annotation links or org-capture links.

Actually, having written this now, I think that both solutions have
their own use cases. Org cite is clearly about providing evidence for,
or a scholarly reference for something, and critically it can embed
some metadata about that reference in the document as a citation or
perhaps as an excerpt (and extension of what org-ref does now when the
cursor is over a reference?). Regular links do not provide any way to
embed metadata within the document, they are purely pointers.

That being said, it seems that there are a number of use cases where
org-ref links are simply internal document links that can point to an
element with a specific #+name: and no embedded information about the
target is needed. However, I think it would be a mistake to use up
equation/eq and table/tbl or figure/fig prefixes for references that
are internal to org, because it implicitly limits/collides with the
#+link: keyword.

Best,
Tom



Re: [Concept talk] Org-connector

2021-08-10 Thread Tom Gillespie
Hi Sébastien,
I think you are probably looking for org-sync which implements
exactly this functionality. You would need to write a new backend for
your particular ticketing system, but github, bit bucket, and redmine
backends already exist and can serve as an example. Best,
Tom

https://orgmode.org/worg/org-contrib/gsoc2012/student-projects/org-sync/tutorial/



Re: bug: Error handling in source blocks.

2021-08-10 Thread Tom Gillespie
I will also chime in here to say that managing output streams and
errors for babel is a major new feature that I am interested in. The
issue, as Tim points out, is that there is a lot of complexity lurking
here due to the fact that certain languages have fundamentally
different capabilities and ways of handling or not handling errors,
and of running code (on arbitrary hosts) in the first place.

What works for one will almost certainly not work for another. Take
for example ob-lisp where there is already built in error handling in
emacs itself. Compare that with python where someone would likely need
to implement a special PYTHONBREAKPOINT entrypoint or something like
that, if it were possible at all.

I have had a draft of a document on what I called "babel
regularization" for well over a year now, but it is not in a state
that would be productive to share due to the sheer number of ob-langs
and systems affected and the need to be able to clearly catalog and
articulate the diversity of existing behaviors.

If you dig through old conversations on this list you will find a
discussion of the default behavior for ob-shell :returns values vs
output as the default, we were barely able to agree on which
principles should be followed to make the decision. In that case we
were lucky that there was already a way for users to set their desired
behavior in their init file or in a setup file or in the file itself.
How to handle errors will be much more complex, in part because it
will touch on what ob-lang implementations are able to overwrite
and/or must provide in order to actually function. At the moment there
are practically no constraints.

Lots of work to do here, so grateful for a report on the variability
in the behavior of the existing system.

Best!
Tom



Re: [PATCH] Rename headline to heading

2021-08-08 Thread Tom Gillespie
Hi André,
Thanks for taking a first pass at this. I think that this patch is
difficult to review. Could you break it into two separate patches, one
for documentation (non-code, e.g. docstring and comment) changes and
one for code changes?  That way we could more easily see where we may
need to mitigate the kind of issues Maxim noticed. Best!
Tom



Re: Help requested: Support for basic Org mode support in tools outside of Emacs

2021-08-03 Thread Tom Gillespie
Hi Karl,
   Great initiative. For many of the things in the table you will
probably want to link to the underlying library For example for github
and gitlab there is https://github.com/wallyqs/org-ruby (which I have
been trying to find time to submit fixes to). I've linked a couple
relevant threads and repos. Best!
Tom

python https://github.com/novoid/Memacs
python https://github.com/karlicoss/orgparse
python https://github.com/bjonnh/PyOrgMode
racket https://github.com/tgbugs/laundry/tree/next
racket https://github.com/jeapostrophe/org-mode
racket https://github.com/antoineB/org-mode
See https://github.com/tgbugs/laundry/blob/next/laundry/cursed.org for
an org file that github fails to render
clojure https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf

https://orgmode.org/list/ca+g3_pobab1qx1zv8q9sjfh4khuhvmanxp3xo7__6eosdxk...@mail.gmail.com/
https://orgmode.org/list/ca+g3_pnj6pekqv+twfkwbd778xhw9wsfx+kjjhjsoreplhu...@mail.gmail.com/

On Tue, Aug 3, 2021 at 11:46 AM Greg Minshall  wrote:
>
> Karl,
>
> orgtbl-query is a script for querying tables in .org files.  it doesn't
> do any special text formatting.
>
> https://gitlab.com/minshall/orqtbl-query
>
> cheers, Greg
>



[PATCH] lisp/ox-html.el: Restore org-svg class

2021-07-30 Thread Tom Gillespie
Hi,
   This patch restores the addition of class="org-svg" to svg images
during html export. Best!
Tom
From 4363eec0913ccd0d05ecf3d6346208c62d3597f8 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Fri, 30 Jul 2021 20:53:07 -0700
Subject: [PATCH] lisp/ox-html.el: Restore org-svg class.

* lisp/ox-html.el (org-html--format-image): Restore org-svg class.
d96e8975791bd3b1a5f8fdb75609d73f134dc831 removed the org-svg class
which is necessary even when using  tags otherwise svg images
will render at absurdly large sizes.
---
 lisp/ox-html.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index bd6771a76..f25a9731e 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -1707,7 +1707,9 @@ a communication channel."
 (org-html-encode-plain-text
  (org-find-text-property-in-string 'org-latex-src source))
   (file-name-nondirectory source)))
- attributes))
+ (if (string= "svg" (file-name-extension source))
+ (org-combine-plists '(:class "org-svg") attributes '(:fallback nil))
+   attributes)))
info))
 
 (defun org-html--textarea-block (element)
-- 
2.31.1



Re: Headings and Headlines

2021-07-23 Thread Tom Gillespie
I enthusiastically support changing the documentation to use heading.
I use heading in my formal grammar because I have found there are more
ways that it can be modified and remain grammatically correct when
used in english sentences. The internal implementation in elisp still
refers to headlines, but changing the docs would be a good first step.
Best!
Tom



Re: [PATCH] ob-core: tangle check library of babel after current buffer

2021-07-17 Thread Tom Gillespie
Pinging on this to see if anyone can test it so that it can be merged.
Tom

On Wed, Jun 16, 2021 at 4:29 PM Tom Gillespie  wrote:
>
> Hi,
>This is a patch that fixes tangling behavior when a block has been
> ingested into the library of babel and then modified. Best!
> Tom



Re: A requires/provides approach to linking source code blocks

2021-07-13 Thread Tom Gillespie
We have been receiving many new feature suggestions and requests
coming in for org babel. I think that Tim's suggestion is the right
one. Nearly all of these need to be implemented as an extension first
and tested independently. Further, even if this is done, it should be
clear that there is zero expectation that such extensions will be
incorporated.

Once I wrap up the formal grammar for org, one of the next things I
plan to work on is a clear specification for org babel. This is
critical because so many of the suggestions that come in deal with
individuals' specific problems and thus fail to account for how such
features interact with existing features and how the newly proposed
feature would block some other features in the future, confuse users,
etc. Such suggestions also often fail to account for increased
complexity, nor have they been exposed to a sufficient number of
examples to reveal fundamental ambiguities in how they could be
interpreted. The issues with variable behavior between ob langs for
:pre :post :prologue :epilogue etc. are already enough to keep us busy
for quite some time.

With regard to this thread in particular, it is of some interest, but
there are fundamental issues, including the fact that certain
languages (e.g. racket) expect module code to exist somewhere on the
file system. There are ways around many of these issues, in fact there
are likely many ways around any individual issue, so org babel needs
to systematically consider the issues and provide a clear
specification, or at least a guide for how such cases should be
handled.

To give an example from one of my continual pain points: I start
writing python or racket in an org src block and then I want it to be
a library so that it can be reused by other code both inside and
outside the org file without having to resort to noweb.

What is the best way to handle this? I don't know. Right now I tangle
things and resort to awful hacks for the reuse-in-this-org-file case, but
I'm guessing there is a better generic solution which would allow _any_
org block to be exported as a library instead of nowebbed in.

Before jumping for any particular suggestion for how to handle this
we need to explore the diversity of cases that various ob langs
present, so that we can find a solution that will work for all of
them. After all, packaging code to a library for reuse is an
extremely common pattern that org babel should be able to
abstract, but it is a major undertaking, not just the addition of a
keyword here and there.

In short I suggest that we issue a general moratorium on new org babel
feature suggestions until we can stabilize what we already have and
provide a clear specification for correct behavior. Until we have that spec
we could encourage users to create extensions that implement those
features.

Best,
Tom


PS The other next thing that I am working on might be another way out
for this particular feature request. Namely, it is simplifying and
extending org keyword syntax so that new keywords (with options) and
associated keywords can be specified using keyword syntax within a
single org file. This would make it possible to get useful high level
keyword behavior in a single file without burdening the core
implementation with more special cases for associated keywords, and it
would allow users to write small elisp functions that could do some of
what is suggested here, all without need to add anything to the core
org implementation.



Re: [PATCH] Allow tangling to a list of files

2021-07-07 Thread Tom Gillespie
Reading over this with the new information about the use case, it
seems that using noweb to manage the many-to-many nature of a mapping
between blocks and files is a much better way to achieve the desired
result. In addition it is already supported and does not add more
complexity to an already complex part of org.

The one area that a noweb approach does not support is dynamic
construction of files at runtime on the basis of some information that
is only available at runtime, however that does not seem to be
important for this use case.

Therefore I suggest that the tangling behavior be left 1:1 block:file,
and if there is some desire to tangle to multiple files then noweb
should be used with multiple blocks. Obviously there is some
performance penalty here. Also this doesn't help with cases where we
want to tangle to hundreds of servers using tramp, but if that is the
use case then I would suggest that that operation not be hidden behind
:tangle. Instead tangle once and then use another elisp block write
all the files to their final destinations using tramp, ssh, or some
other means.

I personally have use cases for things like this, but even so I don't
think we wan't the :tangle parameter to be the way to do it. I would
suggest instead that if we want to enable a tangled file to be
automatically distributed to a number of different locations that we
provide a separate header argument so that the functionality is not
conflated with the tangle functionality. I don't have a good name for
it, but the objective seems to be something like :tangle-copy-to that
accepts a function returning zero or more paths, or a list of multiple
paths (I don't recall how/whether any of the babel args deal with
accepting multiple values).

Best,
Tom



Re: Large source block causes org-mode to be unusable

2021-06-21 Thread Tom Gillespie
> That said, I think keeping 2000 lines of source code inside an
> org src block is neither a standard use case nor a reasonable idea.

I would say that it certainly is a standard use case for people who
want to keep everything in a single file (e.g. to simplify
reproducibility and avoid the mess of trying to distribute multiple
files to non-technical users). #+INCLUDE is not a substitute if you
are going to be tangling files, breaks many workflows, and as a result
should rarely be recommended as a solution when src blocks are
involved. Org should definitely be able to handle this case because
there is no reason why performance should be any worse than having a
2000 line file in another buffer.

Org babel has many basic interactivity performance pitfalls that need
to be investigated. I personally have many workarounds for bad emacs
performance degradations related to code executing in the event loop
because I need to get on with the task at hand, but they need to be
fixed, not dismissed.



[PATCH] ob-core: tangle check library of babel after current buffer

2021-06-16 Thread Tom Gillespie
Hi,
   This is a patch that fixes tangling behavior when a block has been
ingested into the library of babel and then modified. Best!
Tom
From 22d0689257f977d09b013a143e899f788b45a039 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Mon, 14 Jun 2021 19:18:28 -0700
Subject: [PATCH] ob-core: tangle check library of babel after current buffer

* lisp/ob-core.el (org-babel-expand-noweb-references): Fix order when
searching for named babel blocks so that blocks in the current buffer
are always found first. This fixes a bug where stale versions of
blocks that have been ingested into the library of babel were being
preferentially tangled instead of newly modified versions from the
current buffer.
---
 lisp/ob-core.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 857e03e55..384c06c9a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2828,8 +2828,6 @@ block but are passed literally to the \"example-block\"."
 		 (setq cache nil)
 		 (let ((raw (org-babel-ref-resolve id)))
 		   (if (stringp raw) raw (format "%S" raw
-		;; Retrieve from the Library of Babel.
-		((nth 2 (assoc-string id org-babel-library-of-babel)))
 		;; Return the contents of headlines literally.
 		((org-babel-ref-goto-headline-id id)
 		 (org-babel-ref-headline-body))
@@ -2842,6 +2840,8 @@ block but are passed literally to the \"example-block\"."
 			  (not (org-in-commented-heading-p))
 			  (funcall expand-body
    (org-babel-get-src-block-info t))
+		;; Retrieve from the Library of Babel.
+		((nth 2 (assoc-string id org-babel-library-of-babel)))
 		;; All Noweb references were cached in a previous
 		;; run.  Extract the information from the cache.
 		((hash-table-p cache)
-- 
2.31.1



Re: colored src blocks question

2021-06-01 Thread Tom Gillespie
Hi John,
Are you perhaps missing the :extend t directive in the font spec? Best,
Tom

>


Re: A formal grammar for Org

2021-06-01 Thread Tom Gillespie
Hi Jakob,
Thank you for getting in touch. I had been meaning to after
someone pointed me to your repo in a reddit thread, but you beat me to
it. Replies in line. Best!
Tom

PS ccing this back to the list for the record.

On Tue, Jun 1, 2021 at 1:56 AM Jakob Schöttl  wrote:
>
> Hi Tom,
>
> I came to your post at the mailing list from here:
> https://github.com/gagbo/LuaOrgParser/issues/1
> Sorry, I don't know, how I can answer on the mailing list when I don't have 
> received the original mail.

No worries, I never managed to figure that out either so I just
subscribed. Maybe by matching the subject as you do here and ccing the
list (attempting it in this email to see what happens)?

> We have a pretty similar project, org-parser[1]. It's also written in a Lisp 
> dialect, Clojure, but it uses instaparse instead of brag as parser library.

https://github.com/tgbugs/laundry/tree/next#similar-projects I managed
to get it into my README as a reminder to myself to have a thorough
look at it, but have been occupied with other work since then.

> My idea was, to transform the formal grammar to a grammar.js for tree-sitter. 
> It would be so cool, if it could be generated from one formal specification.

Yes, that would be great. It would be a major step to have a couple of
grammars for org that can be used for stuff like this and compared to
each other, along with test cases that we can use to define correct
behavior.

One issue that I don't have a full understanding of at the
moment is how certain ambiguous forms will impact the ability to
transform directly into the tree sitter grammar.

The reason I mention
this is because I have had to move to a two phase parser in order to
deal with ambiguous parses.

Having not looked carefully at your
approach I don't know whether you have encountered similar issues. For
the tree sitter use case in particular I'm not entirely sure that the
ambiguity matters, but I haven't had a chance to look at it yet.

> Do you plan, in your parser, to do a transformation step from the raw parser 
> AST to a higher-level AST? E.g. the raw parser AST would parse a (:date  
> "2021-06-01") and the transformed AST would transform this to a higher-level 
> timestamp object.

Yes. I already do that to a certain extent in the expander
https://github.com/tgbugs/laundry/blob/next/laundry/expander.rkt (the
raw AST is hard to work with directly), but there will be more. I also
expect that I will add an intermediate step where the AST is
rearranged to account for aspects of org semantics that cannot be
captured by the context free part of the grammar.

After that step there are a number of potential conversions, one of which will
transform the AST into Racket structs, but I haven't made it quite
that far yet. That said, I think that in terms of defining a canonical
parse, I am aiming to do that in the transformed intermediate
s-expression representation because I think it will be easier to
define the correctness of certain user interactions on that form rather than
on the higher level object representation, even if the higher level
objects are ultimately used to actually implement that behavior.

> Do you have any automated tests for your parser?

Yes. See https://github.com/tgbugs/laundry/blob/next/laundry/test.rkt
you can run them from the working directory via =raco test laundry=.
I haven't fully specified the expected AST (and transforms) in most
cases because I'm still hammering out details. In some cases I do
specify the parse that I expect, e.g. for headings I specify when
tags are expected in cases where there might be some ambiguity. If you
are looking for edge cases there are a number that are not yet in the
automated tests but that are in
https://github.com/tgbugs/laundry/blob/next/laundry/cursed.org because
they hit on some cases of extreme ambiguity and internal inconsistency
in the elisp implementation or on weird behavior under user
interaction (I also have some other test cases that haven't been
committed to the repo yet).

It would be great to align the grammars and the behavior using a set
of common test cases.



Re: Empty headline titles unsupported: Bug?

2021-05-29 Thread Tom Gillespie
Hi Ihor,
Yes, happy to put my test cases into the org element cases and
visa versa. My long term plan is to come up with a set of test cases
that are unambiguous and potentially ambiguous so that we can
determine the expected behavior in those cases, so this is a great
first step. Best,
Tom



Re: Empty headline titles unsupported: Bug?

2021-05-29 Thread Tom Gillespie
Hi David,
Laundry produces a full s-expression representation of the org
parse tree (though it is still evolving). I haven't added a pass that
converts it to some Racket internal representation (probably will be
structs). If you get it installed and put #lang org at the top of an
org file you can use racket-mode to parse arbitrary org files, though
you may hit an error and will definitely encounter an
incomplete/incorrect parse since it is still a work in progress. Best,
Tom



Re: Empty headline titles unsupported: Bug?

2021-05-27 Thread Tom Gillespie
Hi all,
 Here is the 4th (or so) iteration of the grammar for titles that
I think deals with most of the issues in this thread along with a
bunch of nasty test cases. The previous attempts can be inspected in
the git history, but long story short, it is extremely hard to find a
grammar that follows the principle of least surprise and you have to
use the tokenizer to ensure that the tags pattern always parses as
such so that tags don't magically switch to being the title when you
remove the rest of the contents of the title. The final example
L1648-L1665 shows many of the things that should parse as tags and do
with this tokenizer/grammar combination. The key to dealing with the
ambiguity of empty title and tags vs something that looks like tags
but parses as a title (which is surprising) is to use the tokenizer to
greedily recognize tags at the end of the line. This ensures that the
tags pattern at the end of the line always parses as tags and doesn't
switch just because the title is empty. Happy to elaborate. Best,
Tom

https://github.com/tgbugs/laundry/blob/next/laundry/heading.rkt
https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/tokenizer.rkt#L98-L140

https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/test.rkt#L326-L367
https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/test.rkt#L400-L558
https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/test.rkt#L1298-L1369
https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/test.rkt#L1371-L1419
https://github.com/tgbugs/laundry/blob/971cf35683cd60156868c12b070c2dd9e19d8d06/laundry/test.rkt#L1648-L1665



bug#48676: Arbitrary code execution in Org export macros

2021-05-26 Thread Tom Gillespie
Hi Glenn,
 The definition for local variables doesn't cover things like org
macros, though the spirit of the policy is something worth keeping in
mind. Running M-x org-export-dispatch and hitting two keys means that
the user has to do something to trigger code execution, much like they
would have to intentionally accept certain risky local variables.

That said, the fact that many org operations can run arbitrary code is
definitely something that needs clearer documentation. It might make
sense to add a setting to detect closures that appear in org files to
ask for permission before running, but it likely should not be on by
default.

For a fairly extensive discussion of code execution in org see this
thread from Nov 2020.
https://orgmode.org/list/robi94$ma$1...@ciao.gmane.io/#t
Best,
Tom





Re: execute elisp link without prompt

2021-05-21 Thread Tom Gillespie
> In the end I've set as to nil as a local variable

If you want something a bit more secure you could use a function that
checks the block name ("some-block" in this example). Best!
Tom

(lambda (_lang _body)
   (not
(string= "some-block"
 (plist-get (cadr (org-element-at-point)) :name

#+begin_src elisp
(setq-local
 org-confirm-babel-evaluate
 (lambda (_lang _body)
   (not
(string= "some-block"
 (plist-get (cadr (org-element-at-point)) :name)
#+end_src

#+name: some-block
#+begin_src elisp
(message "yay!")
#+end_src

#+RESULTS: some-block
: yay!

#+name: some-other-block
#+begin_src elisp
(message "I ask to run")
#+end_src

#+RESULTS: some-other-block
: I ask to run



Re: URLs with brackets not recognised

2021-05-12 Thread Tom Gillespie
A quick fix is to percent encode the troublesome characters, but the
underlying issue is in org-link-any-re which is defined in
org-link-make-regexps which is what org uses to find the next link.
Some improvements might be possible for some of the edge cases there,
but a complete solution for bare urls is not possible due to conflicts
with native org syntax.

Org doesn't handle these cases well because in some cases org's own
syntax takes priority over url syntax at the moment adding bare url
syntax as part of org syntax is something that could be considered.
However, I would suggest against that because it will taint any org
parser in the future by forcing it to implement full url parsing at
arbitrary positions in paragraphs, which adds a lot of complexity. I
also suggest against it because org already has clear ways to
demarcate links using <> and [[]] which are guaranteed to behave
correctly even in cases where org syntax will always take priority.
For example with
https://en.wikipedia.org/wiki/Cathedral_Basilica_of_St._John_the_Baptist_[[Savannah,_Georgia]].

> It might be worthwhile to issue an warning each time a url is written in
> an org file without enclosing brackets < > or [[ ]].

Unfortunately warning on links without < > or [[ ]] will generate
countless annoying false positives for anyone who doesn't hit this
edge case. Maybe a separate function could be added to org lint that
would not run all the time?



Re: Multiple calc commands with orgbabel

2021-05-07 Thread Tom Gillespie
Hi Bastien,
Here's a patch to make it official. :)
Tom
From 3a61289e8fa4442f6d340138dcb67b950e980212 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Thu, 6 May 2021 23:52:21 -0700
Subject: [PATCH] lisp/ob-calc.el: Add Tom Gillespie as the maintainer

* lisp/ob-calc.el: Add Tom Gillespie as the maintainer.
---
 lisp/ob-calc.el | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lisp/ob-calc.el b/lisp/ob-calc.el
index 39ebce100..520f39145 100644
--- a/lisp/ob-calc.el
+++ b/lisp/ob-calc.el
@@ -3,6 +3,7 @@
 ;; Copyright (C) 2010-2021 Free Software Foundation, Inc.
 
 ;; Author: Eric Schulte
+;; Maintainer: Tom Gillespie 
 ;; Keywords: literate programming, reproducible research
 ;; Homepage: https://orgmode.org
 
-- 
2.26.3



Re: Multiple calc commands with orgbabel

2021-05-06 Thread Tom Gillespie
Hi Bastien,
Given the short length of the file, the fact that I now have a
fairly good idea of how it works, and the fact that I share a last
name with the original author of calc, I would be happy to. I'll hunt
down the steps you mentioned for becoming an ob- maintainer and ping
back when they are done. Best!
Tom



Re: Multiple calc commands with orgbabel

2021-05-05 Thread Tom Gillespie
Here is a quick and dirty implementation that more or less does what
you want (I think). The if t would probably need to be replaced by a
value that corresponded to an option that indicated that ob-calc
should resolve all expressions on the stack. This isn't really an
issue of return value, it is due to the fact that ob-calc makes
stateful modifications to calc. If you want a stateless (idempotent?)
ob-calc block you would need to do something like this as well, and
then you would need an option to discard the additional values instead
of retruning them as I do here. Best!
Tom
diff --git a/lisp/ob-calc.el b/lisp/ob-calc.el
index 39ebce100..e2102feca 100644
--- a/lisp/ob-calc.el
+++ b/lisp/ob-calc.el
@@ -48,6 +48,7 @@
   "Execute a block of calc code with Babel."
   (unless (get-buffer "*Calculator*")
 (save-window-excursion (calc) (calc-quit)))
+  (let ((unpopped 0))
   (let* ((vars (org-babel--get-vars params))
 	 (org--var-syms (mapcar #'car vars))
 	 (var-names (mapcar #'symbol-name org--var-syms)))
@@ -58,12 +59,14 @@
  vars)
 (mapc
  (lambda (line)
+   (setq unpopped (1+ unpopped)) ; ICK
(when (> (length line) 0)
 	 (cond
 	  ;; simple variable name
 	  ((member line var-names) (calc-recall (intern line)))
 	  ;; stack operation
 	  ((string= "'" (substring line 0 1))
+   (setq unpopped (- unpopped 2))
 	   (funcall (lookup-key calc-mode-map (substring line 1)) nil))
 	  ;; complex expression
 	  (t
@@ -89,9 +92,17 @@
 	 (split-string (org-babel-expand-body:calc body params) "[\n\r]"
   (save-excursion
 (with-current-buffer (get-buffer "*Calculator*")
-  (prog1
-(calc-eval (calc-top 1))
-(calc-pop 1)
+  (if t
+  (let ((out
+ (cl-loop for i from 1 to unpopped
+  do (message "%S %S" unpopped calc-stack)
+  collect (calc-eval (calc-top 1))
+  do (calc-pop 1
+(message "%S" out)
+(mapcar #'list (reverse out)))
+(prog1
+(calc-eval (calc-top 1))
+(calc-pop 1)))
 
 (defun org-babel-calc-maybe-resolve-var (el)
   (if (consp el)


Re: Multiple calc commands with orgbabel

2021-05-05 Thread Tom Gillespie
Looking at ob-calc there is a call to calc-push-list. Knowing the
length of that list (i.e. the number of arguments) it should be
possible to inspect calc-stack to retrieve the other values on the
stack from the current block. You can see this if you run M-:
calc-stack. This would probably need a specialized result type if it
were implemented. Best,
Tom

On Wed, May 5, 2021 at 8:33 AM  wrote:
>
>
> Example
>
> (require 'ob-calc)
>  (org-babel-do-load-languages
>  'org-babel-load-languages
>  '( (calc . t) )
>
>  calc.org 
>
> # To execute, place cursor point on a line, then hit "C-c * u" hard with no 
> harm.
>
> #+name: Simplifying Formulas
> #+begin_src calc
>
> simplify((x + y) (x + y)) =>
>
> simplify(a x^2 b / (c x^3 d)) =>
>
> simplify((4 x + 6) / (8 x)) =>
>
> simplify((1 + 2 i) (3 + 4 i)) =>
>
> simplify(5 + i^2 + i - 8 i) =>
>
> simplify((1, 2) + (3, 4)) =>
>
> simplify((1, 2) (3, 4)) =>
>
> #+end_src
>
>
>
> Sent: Thursday, May 06, 2021 at 3:11 AM
> From: "Matt Price" 
> To: "Org Mode List" 
> Cc: pie...@caramail.com
> Subject: Re: Multiple calc commands with orgbabel
> Can you explain how you get calc embedded mode working in org? I have never 
> used it and it sounds interesting, but I don't understand what hte delimiters 
> are.
>
> On Wed, May 5, 2021 at 2:35 AM Eric S Fraga  wrote:
>>
>> On Wednesday,  5 May 2021 at 07:46, pie...@caramail.com wrote:
>> > Have been trying to execute multiple calc commands, but when I
>> > evaluate the calc expressions, I get just one result.
>>
>> ob-calc returns the top element of the stack when finished and this will
>> be the result of the last operation in the src block.  I don't think
>> there's any way around this.
>>
>> I use embedded Calc for this reason.  You could rewrite your equations
>> as simple lines (separated by empty lines from the surroundings) and
>> evaluate each in turn with "C-x * u":
>>
>> fsolve(x 2 + x = 4, x) => x = 1.333
>>
>> fsolve([x + y = a, x - y = b], [x, y]) => [x = a + (b - a) / 2, y = (a - b) 
>> / 2]
>>
>> I added the "=>" at the end of each expression so that the result is
>> shown to the right instead of replacing the expression itself (default
>> embedded Calc behaviour).
>>
>> --
>> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd
>>



Re: About multilingual documents

2021-05-03 Thread Tom Gillespie
I like Aleksandar's solution quite a bit because it also works inline
e.g. as src_org[:lang de]{Meine deutsch ist zher schlect!}. In
principle this means that you could leverage the org-babel and org-src
buffer system to get flyspell results in that language in line as well
(though I don't think transporting overlays into the original buffer
has been implemented). Best!
Tom



Re: <> and ?font-lock? fly-check, ...

2021-05-03 Thread Tom Gillespie
Hi Greg,
   I just checked and it induces a syntax error, which I did not know,
but turns out to be quite useful because it means that an untangled or
incorrectly tangled file will fail to run beyond that point. Best!
Tom

On Sun, May 2, 2021 at 9:11 PM Greg Minshall  wrote:
>
> Tom, that is quite devious, actually.  thank you very much!  do you
> know, by the way, what flycheck and/or the shell make the "<<&"
> construct out to be?  cheers, Greg
>



Re: [PATCH] Fontification for inline src blocks

2021-05-02 Thread Tom Gillespie
Hi Timothy,
Another thought about this. In some languages (e.g. python) blocks
require an explicit return by default. It would be nice to be able to
set header arguments in the property drawer separately for inline
source blocks in such cases.

src_python[:prologue "x = (" :epilogue ")\nreturn x"]{1 + 2} {{{results(=3=)}}}

A quick review of ob-core and a check of the behavior suggests that
there is a concept of inline-header-args, but only for default
arguments, and that :inline-header-args:python: does not work.

Extending the concept so that inline blocks can have headers set via
property drawers separate from regular blocks seems important.
Especially because inline blocks can accidentally inherit header-args
that are incompatible (e.g. :results list). I don't think these
patches depend on that though, so probably better to deal with that
separately.

Best,
Tom



Re: [PATCH] Fontification for inline src blocks

2021-05-02 Thread Tom Gillespie
> I see. I imagine the expected behaviour of such a function would be to
> toggle org-inline-src-prettify-results and redisplay?

Yeah, see org-toggle-link-display for inspiration I think.

;;;###autoload
(defun org-toggle-link-display ()
  "Toggle the literal or descriptive display of links."
  (interactive)
  (if org-link-descriptive (remove-from-invisibility-spec '(org-link))
(add-to-invisibility-spec '(org-link)))
  (org-restart-font-lock)
  (setq org-link-descriptive (not org-link-descriptive)))



Re: [PATCH] Fontification for inline src blocks

2021-05-02 Thread Tom Gillespie
Hi Timothy,
   It seems to work more or less as expected. A few comments below. Best,
Tom

1. I think there needs to be a function to toggle
org-inline-src-prettify-results as there is e.g. for hyperlinks. I was
quite confused by the prettified results.

2. I'm also not sure that this approach to prettify is a good idea.
There are issues with unexpected killing/yanking and basic navigation
behavior of the prettified text which seem worse than the already
troublesome issues with hyperlinks. I'm not sure we can do anything
about this though?

3. I'm not sure about the default choice for prettified delimiters. I
see there is already a way to customize the delimiters by providing a
cons. I think a default value of '("" . "") might be a better choice
since ⟨ and ⟩ being hardcoded seems like it introduces completely
alien characters. Going with empty strings also seems consistent with
the behavior for hyperlinks.

4. There is an interaction with rainbow delimiters that there isn't an
easy solution for. I wish there was a syntax type that was "this is a
paren for electric pair mode but not for font locking."

5. I'm not sure that the faces selected for src_ and lang are the
right ones. Is there any issue with adding new faces specifically for
those rather than reusing existing faces? I thought that matching the
font locking of #+begin_src lines might make sense, but then I
realized that that doesn't make sense because that is for blocks more
generally.



Re: [POLL] Setting `org-adapt-indentation' to nil by default?

2021-05-02 Thread Tom Gillespie
Hi Nicolas,
   Sorry, I did not mean to imply that such things were not possible
currently. I was writing in the context of how to specify the current
behavior formally. As you point out they absolutely are possible. More
replies in line. Best,
Tom

> This is inaccurate.
>
> The following is a perfectly valid list.
>
> --8<---cut here---start->8---
>   1. foo
>
>  #+begin_src emacs-lisp
> (+ 1 1)
>  #+end_src
>
>   2. bar
> --8<---cut here---end--->8---

Yes. My question is about how to deal with cases like

--8<---cut here---start->8---
 1. foo

#+begin_src emacs-lisp
  (+ 1 1)
#+end_src

 2. bar
--8<---cut here---end--->8---


> Source blocks for languages that have significant whitespace should use
> the -i flag.

My known issues with switches aside, the misaligned cases are the ones
that I worry about, and I don't think being able to flag a block as
being indentation sensitive helps resolve the potential ambiguity
there.

> What makes you think this is not the case?

Sorry, my wording is unclear here. I was not talking about the current
implementation which can and does do this, but instead about how to
formally specify what should be done in such cases.



Re: [Feature request] String escaped noweb expansion

2021-05-02 Thread Tom Gillespie
Hi Sebastien,
I have encountered issues with this before when trying to noweb code
into a string that was code to be sent via ssh. I ended up switching
to use typeset -f in bash in most cases now, but that is not possible
for other languages. Some languages also have enough different types
of syntax for strings that they can work around such cases, but again,
not all do.

One potential issue with this suggestion is how it would interact with
multi-line blocks, because you can't have anything on the same
starting line as the noweb expressions since it will be repeated in
front of every subsequent line.

This would also require each org-babel lang implementation to provide
a method for correctly string-escaping the nowebbed values (in some
cases e.g. shell this is decidedly non-trivial).

With all of these things in mind, I would thus suggest not trying to
overload the noweb operator for this purpose. Having a string escaped
equivalent would be nice, but because it requires more than just a
simple copy/paste into the buffer, it seems like it probably needs
separate notation. Best,
Tom



Re: <> and ?font-lock? fly-check, ...

2021-05-02 Thread Tom Gillespie
Hi Greg,
A slightly different suggestion that doesn't break other org
processors (which might not allow users to change
org-babel-noweb-wrap- values) is to prefix the names of the blocks
with & (e.g. <<>>) as I do in multiple places in
https://github.com/tgbugs/pyontutils/blob/master/docs/release.org#build-release.
I found this solution while fighting with the font-locking behavior in
shell blocks. Best!
Tom



Re: [POLL] Setting `org-adapt-indentation' to nil by default?

2021-05-02 Thread Tom Gillespie
Hi Bastien,
Strong +1 here. Users can get the same visual effect without
materializing the whitespace into the file.

Materializing the whitespace causes many potential issues with source
blocks for languages that have significant whitespace, issues with
#+begin_src and #+end_src having different levels of indentation
(still an issue if you want a block in a plain list), weirdness with
noweb, obligatory two pass parsing to get the spacing correct in
paragraphs, etc.

There are many cases where adapting indentation requires the
specification of extremely detailed heuristics that must be followed
exactly in order to get at least a consistent parse of a source block.
The only sane way forward for a language specification would be to
avoid that leading whitespace or avoid trying to specify the
interpretation of source blocks in contexts with leading whitespace
(src blocks in plain lists may come back to haunt us here).

Setting org-adapt-indentation to nil by default would be a major step
toward resolving these issues and frankly I couldn't ask for more.

Best!
Tom

PS I have included some notes on the worg/dev/org-syntax.org
file that I wrote while working on the formal grammar. I would
qualify what I wrote slightly to state that users could in principle
have leading whitespace before source blocks but that the behavior of
org in such cases would be left unspecified in the not quite nasal
demons sense, but that it might be better to have the behavior
described below with a note that no attempt to deal with correctly
preserving leading whitespace is required, user beware. A final
aside: maybe plain lists could have the #+begin_ and #+end_
lines indented to the level of the plain list but maybe not the body?
-

Eliminate leading whitespace in the canonical representation.

There are other ways that make it possible to have the indentation
visibility without adding massive complexity to the implementation.

The existing implementation can continue to support it, but any other
implementation SHALL convert indented sections to the canonical form
where there is NO leading whitespace. This eliminates the problem of
significant whitespace for everything except plain lists.

Users will need a migration path and this will require extensive
testing to make sure that the tooling catches as many of the issues as
possible. However, the benefits in the long run are vastly reduced
complexity without all the risks of accidentally botching an indent
somewhere.



Re: [PATCH] ob-tangle.el: Speed up tangling

2021-04-20 Thread Tom Gillespie
Hi Sébastien,
The temp -> rename approach is good, but you should probably use
make-temp-file to create the file to reduce the risk of
collisions/race conditions. For example as (make-temp-file (concat
file-name ".tangling")).

I think that the location of condition-case is ok, but I wonder what
would happen if something were to fail before entering that? I think
that only a subset of the files would be tangled, but they would all
have their correct modes, so I think that that is ok.

I also think that the message to the user should probably not be
changed right now. While it might can be useful for debug, if someone
is tangling to a large number of files then the filenames/paths are
going to flood messages, so I would leave it out of this patch, and
possibly submit it as another patch for a separate discussion.

Best!
Tom



Re: Concerns about community contributor support

2021-04-20 Thread Tom Gillespie
Hi Tim, David, and Gustav,
I am fairly certain that with only a few exceptions it is possible
to specify a context free grammar for org syntax, followed by a second
pass that deals specifically with markup and a few other forms,
notably the reassembly of things like plain lists. The fact that this
is possible because most org constructs are line oriented.

Just a note that the linked parser.rkt [0] is indeed a BNF describing org
syntax in the same style as a bison/yacc grammar. One of the reasons
why I set out to work on this was precisely so that there could be a
reference that could be consulted by the community when questions
about extended org come up.

There are proposals for new syntax that appear on this list with
terrifying frequency, and they are routinely shot down or simply
ignored for good reason, however it is hard to communicate that to
enthusiastic contributors who have an immediate use case that they
want to solve and share and are unlikely to be aware of side effects.
Having a grammar where such issues can be tested empirically should
provide a significant safeguard while also making it easier for
contributors to play with the grammar and see the issues.

In all my work on the grammar I have found maybe 2 or 3 places where
the grammar could be "extended" but it isn't so much extended as it is
regularized, where some parts of org already parse a more complex
grammar while other very similar parts choose not to. Overall the cost
of not parsing certain forms in certain situations adds complexity
rather than reducing it.

The situation for contribution is also further complicated by the fact
that the elisp implementation of org mode is internally inconsistent
in its behavior with regard to the syntax, so great care has to be
taken if someone tries to make and argument based on the behavior of
one component.

All this to say that the need for a conservative approach to changes
and extensions combined with the internally inconsistent behavior of
different parts of the elisp implementation means that the
introduction of new features is extremely difficult because it is hard
to predict the consequences on other parts of org.

Overcoming this is why I started working on the grammar, because
in the absence of a formal spec for what org should do, it is very hard
to make changes to what it is currently doing without having nasty
side effects.

Best!
Tom

0. https://github.com/tgbugs/laundry/blob/next/laundry/parser.rkt note
the upcoming path change (which I will note in the original thread when
it happens).

PS I'm planning to reply to the main thread as well. My short take is
finding a dedicated and responsive maintainer that can take over from
Bastien is a high priority. The only other thing that might help is to
have some way to track outstanding and closed patches, issues, etc.
that is more accessible than trolling through years worth of posts on
this mailing list, but that is a can of worms that has already been
shot down multiple times.



Re: [Patch] to correctly sort the items with emphasis marks in a list

2021-04-19 Thread Tom Gillespie
Hi Greg,
seq cannot be used because it is not available in older versions
of emacs that org still supports. When support for those older
versions is dropped then seq could be used. Best,
Tom



Re: [PATCH] ob-tangle.el: Speed up tangling

2021-04-18 Thread Tom Gillespie
Hi Sébastien,
   Some comments while looking over this (will report back when I have
tested it out as well). This is a section of the ob export
functionality that I have been looking for on and off for quite a
while because it is responsible for some bad and insecure behavior. I
think that some of your changes may have fixed/improved this as a side
effect. I don't know whether it is worth doing anything about the
issues in this patch, but since we are here, I think they are worth
mentioning. All of the issues that I'm aware of are related to what
happens if tangling fails part way through the process. First, your
patch already fixes a major issue which is that the modes of all files
would not be set if any one of them failed to tangle. Next, during the
process the existing file is deleted prior to tangling, which means
that it cannot be restored if tangling fails, it would be better if
the old file was moved to a temporary location and then deleted on
success or replaced on failure. This likely requires wrapping the bits
that can fail in unwind-protect and restoring on failure or fully
deleting at the end of success. The next issue is that setting the
tangle mode should happen before the file is written, an empty file
should be created, the mode should then be set, the contents of the
file should be written only after the mode has been set. This involves
a bit of reordering of operations in lines 124-126 of your patch. This
ordering of opertions prevents security issues related to race
conditions and potential errors being evoked during write-region
(though again, your changes already make the tangling code much more
secure by setting the modes on each file immediately after writing
instead of how it works currently where if any other block encounters
an error then no modes were set). Best!
Tom

On Sun, Apr 18, 2021 at 12:23 AM Sébastien Miquel
 wrote:
>
> Hi,
>
> The attached patch modifies the ~org-babel-tangle~ function to avoid a
> quadratic behavior in the number of blocks tangled to a single file.
>
> Tangling an org buffer with 200 blocks to 5 different files yields a
> 25 % speedup.
>
>
> * lisp/ob-tangle.el (org-babel-tangle-collect-blocks): Group
> collected blocks by tangled file name.
> (org-babel-tangle): Avoid quadratic behavior in number of blocks.
>
> --
> Sébastien Miquel



Re: Bug: inconsistent escaping of coderef regexp

2021-04-07 Thread Tom Gillespie
Hi Nicolas,
I've included the simplest patch I could come up with for the
divergence in behavior between org-babel-tangle-single-file and
org-link-search. I think there are two new threads that I need to
create. One is related to how to make it possible to specify what
should be removed along with the coderef (i.e. coderef prefix), the
other is the addition of header arguments that provide the same
functionality as switches. Best,
Tom

> This is already conflating the two. I'd like to solve the issue at hand
> without having header args interfere at all.
>
> This can happen later, after a discussion on the ML.

Ok. I've included the simplest version of the fix, which is to use
org-src-coderef-regexp in org-babel-tangle-single-file.

> Would you mind answering my questions first? I still don't follow you
> about the coderef prefix/regexp.

https://code.orgmode.org/bzg/org-mode/src/2d78ea57cfad1ddc3e993c949daf117b76315170/lisp/org-src.el#L882

That line defines a hardcoded regular expression for matching
coderefs. The codref prefix is the first =[ \t]*= and the coderef
regexp is the equivalent to the fully formatted version of that format
string. Neither of those can currently be specified by the user. The
user should not be able to specify the coderef regexp due to the fact
that it is too easy to specify a regexp that will not work correctly
and because the format string is needed to make org-link-search work
for named coderefs (otherwise you wind up trying to replace .+ in the
coderef regexp which is a nightmare). The coderef prefix is something
that should probably be configurable by the user so that empty
comments are not left in the file. I also looked into detecting the
comment character for the language in question, but that is
significantly more difficult even using (with-temp-buffer (funcall
lang-mode) comment-start) because not all languages have sane comment
start values and comment-start is not complete, so we would need a way
to manually specify what to exclude anyway.
From c30913da6b1c8d6be3670a59ae867df019505af3 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Wed, 7 Apr 2021 12:29:01 -0700
Subject: [PATCH] lisp/ob-tangle.el: Fix coderef removal during tangling

* lisp/ob-tangle.el (orb-babel-tangle-single-block): Regularize
behavior when removing coderefs during tangling. This fixes an issue
where trailing whitespace would be retained when coderefs were removed
for tangling.
---
 lisp/ob-tangle.el | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index aa0373ab8..4c0c3132d 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -414,9 +414,8 @@ non-nil, return the full association list to be used by
 	 (src-lang (nth 0 info))
 	 (params (nth 2 info))
 	 (extra (nth 3 info))
-	 (cref-fmt (or (and (string-match "-l \"\\(.+\\)\"" extra)
-			(match-string 1 extra))
-		   org-coderef-label-format))
+ (coderef (nth 6 info))
+	 (cref-regexp (org-src-coderef-regexp coderef))
 	 (link (let ((l (org-no-properties (org-store-link nil
  (and (string-match org-link-bracket-re l)
   (match-string 1 l
@@ -445,8 +444,7 @@ non-nil, return the full association list to be used by
 	(funcall assignments-cmd params))
 	  (when (string-match "-r" extra)
 		(goto-char (point-min))
-		(while (re-search-forward
-			(replace-regexp-in-string "%s" ".+" cref-fmt) nil t)
+		(while (re-search-forward cref-regexp nil t)
 		  (replace-match "")))
 	  (run-hooks 'org-babel-tangle-body-hook)
 	  (buffer-string
-- 
2.26.3



Re: Bug: inconsistent escaping of coderef regexp

2021-04-05 Thread Tom Gillespie
Missed removing a debug message. Here is the correct patch. Best,
Tom

On Sun, Apr 4, 2021 at 10:22 PM Tom Gillespie  wrote:
>
> Hi Nicolas,
>I've attached a patch with a first pass implementation that I think
> resolves most of the issues. It probably needs a few tests to go along
> with it, but I think it is the simplest way forward. I tried to make the
> changes without disrupting the org-babel info structure, but it comes
> with the cost of having to pull out :coderef-prefix in a number of separate
> contexts. Best,
> Tom
>
> > If possible, I'd like not to conflate current issue with switches
> > deprecation, which needs to be discussed separately.
>
> We can decouple them, so not an issue. The attached patch implements
> the header arg equivalents of -r and -l without making any changes to the
> existing switch behavior.
>
> > What do you mean by "it is impossible for the user to specify their own
> > coderef regexp that can be used in both cases"? In particular, what is
> > a coderef regexp in this context? I know about coderef format, but
> > I don't think users are supposed to provide a regexp here.
>
> I did a first pass implementation and realized that allowing users to
> specify coderef-regexp is a bad idea. The attached patch fixes the
> divergent behavior of org-bable-tangle-single-block and provides a
> standard way to specify a :coderef-prefix regexp so that empty
> comments can be stripped.
From 91aa10a5a14737b770e58b1a7f9f0e0b563dae62 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Apr 2021 21:40:32 -0700
Subject: [PATCH] improve org-src-coderef-regexp and regularize usage

* lisp/ob-core.el
org-babel-common-header-args-w-values: new :coderef- header args
org-babel-safe-header-args: include the new :coderef- header args
(org-babel-get-src-block-info): calulate params before info in let* so
that they can be used to set the coderef-format field (nth 6 info)
(org-babel--expand-body): use coderef-prefix to correctly strip
coderefs when expanding

* lisp/ob-tangle.el (orb-babel-tangle-single-block): Regularize
behavior when removing coderefs during tangling. This fixes an issue
where trailing whitespace would be retained when coderefs were removed
for tangling. Make the header argument :coderef-tangle no work the
same way that the -r switch currently works

* lisp/ol.el (org-link-search): use org babel info to match the
coderef format for each block

* lisp/org-src.el (org-src-coderef-regexp): now takes an additional
argument rx-prefix that can be used to customize the text preceeding
the coderef that should be removed during tangling, this is most
useful for removing comments and trailing whitespace.

* lisp/ox.el (org-export-resolve-coderef)
and (org-export-unravel-code): use org babel info to
correctly match the coderef format for each block.

This commit adds support for three new src block header arguments,
:coderef-format :coderef-prefix and :coderef-tangle. :coderef-format
has the same behavior has the org src switch -l and :coderef-tangle
has the same behavior as org src switch -r. :coderef-prefix provides
new functionality and makes it possible to set the regexp for text
leading up to the coderef. In particular this can be used to strip
comments, which are required if authoring an org file that works with
older versions of org.
---
 lisp/ob-core.el   | 43 +--
 lisp/ob-tangle.el | 17 ++---
 lisp/ol.el| 17 +++--
 lisp/org-src.el   |  5 +++--
 lisp/ox.el| 15 +++
 5 files changed, 60 insertions(+), 37 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 2e78ac3e6..feb6f2235 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -76,7 +76,7 @@
 (declare-function org-previous-block "org" (arg  block-regexp))
 (declare-function org-show-context "org" ( key))
 (declare-function org-src-coderef-format "org-src" ( element))
-(declare-function org-src-coderef-regexp "org-src" (fmt  label))
+(declare-function org-src-coderef-regexp "org-src" (fmt  label rx-prefix))
 (declare-function org-src-get-lang-mode "org-src" (lang))
 (declare-function org-table-align "org-table" ())
 (declare-function org-table-convert-region "org-table" (beg0 end0  separator))
@@ -392,6 +392,9 @@ then run `org-babel-switch-to-session'."
 (defconst org-babel-common-header-args-w-values
   '((cache	. ((no yes)))
 (cmdline	. :any)
+(coderef-format . :any)
+(coderef-prefix . :any)
+(coderef-tangle . ((nil yes no)))
 (colnames	. ((nil no yes)))
 (comments	. ((no link yes org both noweb)))
 (dir	. :any)
@@ -434,7 +437,8 @@ Note that individual languages may define their own language
 specific header arguments as well.")
 
 (defconst org-babel-safe-header-args
-  '(:cache :colnames :comme

Re: Bug: inconsistent escaping of coderef regexp

2021-04-04 Thread Tom Gillespie
Hi Nicolas,
   I've attached a patch with a first pass implementation that I think
resolves most of the issues. It probably needs a few tests to go along
with it, but I think it is the simplest way forward. I tried to make the
changes without disrupting the org-babel info structure, but it comes
with the cost of having to pull out :coderef-prefix in a number of separate
contexts. Best,
Tom

> If possible, I'd like not to conflate current issue with switches
> deprecation, which needs to be discussed separately.

We can decouple them, so not an issue. The attached patch implements
the header arg equivalents of -r and -l without making any changes to the
existing switch behavior.

> What do you mean by "it is impossible for the user to specify their own
> coderef regexp that can be used in both cases"? In particular, what is
> a coderef regexp in this context? I know about coderef format, but
> I don't think users are supposed to provide a regexp here.

I did a first pass implementation and realized that allowing users to
specify coderef-regexp is a bad idea. The attached patch fixes the
divergent behavior of org-bable-tangle-single-block and provides a
standard way to specify a :coderef-prefix regexp so that empty
comments can be stripped.
From e017fe3f4fb36da2c8560ae526b8bdfd42dc Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Apr 2021 21:40:32 -0700
Subject: [PATCH] improve org-src-coderef-regexp and regularize usage

* lisp/ob-core.el
org-babel-common-header-args-w-values: new :coderef- header args
org-babel-safe-header-args: include the new :coderef- header args
(org-babel-get-src-block-info): calulate params before info in let* so
that they can be used to set the coderef-format field (nth 6 info)
(org-babel--expand-body): use coderef-prefix to correctly strip
coderefs when expanding

* lisp/ob-tangle.el (orb-babel-tangle-single-block): Regularize
behavior when removing coderefs during tangling. This fixes an issue
where trailing whitespace would be retained when coderefs were removed
for tangling. Make the header argument :coderef-tangle no work the
same way that the -r switch currently works

* lisp/ol.el (org-link-search): use org babel info to match the
coderef format for each block

* lisp/org-src.el (org-src-coderef-regexp): now takes an additional
argument rx-prefix that can be used to customize the text preceeding
the coderef that should be removed during tangling, this is most
useful for removing comments and trailing whitespace.

* lisp/ox.el (org-export-resolve-coderef)
and (org-export-unravel-code): use org babel info to
correctly match the coderef format for each block.

This commit adds support for three new src block header arguments,
:coderef-format :coderef-prefix and :coderef-tangle. :coderef-format
has the same behavior has the org src switch -l and :coderef-tangle
has the same behavior as org src switch -r. :coderef-prefix provides
new functionality and makes it possible to set the regexp for text
leading up to the coderef. In particular this can be used to strip
comments, which are required if authoring an org file that works with
older versions of org.
---
 lisp/ob-core.el   | 43 +--
 lisp/ob-tangle.el | 18 +++---
 lisp/ol.el| 17 +++--
 lisp/org-src.el   |  5 +++--
 lisp/ox.el| 15 +++
 5 files changed, 61 insertions(+), 37 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 2e78ac3e6..feb6f2235 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -76,7 +76,7 @@
 (declare-function org-previous-block "org" (arg  block-regexp))
 (declare-function org-show-context "org" ( key))
 (declare-function org-src-coderef-format "org-src" ( element))
-(declare-function org-src-coderef-regexp "org-src" (fmt  label))
+(declare-function org-src-coderef-regexp "org-src" (fmt  label rx-prefix))
 (declare-function org-src-get-lang-mode "org-src" (lang))
 (declare-function org-table-align "org-table" ())
 (declare-function org-table-convert-region "org-table" (beg0 end0  separator))
@@ -392,6 +392,9 @@ then run `org-babel-switch-to-session'."
 (defconst org-babel-common-header-args-w-values
   '((cache	. ((no yes)))
 (cmdline	. :any)
+(coderef-format . :any)
+(coderef-prefix . :any)
+(coderef-tangle . ((nil yes no)))
 (colnames	. ((nil no yes)))
 (comments	. ((no link yes org both noweb)))
 (dir	. :any)
@@ -434,7 +437,8 @@ Note that individual languages may define their own language
 specific header arguments as well.")
 
 (defconst org-babel-safe-header-args
-  '(:cache :colnames :comments :exports :epilogue :hlines :noeval
+  '(:cache :coderef-format :coderef-prefix :coderef-tangle
+   :colnames :comments :exports :epilogue :hlines :noeval
 	   :noweb :noweb-ref :noweb-sep :padline :prologue :rownames
 	   :sep :session :tangl

Re: Bug: inconsistent escaping of coderef regexp

2021-04-04 Thread Tom Gillespie
Hi Nicolas,
After a bit of investigation I understand the issue better now.
There are two problems here. One is an easy single line change,
the other is a deeper issue, which is that it is impossible for the
user to specify their own coderef regexp that can be used in both
cases. No matter what change we make we are likely to break
existing org files if users relied on one behavior and not the other.

Given this, I would say that it is worse to break tangling behavior
than it is to break coderef search because it is obvious to the user
when coderef search breaks, whereas a change in tanging
behavior is a silent change that users will not be aware of.

If we want a temporary fix, a patch is attached, but I would suggest
against changing the behavior right now and instead work toward
a new, more consistent system using header args.

I think that moving to use header args to control these is an
opportunity to resolve both issues, and to make a start toward
eventually deprecating the switches. The only question that I
have right now regarding that implementation is whether we
provide header args for just the coderef regexp or also for the
coderef format, with the coderef regexp taking precedence. The
deeper issue is that the format string that appears in the org-src
snippit below is hard coded, and if we allow users to set the
coderef format, then it may make sense to let them set that
format string. However, this would duplicate the simpler
functionality of simply allowing the user to provide their own
coderef regexp.

At the moment I have two proposed header args which are
:coderef-regexp with default matching the current output of the
org-src snippit below, and :coderef-tangle which defaults to yes
matching the behavior of the existing =-r= switch. There is an
option for a 3rd header arg that would directly replaced the =-l=
switch :coderef-format, however as mentioned above it adds
significant complexity and requires a fourth argument
:coderef-surround or something like that which is the
hard coded format string in the org-src snippit.

I'm working on a basic implementation and will respond in this
thread again when I have something worth looking at. Best!
Tom

For the record there are at least 3 different inconsistent regex
that are used to detect coderefs.

org-element:
(string-match "-l +\"\\([^\"\n]+\\)\"" switches)
is duplicated between org-element-src-block-parser and
org-element-example-block-parser

org-src:
(format "\\([ \t]*\\(%s\\)[ \t]*\\)$"
  (replace-regexp-in-string
   "%s"
   (if label (regexp-quote label) "\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)")
   (regexp-quote fmt)
   nil t))
ob-tangle:
(re-search-forward (replace-regexp-in-string "%s" ".+" cref-fmt) nil t)
diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index aa0373ab8..677d9d8ba 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -446,7 +446,7 @@ non-nil, return the full association list to be used by
 	  (when (string-match "-r" extra)
 		(goto-char (point-min))
 		(while (re-search-forward
-			(replace-regexp-in-string "%s" ".+" cref-fmt) nil t)
+			(replace-regexp-in-string "%s" ".+" (org-src-coderef-regexp cref-fmt)) nil t)
 		  (replace-match "")))
 	  (run-hooks 'org-babel-tangle-body-hook)
 	  (buffer-string


A formal grammar for Org

2021-04-04 Thread Tom Gillespie
Dear all,
   Here is a draft of a formal grammar for Org mode [1]. It is still
in a rough state, despite quite a bit of work. However, following some
changes to improve performance for parsing real (big) Org files, I
think it is time to share it with the community so that we can start
to gather feedback. There are a number of opportunities that I have
found for simplifying the org grammar (sometimes by extending it to
make it more regular, and in the process adding useful features) that
are much easier to understand with this grammar in hand as a
reference. The grammar itself is implemented using Racket's #lang brag
(see [2] for an overview of brag's syntax). I had considered trying to
break it up into literate sections in an Org file, but for now decided
to leave it as a single file to simplify the development workflow. As
a result the full implementation is fairly long [3]. Comments and
feedback would be greatly appreciated. Best!
Tom

1. https://github.com/tgbugs/laundry
2. https://docs.racket-lang.org/brag/#%28part._.The_language%29
3. https://github.com/tgbugs/laundry/blob/master/org-mode/parser.rkt



Re: Using backticks for the inline code delimeter?

2021-04-03 Thread Tom Gillespie
Is there any reason why folks couldn't just customize
org-emphasis-alist using a file local variable? Just add ("`" org-code
verbatim) and see what happens. Knowing a bit about the codebase this
will probably lead to trouble during export because the backends are
likely unaware, but at least it can be specified in the file. In
general adding a token that duplicates the  function of an existing
token is a bad idea. For a similar reason mixed delimiters cannot be
allowed, they make the grammar completely ambiguous. Best,
Tom



Re: Idea for handling timezones

2021-04-03 Thread Tom Gillespie
Time zones are tricky, but there are some requirements that make it
possible to rule out many apparent solutions because they violate some
critical invariant. For example. Timezone cannot be scoped to the
file. This is because many users (myself included) have a single org
file that is used in and across multiple timezones. Thus the original
proposal in this thread to have a #+timezone: keyword is insufficient
for many use cases, and can lead to significant confusion and bad
data.

Similarly it is probably a bad idea to always use UTC because even
though it will technically always be correct (while on Earth ...), it
means that the user will need to have kept track of their spatial
coordinates in addition, and in addition there will be an eternal
dependency on the timezone database in order to correctly reconstruct
what the timezone would have been. All in all it is better to assume
that the user has correctly configured their clock for their needs,
and record the timestamp with timezone offset (NOTE textual timezones
cannot be used in timestamps for a variety of reasons not the least of
which is that they are ambiguous, see CST for example). Sadly, the
ISO8601 timestamp specification for a negative timezone offset uses a
hyphen minus, the same as the date separator, so that baggage is going
to be with us for the rest of time (at least until someone comes up
with something better than ISO8601), but at least it requires no
additional information to capture all the information needed to
correctly reconstruct the time that it stamped.

Finding a timestamp format that has a regular grammar, is invariant to
changes in the location and time of the user (What if the user is on
Mars? What happens after year ? What happens if someone needs to
reference a date in the far future? What if someone wants to mention a
historical date e.g. 1000BCE?), doesn't require external information,
and is also somewhat human readable, is a major challenge. If we could
serialize something like the unix epoch into a file and render it
differently in the buffer that might work, however that defeats one of
the major points of using org as a plain text format.

My proposal at the moment based on the constraints imposed by the
current timestamp format that includes the repeat or delay syntax
would be the following (date and time and day are separated by a
space)

date: ([+-][0-9]\+|[0-9]{4})(-[0-9]{2}){2}
time: ([0-9]{2}:[0-9]{2}(:[0-9]{2}(,[0-9]{1,9})?)?[+-][0-9]{2}:[0-9]{2})?
day: ([a-zA-Z0-9]\+)?

followed by a space and then the repeat or delay for example
[+1-01-01 10:11:00,992315771-04:00 Sat] or
[-480-08-20 05:00+01:00]
or more temporally local
[2021-03-03 17:43-07:00 Sat]

This is the closest I have been able to get while working through
formalizing all of org syntax and trying to come up with more robust
solutions for timestamps.
For comparison see org-ts-regex and friends in org.el. I have also not
come up with a better solution for the repeat or delay syntax, though
ISO8601 interval
specification might be an option. Similarly there are extensions for
dealing with uncertain dates and times, but I don't have good
proposals for those right now,
and the use cases are also somewhat out of scope.

Best,
Tom



Re: Font lock in org+elisp confused with ?\[

2021-04-02 Thread Tom Gillespie
Reposting my reply to the emacs-devel thread here as well. The hack I
mention that has performance issues was derived from John's solution
for the <> issue (though the performance issues are all of my own
creation). Best,
Tom

This is a known issue with org babel blocks. It is due to the fact
that org babel translates the font locking for the language but not
the syntax propertization. Another frequent cause is the bash case
statement. The end result is that unmatched parens leak out from the
babel blocks and wreak havoc elsewhere in the org file unless you
balance out the parens e.g. in a comment. I have a hacked fix for
this, but it has horrible performance, especially with line numbers
enabled. I think that a proper solution would run arbitrary syntax
propertization on subsets of a buffer without having to continually
check where those subsets start or end.



Re: Syntax Proposal: Multi-line Table Cells/Text Wrapping

2021-04-01 Thread Tom Gillespie
Tim,
   Your concerns are well founded. Essentially any cosmetic extension
to the org table syntax will be an unmaintainable, bug ridden
nightmare and would be an eternal burden on any attempts to formalize
correct behavior. I have a draft of a grammar for a significant
portion of Org syntax (forthcoming), and getting tables right was
quite hard, despite their seeming simplicity. Any surface syntax level
additions would require updating an unknown number of parts of the
codebase to accommodate the change, since every part of org implements
its own parser (e.g. there is already a bug in the interaction between
tables and #+macro: commands containing the pipe character that
org-export implements incorrectly). Relatively speaking, the potential
approaches discussed in Timothy's previous proposal are much more
likely to be tractable. Best,
Tom



Re: Properties on buffer level

2021-02-12 Thread Tom Gillespie
You should be able to run C-c C-c on #+property: directives before the
first headline and they will be updated without reloading the buffer.
Best,
Tom



Re: Display ellipsis at end of headline instead of after tags

2021-01-10 Thread Tom Gillespie
Not 100% sure about this, but take a look at the hs-set-up-overlay
variable, it seems like it might be possible to customize that to
achieve this behavior. Best,
Tom



Re: Possibility to copy text outside EMACS and send it to orgmode document

2021-01-06 Thread Tom Gillespie
It is built into the core org distribution.



Re: temporary inclusion of a read-only file / inline element into orgmode buffer

2021-01-06 Thread Tom Gillespie
Check out org-transclusion, it covers some of the use cases you
describe or could be used to implement them. The issue with giant
results, especially those that have very long lines, is more
challenging, but being able to specify that different output streams
should go to files instead of results might be a start, or having them
go to a file and having process do the equivalent of tail on the file
to continually update the results stream of the buffer would be really
useful (if it could be done without disrupting other editing flows.
The shortest path to splitting file vs buffer would probably be to
decorate org-bable-insert-result (in ob-core.el) to detect the size of
the result and write the full result to a separate buffer if it is
beyond the limit you set. Best,
Tom

https://github.com/nobiot/org-transclusion



Re: Possibility to copy text outside EMACS and send it to orgmode document

2021-01-05 Thread Tom Gillespie
You can achieve something a bit like this using
org-protocol-capture-html
https://github.com/alphapapa/org-protocol-capture-html. I'm not
entirely sure whether you can bind the equivalent of a hook in
javascript to run every time you C-c, but if you can, then it should
be possible to match this exactly. You might also be able to wire up
org-protocol to the X clipboard (if you are on linux).

Best,
Tom



Bug: inconsistent escaping of coderef regexp

2021-01-04 Thread Tom Gillespie
It is not possible to strip coderefs when tangling and also search for
those coderefs using org-link-search. This is because org-link-search
uses org-src-coderef-regexp which calls regexp-quote on the regexp
string while org-babel-tangle-single-block does not and uses the
regexp string directly without quoting it. I'm not sure about the best
way to fix this. It seems to me that the call to regexp-quote should
be removed but I'm not entirely sure of the consequences of doing
that. Thoughts? Best,
Tom

PS While on the topic of coderefs, let me drop a note that is a
preview of some of the issues I have encountered while working on a
full formal grammar for org. Having the -l switch control this is an
awful design that induces more complexity into the org-mode grammar
than nearly any other feature. Source block switches are completely
inconsistent with the rest of org and completely undiscoverable. I had
no idea they even existed until I was trying to figure out which
header argument could be used to set the coderef regexp. The -l option
and switches in general need to have their behavior implemented as
part of the standard header arguments like everything else so that
users can migrate away from switches with an eye toward removing them
entirely.



Re: did behaviour of RET change again?

2020-12-23 Thread Tom Gillespie
> possibly i'm misunderstanding, but my sense is that the value of org
> adapt indentation doesn't change what you might actually find ("in a
> .org file in the wild").  so, whatever its value, your grammar would
> have to deal with all cases?

Yep, we can't magically change all the files out in the wild.
Until I wrote this email out I agreed about the grammar too, but
as it turns out I think there might be a compromise, which is that
a grammar could be allowed to interpret certain types of lines
with leading whitespace as if they were paragraph lines instead of
as say, the start of an org-babel block. That way an interoperable
org spec could be made simpler without preventing e.g. the elisp
implementation from going above and beyond the spect to provide
support for leading whitespace.

My interest in the org-adapt-indentation setting is to try to align what
most users are doing out in the wild with a representation for their
org files that is less ambiguous and more performant (among other
things).

If I had to guess I would say that most new org files have leading
whitespace precisely because org-adapt-indentation is set to t by
default. Getting users to transition their files requires an incentive,
and some users may find that they use org in such a way that they
don't need to.

While writing this email I thought of a reasonable incentive, which is
that only files without leading significant whitespace (i.e. that look like
those that are authored with org-adapt-indentation nil) have specified
behavior for things like org-babel blocks. This would allow best effort
by the elisp org-mode implementation to allow users who don't care
about interoperability to continue as they have been doing, while also
providing clear guidance for what users must do if they want
consistent behavior on other tools that consume org formatted files.
This is critical for keeping the org spec from becoming overly complex.

The first step would thus be to reduce the rate at which new org files
are created with leading whitespace by changing org-adapt-indentation
to be nil by default.

The second step would be to give users a clear way forward to safely
normalizing their files. Org has a habit of reindenting subsets of files
from time to time, but in general doing a complete switch to have no
significant whitespace can be risky.

The third step would be to let the incentives and needs of users
play out over time, but users would generally probably be happier
because by default they would be in the "my files are portable and
generally interpretable without me having to do any additional work"
state instead of the "why did no one tell me the defaults weren't
portable" state.

> or, and maybe this would make sense, you'd define an "interoperability"
> form of .org, that all "wild" .org files could be (programmatically)
> converted into, without losing any of their semantics?

Yep exactly. For many use cases the cost of stripping the whitespace
that corresponds to heading level is not unreasonable, e.g. since it
would only have to be done once. However, if you are writing another
org implementation, then every single time you parse a heading and
its accompanying section you have to strip the whitespace, and that
will happen every single time a user modifies the file, which starts to
get expensive. Alternately you could implement it so that everything
is stripped once and then you keep a version in memory that the user
edits which doesn't have leading whitespace, but then every time you
save you have to splice it all back in to preserve the roundtrip.

This also doesn't even begin to deal with the fact that users can create
malformed org files that can have lines that are less than the expected
significant indentation. This becomes a complete nightmare when trying
to come up with some rational way of dealing with org babel blocks for
languages like python where there is significant whitespace.

Consider the horror if trying to specify the correct behavior in a situation
like this. Better just to declare it a malformed babel block and move on.
Unfortunately you can't always detect that such things are malformed
during the first pass of parsing because you have to count the number of
spaces.

#+begin_src org
# -*- org-adapt-indentation: t -*-
,*** Oh No
,#+begin_src python
  class What:
  have = 'you'
  done = 'now'
,#+end_src
#+end_src

In order to ensure that org files can be reliably interpreted this either
means that the specification for handling cases like this becomes
extremely complex, or the spec can say "you can have leading
whitespace, but nasal demons may come for you, there is only
specified behavior if you do not have leading whitespace."

Having unspecified behavior in cases like that would mean that an
implementation could be fully compliant if it were to treat any
#+begin_src lang line that started with whitespace as if it were
a paragraph instead of a babel block. I suspect that this will be
the best way 

Re: did behaviour of RET change again?

2020-12-23 Thread Tom Gillespie
> in case not obvious, i am suggesting a nil value for org adapt indentation.
> thus no physical indentation of all lines including planning lines.
> i'd even suggest no physical indentation as default for example and
> source blocks, but that is a can of worms.

I know that this is a can of worms, but I agree. Given that the effects of
org-adapt-indentation can be mimicked in other ways without having the
literal spaces present in the file it may not be as big a deal as we think.

The other reason I think this is a good idea is because I have been working
on a formal grammar for the org syntax, and everything would be SO much
simpler about the implementation after the first pass parse if the canonical
representation of an Org file did not allow significant whitespace (with an
exception for plain lists).

Just avoiding having to deal with any number of nasty edge cases for correctly
aligning org babel blocks would be worth it. Not to mention the fact that it
means that you have to do a triple pass over each incoming line in order to be
sure that what you are passing to an org babel block has had the leading
whitespace removed (once for a normal parse, second time to adjust whitespace
and a third time to actually parse the babel block). No significant
leading whitespace
would remove the need for an entire pass in the parser.

I will have more on the subject when I finally get around to sharing the
grammar, but suffice to say, that having org-adapt-indentation set to true
and putting the leading spaces in the file (instead of doing whatever it is
that doom does by default) induces significant complexity into the
implementation. I would love to see it gone, as I'm sure anyone wanting
to parse org files in future will too.

Best!
Tom



Re: Bug: org-element does not recognize table.el tables [9.4 (release_9.4-53-g23f941 @ /home/nick/elisp/org-mode/lisp/)]

2020-12-21 Thread Tom Gillespie
A few years ago I was trying to format tables in a certain specific
way and it was essentially impossible with org tables for reasons that
now escape me. However, it was possible to accomplish it using
tabels.el tables. I don't think I ended up actually using the
tables.el solution, but at the time it would have been the only
option. If memory serves I think it had to do with multiline cells or
something of that nature. While it would be nice to be able to
converge on a single implementation for tables, I suspect that there
are still users of tabels.el that would be left out in the cold if
that functionality was removed. It would be great if we could find
some examples of org files or users that use the tabels.el
functionality so we can understand what org tables are missing. Best,
Tom



Re: [PATCH] Enhance org-html--build-meta-info

2020-12-15 Thread Tom Gillespie
Hi Timothy,
I understand now. Having a way to implement this in the config is
a good thing as it covers a slightly different set of use cases and
workflows than always using a common #+setupfile: line. That way if
you are working with files that don't have a #+setupfile: specified
you can still add metadata without having to modify the files. This
would vastly simplify some of my documentation generation code where I
modify the first section of a bunch of org files as I process them
rather than modifying the config. Thanks!
Tom



Re: [PATCH] Enhance org-html--build-meta-info

2020-12-15 Thread Tom Gillespie
A question from the slightly uninformed. Why not just use #+html_head:
possibly with a macro to fill in variable values? That is fully
extensible and doesn't overload keywords. For title, date, author,
etc. those can have clearly defined mappings to the html, but
everything else seems to be handled more sanely with #+html_head:. Am
I missing something? Best,
Tom



Re: Bring up a screen giving option to open a series of orgmode files

2020-12-15 Thread Tom Gillespie
To hop in on the hypothes.is thread. I have spent quite a bit of time
working with hypothes.is and related tooling (mostly in python), so
here is a brain dump on interactions between org and hypothes.is. As
others have mentioned, this could easily be its own thread. Best!
Tom

A quick note on security for hypothes.is. Last I checked (about 30
seconds ago) there is still no way to control access to groups, if the
identifier for the group leaks then anyone can access it. This is not
the case for private annotations, those can only be seen by someone
with your api key (hopefully just you).

If you are looking for a light weight client that is hypothesis
compatible that could be used to build a tool that can push
annotations to an alternate backend then
https://github.com/judell/hlib might be a reasonable place to start.
Jon has previously used that to create a client that sent annotations
to an alternate backend, which could in theory be an elisp
implementation of a server for the w3c annotation standard that could
feed things to org-protocol (or similar).

If people are interested in this for org-mode I would suggest that a
starting point would be to write an elisp implementation that can
consume and produce the w3c web annotation standard format for
annotations and/or the hypothesis api format.

There are at least 3 different ways that web annotations can be
anchored, offset, xpath, exact + prefix/suffix. In principle you could
translate those into urls and use query parameters to encode the
target selectors. The problem that you will run into is that there are
some rather sizeable selector patterns like the example below (that I
happened to have up in another terminal) which will be a pain to work
with as urls. As a result of this a reasonable workflow would be to
create a custom link type for the hypothes.is annotation urls e.g. the
equivalent of #+link: hyp https://hyp.is/ and simply paste in a
shortened form of the share links. In addition one might want some
additional tooling so that the contents of the annotation could be
retrieved and cached, or retrieved, transformed, and embedded in the
document as an appendix during export (or similar).

Unifying org-capture, org-protocol, and general org hyperlinking with
the w3c spec seems like it would be hard in the general case, but for
specific use cases I can imagine that some reduced syntax could be
created that would fit in an org hyperlink. It actually would probably
be easier to do this by
coming up with a way to convert structured org sections or blocks to
and from the w3c spec, name those, and then use org hyperlinks to
refer to
the annotations directly in an org file that functioned as an
annotation store. Much less overhead than trying to set up a
stripped-down stand-alone web annotation server, and if you can
produce json to match the hypothes.is API then you could make use of
that to publish and share annotations/links when you go to publish an
org document.

'selector': [{'type': 'FragmentSelector',
 'value': 'pes-1',
 'conformsTo': 'https://tools.ietf.org/html/rfc3236'},
{'type': 'RangeSelector',
 'endOffset': 92,
 'startOffset': 86,
 'endContainer':
'/div[4]/div[1]/div[4]/div[1]/article[1]/section[1]/article[1]/div[1]/div[2]/ul[1]/li[1]/div[3]/div[2]/div[1]/div[1]/div[1]/div[1]/div[1]/div[1]/figure[1]/div[1]/div[1]/div[1]/div[1]/div[2]/div[1]/div[1]',
 'startContainer':
'/div[4]/div[1]/div[4]/div[1]/article[1]/section[1]/article[1]/div[1]/div[2]/ul[1]/li[1]/div[3]/div[2]/div[1]/div[1]/div[1]/div[1]/div[1]/div[1]/figure[1]/div[1]/div[1]/div[1]/div[1]/div[2]/div[1]/div[1]'},
{'end': 3034, 'type': 'TextPositionSelector', 'start': 3028},
{'type': 'TextQuoteSelector',
 'exact': '100kHz',
 'prefix': ' DC - 20kHz\nSampling frequency: ',
 'suffix': '\nOnboard stimulatorNeural Interf'}]}



Re: Emacs as an Org LSP server

2020-12-14 Thread Tom Gillespie
See also. https://lists.gnu.org/archive/html/emacs-devel/2017-04/msg00798.html
and 
https://www.reddit.com/r/emacs/comments/696pv1/rms_supports_language_server_protocol_integration/
for some discussion. Best,
Tom

On Mon, Dec 14, 2020 at 4:31 PM Tim Cross  wrote:
>
>
>
> I am no fan of Microsoft. I have run Linux as my primary desktop since
> 1994. I have been working as a developer since 1988 and have first hand
> experience regarding many of the poor business practices of Microsoft.
> However, I think the LSP is actually a positive imitative and a
> potential benefit to all developers and all editors and development
> tools, both closed and open source. I have outlined some points I think
> are relevant below.
>
> Russell Adams  writes:
>
> > On Tue, Dec 15, 2020 at 01:08:47AM +0800, TEC wrote:
> >>
> >> Jean Louis  writes:
> >>
> >> > [LSP is a evil plot from microsoft]
> >>
> >> I can see that you're overly concerned about Microsoft being able to
> >> somehow exert control over this. It may assuage your concerns to see an
> >> example "technology stack" that Org-LSP could fit into.
> >
> > REST API calls to a remote server as a core part of editing text in
> > your editor isn't concerning? How remote? How would you know? If they
> > use HTTPS could you even see what is sent?
> >
>
> LSP is not restricted to REST. It uses JSON as the message format, but
> can use any form of remote procedure call. It could be REST, it could be
> basic Unix socket interface, it could be some other TCP interface. Any
> interface both sides understand will work.
>
> >> Microsoft has provided a /standard/ that a huge number of editors/IDEs
> >> have adopted with /independent implementations/. At this point there is
> >> /nothing/ M$ could do to interfere with how the above works.
> >
> > Microsoft doesn't make standards that it can't corrupt or take
> > advantage of. See LDAP/AD, HTML extensions, programming language
> > extensions that makes their solutions incompatible with standards.
> >
>
> Yes, Microsoft does have a poor reputation when it comes to standards.
> However, if you consider why they have proposed the LSP and what their
> business model is, it becomes fairly obvious there is no benefit for
> them in changing this specification without good technical reasons.
>
> Microsoft has proposed the LSP because it has the potential to make
> their editors more popular and able to support more languages. They want
> others to implement the LSP server so that they can support the language
> in their editor or development tools with minimal development effort,
> increasing market share and reducing maintenance costs. Nothing unusual
> with that - basic business principal. They won't want to modify or
> change the protocol unnecessarily as that will break their own
> integration with these servers. Their business model relies on
> maintaining the standard they have proposed.
>
> The key point here is that other technologies, including free software
> tools like Emacs, can also benefit from this technology. I'm sure
> Microsoft would prefer to prevent this, but they can't if they want
> others to develop the language server modules.
>
> One of the biggest challenges for editors like Emacs is how to provide
> support for new languages which include all the features people expect
> in a modern editor. Often, it is extremely difficult to provide these
> features without incorporating some form of language parser or compiler.
> this is difficult to do with just Elisp. To try and work around these
> limitations, Emacs has used things like ECLIM to interface with the
> Eclipse editor and leverage off the internal facilities it provides.
> What the LSP does is provide a generic interface definition which works
> in a similar fashion, but is not dependent on an external server like
> Eclipse.
>
> Consider the potential of a future where in addition to defining a new
> language, tools to compile/parse and execute the language the projects
> which develop/maintain that language also provide an LSP server for that
> language. this would mean that in order to use that language in your
> editor, all you need to do is configure your LSP client to communicate
> with that server.
>
> I currently use 4 different LSP servers in my Emacs setup. None of them
> require a web server. They are all just scripts/executables which sit in
> my bin directory. When I open a file in a language which has a LSP
> server, Emacs starts the server and communicates with it to handle
> completion, linting, format hints etc. There is no external network
> connection required, no remote services, no reporting to MS. Even if
> Microsoft changes the specification, it has no impact on my current use.
>
> >> You seem to be focusing on the term "server" in the name. This seems to
> >> be a red herring in this case. In LSP the server is analogous to "emacs
> >> --daemon" and the client to "emacsclient".
> >
> > REST = web server. Using to make JSON requests over what 

Re: Bug fix attached: org-babel sql postgres, fix hardcode

2020-12-12 Thread Tom Gillespie
It looks like the two patches are sequential, there should probably be
a rebase into a single patch. I would remove the comment in the second
patch because it is a single command to jump to the default value and
it might change in the future, so no reason to put it in a comment.
One way to communicate about the source of that variable is to include
(require 'sql) in ob-sql which is likely needed anyway due to the fact
that sql-postgres-program is defined by sql.el and there will be a
byte compiler error because the variable is undefined. Having the
value of "psql" present as a backup in case sql-postgres-program is
somehow nil seems reasonable. Best,
Tom



Re: stability of toc links

2020-12-08 Thread Tom Gillespie
It sounds like you are looking for the CUSTOM_ID property. See
https://orgmode.org/manual/Handling-Links.html and
https://orgmode.org/manual/Internal-Links.html. I don't remember
whether there is a way to generate ids matching headlines within org
itself, but there is
https://github.com/alphapapa/unpackaged.el#export-to-html-with-useful-anchors.
Best!
Tom



Re: new website: not easy to find how to ask for help

2020-12-08 Thread Tom Gillespie
Hi Eric,
   Good point, we are indeed missing a line that says "You can mail
the list directly at mailto:emacs-orgmode@gnu.org.; Here's a patch. I
assume it is probably ok to put the raw email on the site. Best,
Tom
From 490c48a9750d04571e63250208ae90b2cd85 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Tue, 8 Dec 2020 15:50:06 -0500
Subject: [PATCH] community line about mailing list directly

---
 community.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/community.org b/community.org
index 427fa06..9eb3c49 100644
--- a/community.org
+++ b/community.org
@@ -10,11 +10,13 @@
   :END:
 
 Org mailing list is the central place where to connect with Org
-community.  
+community.
 
 You can [[https://lists.gnu.org/mailman/listinfo/emacs-orgmode][subscribe to the list]] and [[https://orgmode.org/list][browse the archive]] here at
 =orgmode.org= or via the [[https://lists.gnu.org/archive/html/emacs-orgmode/][GNU mailman archive]].
 
+You can mail the list directly at mailto:emacs-orgmode@gnu.org.
+
 If you are not subscribed to the mailing list your message may be
 delayed because it will need to be approved by the moderators.
 
-- 
2.26.2



Re: How to evaluate source code while in the edit buffer?

2020-12-04 Thread Tom Gillespie
While we're on the subject of execution in the edit buffer, it is not
entirely clear what such execution would mean. Mirko's desire seems to
be to execute the buffer directly without org-babel as an
intermediary. However, the most consistent approach would be a command
to pop back to the source buffer from the edit buffer and use
org-babel-execute-src-block to run the code. If org were to provide
edit buffer evaluation it would likely have to be that. The reason for
this is that direct execution of code in the major mode of the edit
buffer is not well defined. For some languages (e.g. racket) there is
no easy way to evaluate a buffer that does not correspond directly to
a file on the file system, which is part of why org babel exists in
the first place (to define those semantics). In theory babel languages
could provide an alternate implementation for how an edit buffer can
be run, but absent such an implementation it would be up to the user
to figure out the semantics for each of the languages they wanted to
evaluate directly. Best,
Tom



Re: How to preserve empty headings

2020-11-30 Thread Tom Gillespie
This is caused by elastic-indent-mode. As foretold
https://lists.gnu.org/archive/html/emacs-orgmode/2020-11/msg00325.html.
Tom

On Mon, Nov 30, 2020 at 1:38 PM Titus von der Malsburg
 wrote:
>
>
> On 2020-11-30 Mon 19:25, Diego Zamboni wrote:
> >>
> >> I’m aware of several workarounds and this one is perhaps the best.
> >> However, I’d prefer if RET would just work as expected.  Org sometimes
> >> inserts extra material on RET which I think is okay (e.g. indentation), but
> >> is there any precedent, in Org or Emacs more broadly, for RET deleting
> >> text?  It seems very counter-intuitive to me.
> >>
> >
> >  Could it be that the space is being deleted not when you press RET but
> > when you save the file? I don't see any space deletion when entering an
> > empty headline, but in my config, whitespace at end of lines is deleted
> > on save. In Doom Emacs this is enabled by default, and even before I was
> > using =delete-trailing-whitespace= as part of my =before-save-hook=.
> > --Diego
>
> The space is deleted immediately.  But the fact that it’s not happening on 
> your system perhaps means that there *is* a setting that prevents it.  The 
> question is: which?
>
> By the way, I’m using the master branch from 
> https://code.orgmode.org/bzg/org-mode.git as installed by straight.el.
>
>   Titus
>
>



Re: Emacs inserts hardwired org-agenda-files variable, overwriting user options

2020-11-29 Thread Tom Gillespie
> Would this affect my own custom

What I did when I migrated was to manually move the existing
custom-set-variables and custom-set-faces from .emacs to the new
custom-file (in my case ~/.emacs.d/custom.el), that way the old
definitions will still be loaded. All new customizations will live in
custom-file. I'm not sure what would happen if the old definitions
were still in .emacs, regardless, a good time to make a backup of
.emacs just in case. Best,
Tom



Re: looking for a macro eval workaround (9.1 vs 9.2 and +) for export backend test

2020-11-29 Thread Tom Gillespie
Not sure if this helps, but the example that I came up with for the
quickstart https://orgmode.org/quickstart.html#macros has an example
(see below) of using multiple @@ export snippets in a single macro. If
you have consistent naming conventions for pdf vs svg you might be
able to write a variant #+macro: image that works like
{{{image(file-name-without-extension)}}}. Not sure this will get you
what you want though since I imagine that you want to modify how links
such as [[file:file-name.ext]] are exported. One alternative would be
to define a custom link type
https://orgmode.org/manual/Adding-Hyperlink-Types.html that would do
the type setting for you. If you want those definitions to live in the
org file you could use Eric's eval: (org-sbe startup) local variable
approach to ensure that the elisp definitions are always available.
Best,
Tom

The example from the quickstart:
#+MACRO: red @@html:$1latex:\textcolor{red}{$1}@@



Re: Emacs inserts hardwired org-agenda-files variable, overwriting user options

2020-11-29 Thread Tom Gillespie
Here is a workaround. Emacs klobbering settings in .emacs has caused
many issues for me in the past. The solution I finally came up with
was to ensure that custom variables are loaded before any of my
settings. Near the top of my .emacs (before any calls to setq) I have
the following:

 custom set variables
;;; PLEASE MAKE YOUR WAY TO THE EXIT AND STAY OUT OF MY INIT FILE
;;; These come first so that they will be overwritten by settings in packages
;;; in order to prevent stale variables from klobbering new values
(setq custom-file (expand-file-name  "custom.el" user-emacs-directory))
(when (file-exists-p custom-file)
  (load custom-file))

If you have that then your list will be retained. However Emacs will
probably continue to ask you to remove the missing file until some
file exists at that path. Not sure about the org agenda behavior for
missing files since I populate org-agenda-files by scanning folders
for existing org files and then having a blacklist to exclude files I
do not want.

Best,
Tom



Re: Multiple named code blocks

2020-11-28 Thread Tom Gillespie
Hi Félix,
   I think that it is probably not a good idea to implicitly
concatenate blocks that share the same name. There are a number of
major downsides. One reason is that all the other parts of org-mode
assume that there is only a single block with that name, or rather
have undefined behavior if there is more than one, so all the other
reference resolving infrastructure will not work as expected and it
will be hard to navigate to additional blocks. The concatenation
behavior you describe does work if you specify the same file for
tangling, however that uses completely different code. Further, if we
were to define behavior for what should happen if there are multiple
blocks or sections with the same name it would probably either be to
signal an error (if you are in the more immutable camp), that the
first named block is the "canonical" block (current implementation),
or that the last defined block is the "canonical" block (if you want
org to be more like a programming language). The current
implementation follows the first named block, but the reason why the
manual states that it is undefined is because even that behavior
should not be relied upon (thus why an error is probably the most
friendly thing to do). Consider also that if you have different blocks
by the same name in different sections and you reorder the sections
the order in which they are nowebbed in will change. Given that that
behavior can be actively dangerous (imagine a block with cd
some-folder followed by a block rm contents/ -r getting reordered),
there are major downsides to trying to guess how to concatenate
multiple blocks and trying to specify restrictions on where and how
using the same name is allowed or can be safely used is not something
that anyone would want to try to do. Best!
Tom



Re: Local variables insecurities - Re: One vs many directories

2020-11-25 Thread Tom Gillespie
> As there is the option ! to "apply local variables and permanently
> mark these values" but there is no option "not to apply local
> variables and permanently mark these values".

I have a longer reply that I will send tomorrow, but wanted to respond
to this.

Yes exactly! I have the equivalent complaint in the draft extras from
my previous message! I actually implemented a blacklist for file local
variables in orgstrap because it is a critical security feature. The
fact that it is missing is absolutely a bug and is extremely annoying
for some of my current workflows where I have local variables that I
never want to accept.



Re: Local variables insecurities - Re: One vs many directories

2020-11-25 Thread Tom Gillespie
Hi Jean,

Some points in summary before a long email.
1. Having an accepting default behavior as a user (i.e., saying yes to
   all prompts) is bad security practice. The only thing that systems
   can do is prompt as infrequently as possible in hopes that people
   don't get prompt fatigue. Emacs does this.
2. If mutt is launching Emacs, you can pass --eval "(setq
   enable-local-eval nil)" on the command line and all file local
   variables will be ignored and treated as plain text.
3. If people don't read the manual and don't read the prompt, there
   isn't much more that Emacs can do while still empowering the
   user. In those cases we also can't assume that the community will
   be of much help either, as we are trying to be here.
4. That said, the local variables prompt could be more alarming and
   provide a quick way to access the manual about local
   variables. I'll look into a patch for that.

Now on to the rest!

Full disclosure. I make use of file local variables every day
and I have spent quite a while writing orgstrap which relies heavily
on them, so I am definitely biased.

Emacs is a sharp tool. Experts need sharp tools. If someone complains
about the security of prompts but also admits that they always say yes
to prompts without reading them, then no one will take them seriously
when they try to argue that it is the prompt that is insecure. It is
like complaining that the forest is dangerous while insisting on
eating the berries of all the plants and picking up all the snakes.
The forest is dangerous, but not merely because the berries and the
snakes exist in it. Heck, some snakes even try to warn you! There are
some rattlesnakes that have evolved to not rattle because idiot humans
go find and kill them. Now everyone suffers because there are silent
rattlesnakes that will strike without warning.

Degrading usability because some users are irresponsible just
reinforces the irresponsible behavior and everyone loses. The endgame
for all prompts is a two-man two-key system where the switches are
separated by 30 feet and must be turned within 1 second of each
other. How else could we be sure that the user really meant to say
yes!? Emacs is scary, but not nuclear launch system levels of scary.

Furthermore, Emacs does try to account for users who don't read the
manual and has sane and safe defaults. Namely it does not blindly
execute code but prompts the user and lets them know that something is
going on. Further, when it prompts the user it does not provide a
default action, it forces them to answer so that they must attend to
what they are doing. This provides the user with an opportunity to
become aware of a feature of Emacs that is new to them. The only other
option is to never execute local variables until a user reads the
manual and changes their config. Such a design infantilizes the user
and does not empower them.

Emacs isn't just a dissociated tool, or at least it tries not to be
(lots of people also complain about the splash screen being enabled by
default!). The Emacs community often also goes out of its way to share
its knowledge when such situations arise (as we are doing here). We
can't reach everyone (yet!) but many people donate time and effort to
share. Emacs and Org both actively encourage users to participate in
the mailing lists because venues like StackOverflow are not guaranteed
to be seen by the community and are not officially supported.

Finally on this point, if we are willing to open the manual we see
that Emacs tries to educate users about this, the first line of the
section on file local variables is "File-local variables can be
dangerous."

With regard to the local variables prompt. Maybe the right thing to do
in this case would be to add a "?" option so that users can open the
manual? That seems like it would help. That said, the second line of
the prompt reads "contains variables that may not be safe (*)" which
should be alarming. I guess it could be changed to "THIS FILE COULD
PWN YOU. Please review potentially unsafe variables marked with an
asterisk (*)." Or just make the second line a bold warning? I have a
patch I need to send in to make it possible to use lexical scope in
eval local variables, I'll take a look at this while I'm there.

It is forgivable to not be utterly terrified of systems that can
execute arbitrary code, given that they mostly look like rocks, and we
are surrounded by them all day. However, they are utterly terrifying
existences! While I think it would be great for the Emacs splash
screen to come with a big warning that says "WARNING THIS PROGRAM
ENABLES YOU TO EXECUTE ARBITRARY CODE" I admit that it took me nearly
a decade to begin to understand what arbitrary code execution means
and why I should be scared of it. Those words are meaningless to a 12
year old who at best will think "What is the big deal about arbitrary
code? I can already run any code that I want!"

This is why I pointed you to orgstrap. The eval: (org-sbe startup)

Re: consistent behavior across babel languages

2020-11-25 Thread Tom Gillespie
Hi Ian,
Thanks for getting this started. I have been collecting a list of
org babel issues and worg is definitely a better place to put them
than in one big email. Since there are so many different features that
a babel language implementation can support I don't want to try to put
them all in one table quite yet but I think it may be helpful to do so
eventually. Until that time does it make sense to add new sections to
the file that cover specific features? For example I have tests on
TRAMP support and/or support for execution via a remote session that I
would like to add from my internal notes. Can I add it as a new
section? Best!
Tom



On Wed, Nov 25, 2020 at 2:06 AM Tim Cross  wrote:
>
>
> ian martins  writes:
>
> > Something I've found challenging is the inconsistency between babel
> > languages. It makes it difficult for a babel user to get a source
> > block to do what they want, or for a babel developer to even know what
> > correct behavior is.
> >
> > I'm not sure if anything can be done since changes will likely break
> > existing behavior, but it's good to at least know what the rule is and
> > where the exceptions to the rule are. To that end I started a page on
> > worg [1] to document current behavior for actions taken across babel
> > languages.
> >
> > [1] https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
>
>
> this is a great initiative Ian. First step in addressing inconsistencies
> is documenting them. I will try to allocate time on the weekend to
> review what you have and see if there are any I know of which you have
> not included.
>
>
> --
> Tim Cross
>



Re: Bug: :prologue and :epilogue are ignored in ob-sql code blocks (inter alia)

2020-11-24 Thread Tom Gillespie
Tim,
Thank for the report, and the digging for ob-langs that might be
affected. The underlying issue is that prologue and epilogue are part
of both the user facing parts of org babel as well as the internal
language implementation facing code. This is a fundamental design flaw
in org babel that needs to be fixed. I have been working on writing up
and diagraming a potential solution for community feedback, but it is
not quite ready yet. Thus, I'm filing this thread along with the
others that I have been compiling about org babel issues. Best!
Tom

On Tue, Nov 24, 2020 at 6:40 PM Tim Landscheidt  wrote:
>
> With Emacs 27.1/org-mode 9.3, "(org) Environment of a Code
> Block" ends with:
>
> | Inserting headers and footers
> | -
>
> | The ‘prologue’ header argument is for appending to the top of the code
> | block for execution, like a reset instruction.  For example, you may use
> | ‘:prologue "reset"’ in a Gnuplot code block or, for every such block:
> |
> |  (add-to-list 'org-babel-default-header-args:gnuplot
> |   '((:prologue . "reset")))
>
> |Likewise, the value of the ‘epilogue’ header argument is for
> | appending to the end of the code block for execution.
>
> However it appears as if :prologue and :epilogue are ignored
> in ob-sql code blocks:
>
> | #+NAME: test-for-ob-sql
> | #+BEGIN_SRC sql :engine postgresql :results verbatim :prologue "SELECT 1;" 
> :epilogue "SELECT 5;" :cmdline --no-psqlrc -P format=aligned -P footer=on
> |   SELECT 2;
> |   SELECT 3;
> |   SELECT 4;
> | #+END_SRC
>
> | #+RESULTS: test-for-ob-sql
> | #+begin_example
> |  ?column?
> | --
> | 2
> | (1 Zeile)
>
> |  ?column?
> | --
> | 3
> | (1 Zeile)
>
> |  ?column?
> | --
> | 4
> | (1 Zeile)
>
> | #+end_example
>
> It seems that :prologue and :epilogue are only honoured in
> languages that use org-babel-expand-body:generic and a
> (very) few others; especially, the following languages prob-
> ably ignore them (untested):
>
> | [tim@passepartout ~/src/emacs]$ find lisp/org -type f -name ob-\*.el \
> | > -not -exec fgrep -q 'org-babel-expand-body:generic' {} \; \
> | > -not -exec fgrep -q ':prologue' {} \; \
> | > -print
> | lisp/org/ob-C.el
> | lisp/org/ob-J.el
> | lisp/org/ob-abc.el
> | lisp/org/ob-awk.el
> | lisp/org/ob-calc.el
> | lisp/org/ob-clojure.el
> | lisp/org/ob-comint.el
> | lisp/org/ob-css.el
> | lisp/org/ob-ditaa.el
> | lisp/org/ob-dot.el
> | lisp/org/ob-ebnf.el
> | lisp/org/ob-emacs-lisp.el
> | lisp/org/ob-eval.el
> | lisp/org/ob-exp.el
> | lisp/org/ob-fortran.el
> | lisp/org/ob-hledger.el
> | lisp/org/ob-latex.el
> | lisp/org/ob-ledger.el
> | lisp/org/ob-lisp.el
> | lisp/org/ob-lob.el
> | lisp/org/ob-makefile.el
> | lisp/org/ob-matlab.el
> | lisp/org/ob-mscgen.el
> | lisp/org/ob-org.el
> | lisp/org/ob-picolisp.el
> | lisp/org/ob-ref.el
> | lisp/org/ob-sed.el
> | lisp/org/ob-shen.el
> | lisp/org/ob-sql.el
> | lisp/org/ob-sqlite.el
> | lisp/org/ob-table.el
> | lisp/org/ob-stan.el
> | lisp/org/ob-vala.el
>



Re: One vs many directories

2020-11-24 Thread Tom Gillespie
> > That is security issue.
>
> Why is it a security issue? The variables do need to be close to the end
> — 3000 characters is only about 50 lines.

It isn't a security issue by itself. Emacs never automatically runs
eval file local variables unless you have tampered with
enable-local-eval, in which case the tamperin is the security issue
not the existence of the local variables list.

Thus it is only a security issue if you permanently accept that eval
file local variable and then open random org files that use it with a
malicious startup block. An eval file local variable like that which
blindly executes an org babel block should never be permanently
accepted

Best,
Tom



Re: ob-python: import local package into a session

2020-11-24 Thread Tom Gillespie
I have also been dissatisfied with the current options for making
local python libraries accessible in certain org files. The amount of
setup that is required outside the org file itself was too large,
especially if you want someone else who is not intimately familiar
with python to be able to use it. My old solution was to modify the
PYTHONPATH environment variable for the whole Emacs process. However,
after a bit of digging inspired by this thread I now have a solution
that is entirely local: advise ~org-babel-execute:python~ to set a new
PYTHONPATH via the ~process-environment~ dynamic variable and set that
from a buffer local variable for the local additions to PYTHONPATH
along with getenv PYTHONPATH. A working example below. Best!
Tom

#+name: orgstrap
#+begin_src elisp :results none :noweb noexport
(defvar-local local-python-path nil)

(defun advise--obe-python-path (command  args)
  (let ((process-environment
 (or (and local-python-path
  (cons
   (format
"PYTHONPATH=%s"
(concat local-python-path (getenv "PYTHONPATH")))
   process-environment))
 process-environment)))
(apply command args)))

(advice-add #'org-babel-execute:python :around
#'advise--obe-python-path)

(setq-local local-python-path (concat default-directory "code:"))
#+end_src

On Tue, Nov 24, 2020 at 9:26 AM Jack Kamm  wrote:
>
> Joost Kremers  writes:
>
> > I haven't really considered the option to install the utility functions as a
> > package in the virtual environment, because I expect to change and develop 
> > those
> > functions together with the rest of the project. If it were a separate 
> > package,
> > I'd need to reinstall it every time I make changes to it, which will 
> > probably
> > happen often.
>
> If you install the package using either "python setup.py develop", or
> "pip install -e", then Python will install your code via symlinks
> instead of copying, so then you don't have to worry about reinstalling
> every time you make an edit.
>
> To switch between venv's in emacs, I use pyvenv:
> https://github.com/jorgenschaefer/pyvenv
>



Re: Is Org really so simple?

2020-11-23 Thread Tom Gillespie
I have read many perspectives like this of late on this mailing list.
In summary I think that Org is such an incredibly flexible and
powerful tool that many users have not the faintest idea what other
users are doing with it (for example I am completely mystified by the
level of activity in the one vs many files thread and its counterpart
on the orgmode subreddit). Despite this, in a sense, Org is just as
simple as it has always been which is why people build on top of it,
and thus there isn't really any way to "go back" to a simpler time --
such a time is fictional, it has never existed. I can say for myself
that it is not Org that has changed, it is how I use it. I used to use
it in simple ways, and still can if I want, but now I use it as a
substrate for self describing (sometimes self-modifying) interactive
programs -- as complex as you can get.

For some use cases there are performance issues and for others
workflow issues. The performance issues are likely to be resolved via
more developer time. The workflow issues have to be resolved by
writing custom elisp code, in many times that elisp gets packaged as
org extensions. Thus we arrive back at your original complaint, which
is that Org files are not sufficiently self-contained. In order to
make them self-contained you currently have to copy and paste a bunch
of boilerplate into files (thus the workflow issues). For some things
the boilerplate is seemingly unavoidable, even using #+setupfile:
doesn't work if you are trying to solve a problem in a certain way in
Org. Other times, you could solve the same problem without any
boilerplate using a slightly different approach, but not always.

I have been working on an extension for Org (orgstrap
https://github.com/tgbugs/orgstrap) with the goal of making Org files
self-describing, if not fully self-contained (There is a distinction
between the two, but only for certain failure modes. Also, why force
__DATA__ to be embedded with the file when everything has to be
dereferenced at some point anyway? You could embed it later if it fits
your use case, or you could embed the org file in the data!). I came
at the problem from the perspective of reproducible research, but it
seems like there are many use cases for being able to move information
implicit in the environment into an Org file explicitly. If you want a
database to handle relational data, great, describe what you need to
access it in the org file that will act as the interface to that
relational data, that way if the database access is down you won't get
cryptic errors, but a big warning that says "hey! a service required
to use functionality in this file correctly is missing!" For the
described frustration with needing to track users, give the org-file
an embedded id and use that to access and update the assignees
associated with a file. There is no need to choose Org or something
else, you can have your comfortable org-mode workflow, and your
data-appropriate store.



Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
Terry,
   Thank you very much for the clear articulation of the problem,
it enable me to see what the issue is and find more and deeper
issues with the change.

Speaking as someone who was not affected by this change
due to the peculiarities of my config, let me say as a fairly
impartial participant, this change is completely broken and
it is clear upstream Emacs did zero due diligence on the
impact it would have for org-mode. The current implementation
of elastic-indent-mode is obviously broken for use in org.

What do I mean? Currently, electric-intent-mode t is actively
destructive in certain cases. Consider a headline created by *-space
or M-ret. If you then hit return (because you don't know what to call
that headline but have the idea you want to type) electric-intent-mode
will delete the trailing space making it no longer a headline
WHAT?! This is horrible broken behavior in org mode!

At the very least this must be fixed if electric-indent-mode is to be on
by default in org-mode. It is clear that electric-indent-mode as implemented,
is unsuitable for use in org-mode since it actively ignores significant trailing
whitespace.

The sane thing to do is to revert to electric-indent-mode nil at least until
the obvious brokeness is fixed, and even then, why make people undo
a stupid computer decision by typing backspace when they can just
as easily do what they mean by typing tab!??! We don't even have to
poll the community to figure out who is going to be forced to type more.

I don't mean to sound ungrateful to the folks who have worked to match
with upstream, but it is clear that upstream has done zero testing on
the impact of that change on org-mode (or any other mode for that matter).
Until the upstream behavior can be fixed or org can patch the brokenness
this needs to be reverted. Even then, why is the "smart" option that changes
the meaning of the bloody return key enabled by default? I will point back to
https://orgmode.org/list/87ees6fp8r@nicolasgoaziou.fr/ and state that
had I spotted this thread at the time, I would have spoken up to point out
that it would likely not be a welcome change, as this thread shows. The
good news is that all is not lost and now when users want electric-indent-mode
in org it will be consistent with upstream.

Best,
Tom



Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
> > Ugh, I update my emacs package pretty infrequently and I usually have 30 or
> > more packages updating at a time -- I can't see wading through 30 NEWS
> > files searching for landmines...
> >
>
> Yes, this I think is a problem. Most of those packages probably only
> have minor changes and bug fixes. We really need some way to be able to
> sort or grade those packages based on whether they are minor or major
> upgrades. Some sort of metric which would let you gauge the amount of
> change and decide if checking the NEWS file would be advisable.

An aside.

This is one of the reasons that I generally oppose breaking up repositories
into smaller projects. Some users want more frequent updates to some
parts of a codebase, but they are usually outliers. The end result is that
splitting the repo pushes more and more work onto the users, most of
whom receive no benefit since they are not bleeding edgers, because
now it is up to them to determine compatibility, read the NEWS, etc.

Sometimes deeper coupling can actually help. For example in this
thread if Emacs and Org were more tightly coupled in their releases
then the changes to electric indent mode might not have gone through
at all, or might have been delayed until Org could coordinate the change
so that users only had to deal with it once.

Do we actually want Org and Emacs development to be more coupled?
Probably not, but I suspect there is a systematic bias on the part of
more frequent participants to want greater decoupling because they are
often engaged in the development process and thus have a skewed
perspective on what is important --- namely to make development easier.



Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
> If people don't have time to read the NEWS file, I also doubt they would
> be aware of the mini config file or have the time to add it to their
> setup. There would be an additional burden on developers to maintain the
> mini-config which might not actually result in any real benefit. I would
> also be concerned this might setup an expectation which is difficult to
> meet. It may not always be straight-forward to provide such a
> mini-config and sometimes, it may actually be in the best interests of
> the user to force a change on them because it brings other benefits that
> outweigh the initial 'change pain'.

Clearly this would not work for all types of changes. However there are
numerous cases where a variable is changed from t to nil or similar where
such functionality would be straightforward to implement and might be
able to cover 80% of these kinds of issues.

It should be entirely possible to teach users that as part of their upgrade
process there is an option that they can set or command they can invoke
which will automatically make changes to their config to preserve old
default behavior. M-x org-post-update-restore-old-defaults or something.

There are also intermediate changes such as the addition of the :extend
keyword for faces which was given a default value that changed existing
behavior by defaulting to nil instead of t. This kind of change is a
combination of useful, annoying, and hard to fix without reading the
NEWS if it is something that breaks your workflow. Emacs is usually
pretty good about avoiding these kinds of major changes in behavior
but sometimes the slip through and sometimes a change does need
to be made and the best time to do so is as soon as possible.

I suspect that this might be an area where org could benefit from more
extensive testing coverage. There was a change between 9.1 and 9.3 that
completely broke org babel edit src functionality because it did not
correctly restore the window layout on exit. Now we just need to find
people willing to write the tests in addition to notifying the mailing list :D.

Best,
Tom



Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
Right there with you. My primary org file has a section filled with
rage when some default gets changed in org or some other part of
Emacs. The vast majority of the time the underlying change was in the
NEWS. Since there is already a habit of updating the NEWS it doesn't
seem unreasonable to put all those changes somewhere in an elisp file
that could restore old default behavior.

On Mon, Nov 16, 2020 at 2:41 PM Bill Burdick  wrote:
>
> Ugh, I update my emacs package pretty infrequently and I usually have 30 or 
> more packages updating at a time -- I can't see wading through 30 NEWS files 
> searching for landmines...
>
>
> -- Bill
>
>
> On Mon, Nov 16, 2020 at 9:10 PM Tom Gillespie  wrote:
>>
>> Semver is unlikely to help because the question is what is "broken" by
>> a change in version. Semver would likely be about breaking changes to
>> internal org apis, not changes to default behavior that affect users,
>> so you have two different "semantics" which put us right back where we
>> are now -- to know what really changed you have to read the NEWS.
>> Bastien has also talked about hear-ye versioning, which says when a
>> version changes users need to read the news. Best,
>> Tom
>>
>>
>> On Mon, Nov 16, 2020 at 1:15 PM gyro funch  wrote:
>> >
>> > On 11/16/2020 9:26 AM, Tom Gillespie wrote:
>> > > Would it help if major releases maintained a mini-config that if added
>> > > to init.el would allow users to retain old behavior? That way they
>> > > wouldn't have to read the NEWS but could just add the relevant lines,
>> > > or maybe even just call the org-old-default-behavior-9.1 or
>> > > org-old-default-behavior-9.4. The workflow during development would be
>> > > to account for any change to defaults in those functions. Thoughts?
>> > > Tom
>> > >
>> > >
>> >
>> > I hate to open a new can of worms, but could semantic versioning be used
>> > such that it is obvious when there are changes that are not backwards
>> > compatible?
>> >
>> > -gyro
>> >
>> >
>>



Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
Semver is unlikely to help because the question is what is "broken" by
a change in version. Semver would likely be about breaking changes to
internal org apis, not changes to default behavior that affect users,
so you have two different "semantics" which put us right back where we
are now -- to know what really changed you have to read the NEWS.
Bastien has also talked about hear-ye versioning, which says when a
version changes users need to read the news. Best,
Tom


On Mon, Nov 16, 2020 at 1:15 PM gyro funch  wrote:
>
> On 11/16/2020 9:26 AM, Tom Gillespie wrote:
> > Would it help if major releases maintained a mini-config that if added
> > to init.el would allow users to retain old behavior? That way they
> > wouldn't have to read the NEWS but could just add the relevant lines,
> > or maybe even just call the org-old-default-behavior-9.1 or
> > org-old-default-behavior-9.4. The workflow during development would be
> > to account for any change to defaults in those functions. Thoughts?
> > Tom
> >
> >
>
> I hate to open a new can of worms, but could semantic versioning be used
> such that it is obvious when there are changes that are not backwards
> compatible?
>
> -gyro
>
>



Re: TEC: update the new website ML page?

2020-11-16 Thread Tom Gillespie
Here is a patch that might serve for the purpose. Best,
Tom

On Mon, Nov 16, 2020 at 5:47 AM Russell Adams  wrote:
>
> https://orgmode.org/community.html
>
> This really needs to state that the Org-mode mailing list is
> subscriber only. Membership is open, but that users should subscribe
> prior to posting. The Worg page for the ML says that.
>
> As a second request, can we get a link to Worg on the top level bar?
>
> --
> Russell Adamsrlad...@adamsinfoserv.com
>
> PGP Key ID: 0x1160DCB3   http://www.adamsinfoserv.com/
>
> Fingerprint:1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3
>
From 98057ee1b0c008d38fe7b725bc03c4d9af01ee25 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Mon, 16 Nov 2020 12:02:42 -0500
Subject: [PATCH] add Worg to preamble and note about mailing list

---
 community.org   | 3 +++
 resources/preamble.html | 1 +
 2 files changed, 4 insertions(+)

diff --git a/community.org b/community.org
index 846abca..427fa06 100644
--- a/community.org
+++ b/community.org
@@ -15,6 +15,9 @@ community.
 You can [[https://lists.gnu.org/mailman/listinfo/emacs-orgmode][subscribe to the list]] and [[https://orgmode.org/list][browse the archive]] here at
 =orgmode.org= or via the [[https://lists.gnu.org/archive/html/emacs-orgmode/][GNU mailman archive]].
 
+If you are not subscribed to the mailing list your message may be
+delayed because it will need to be approved by the moderators.
+
 You can also check this page for more [[https://orgmode.org/worg/org-web-social.html][Org news on the social web]].
 
 * Chatting about Org
diff --git a/resources/preamble.html b/resources/preamble.html
index 8a952d1..43024e7 100644
--- a/resources/preamble.html
+++ b/resources/preamble.html
@@ -30,6 +30,7 @@ fathom('trackPageview');
 https://updates.orgmode.org;>Updates
 Install
 Manual
+Worg
 Community
 Contribute
 

Re: Changed list indentation behavior: how to revert?

2020-11-16 Thread Tom Gillespie
Would it help if major releases maintained a mini-config that if added
to init.el would allow users to retain old behavior? That way they
wouldn't have to read the NEWS but could just add the relevant lines,
or maybe even just call the org-old-default-behavior-9.1 or
org-old-default-behavior-9.4. The workflow during development would be
to account for any change to defaults in those functions. Thoughts?
Tom



Re: official orgmode parser

2020-11-12 Thread Tom Gillespie
Hi Bastien,
 I agree it would be great to ask them to contribute to whichever
ruby library they are using. I will see if I can get in touch, but I
have no idea of where to start if we really want to get to the folks
who could make a decision. It looks like gitlab uses the same org-ruby
library as well
https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/Gemfile#L156.
They may be easier to reach out to. I have also cced Wally to see if
he has any insights here. Best!
Tom



  1   2   >