Re: Thoughts on the standardization of Org

[TEC]

Hi all,

Following what I've read on the list I've developed thoughts on 
what the
best approach might be. My current thinking is that it may be 
possible
to have Org registered as a standard in such a way that it does 
not

constrain our development efforts.

How?

We forgo locking down the precise format and behaviour of every 
Org

element. Instead we only submit something like
https://orgmode.org/worg/dev/org-syntax.html which /just/ 
describes the
overall syntax. I don't imagine that 'locking' ourself into the 
current
syntax described in https://orgmode.org/worg/dev/org-syntax.html 
would
actually hurt development, but might be enough for a MIME type 
etc.


Then perhaps just say for description of how specific/special 
instances
of those structural elements are supposed to work see the 
reference

implementation.

Just a few thoughts from me.

All the best,
Timothy.

Asa Zeren  writes:


Hi,

Even though I am new to the org-mode community, I would like to 
share
some thoughts on the specification of org-mode, especially since 
I
have seen some recent discussion of it in relation to 
registering org

as a MIME type.

First, I would like to repeat the importance of developing 
standards
for org-mode. If we want to expand the influence of org, tooling 
must
expand beyond Emacs. While Emacs is an amazing tool, (a) we 
cannot
convince the entire world to use Emacs and (b) org-mode should 
be
integrated into tooling unrelated to text editing, and is 
outside of
the Emacs-Lisp environment. Without additional org 
implementations,
this is impossible. If org catches on before it is standardized, 
we
end up in the situation of Markdown, with many competing 
standards and

non-standards. Hence, standardization is essential.

Standardizing org is much harder than standardizing something 
like
Markdown, but I think by breaking it down as follows will 
maximize the

portability of org while not compromising on development of org.

I see three areas of standardization, which I think should be
standardized separately:
 - Org DOM
 - Org Syntax
 - Org Standard Environments

Before we get to that, a brief note on /how/ I think that org 
should
be specified. I think that org should be specified in terms of 
an
/environment/ that defines the properties, etc. that can be used 
in a
document. For instance, the org standard would say something to 
the
effect of "An environment may specify block bounding keywords 
that may
be used like #+\n...#+. and the environment would 
specify

"begin_src and end_src are a pair of block bounding keyword that
indicates a source code block." This is for two reasons. First, 
this

allows for development of org tool features independent of the
standard. Second, this separates the individual features of org 
mode

from the overall structure.

Org DOM:
The first thing to specify is the org DOM. (Maybe a different 
name

should be used to avoid confusion with the HTML DOM) This is the
structure of an org-mode document, without the textual
representation. Many org-related tools operate on org documents
without needing to use the textual representation. Specifying 
the DOM
separately would (a) create a separation of concerns and (b) 
allow for

better libraries built around org mode.

Org Syntax:
This would be specifying the mapping between the DOM and the 
textual

representation, specified in terms of an environment.

Org Standard Environments:
This is how I would specify elements such as 
#+begin_src..#+end_src
would be specified, as standardized elements of the environment. 
This
would be structured as a number of individual standard 
environments,
such as "Source Blocks" or "Standard Header Properties" 
(specifying

#+title, #+author, etc.)

I would appreciate thoughts on these ideas about how to develop 
and

org specification.

Thanks for reading,
Asa Zeren

Re: Thoughts on the standardization of Org

[Tom Gillespie]

Hi Asa,
My general take is that any active work toward standardization
would be premature. At the very least a full implementation outside
of Emacs would need to exist. In the absence of that there is little
point to standardization. There is ample existing documentation to
build a compliant parser (pandoc exists as well ...) and any effort
toward standardization right now would be better spent improving
the existing implementation or fixing broken ones (e.g. org-ruby).

>From your comments, I would suggest reading through
https://orgmode.org/worg/dev/org-syntax.html if you have not
done so already. Much of what you mention is already there.

If something like standardization is still desired, I would suggest that
the proper framing for any such activities would be as improvement and
clarification in the documentation, and potentially as formalization of some
of the existing behaviors of the system. Org is a fairly stable system,
and as others have said, explicitly leaving things open an unspecified
would be vital.

There are also parts of org (e.g. babel) where the behavior needs to be
regularized and made consistent. At the moment those areas need
contributors, not standardization.

A few more thoughts in line. Best!
Tom

On Sat, Oct 31, 2020 at 8:22 PM Asa Zeren  wrote:

> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.

The situation for Org is not comparable to markdown. There is a single reference
implementation for org at the moment. The codebase is massive. There are many
existing parsers for org files. Many are obviously broken since they
do not match the reference
implementation's behavior. The obviousness is a sign that there is not a need
for standardization at this time. Further, there is little risk that
another impl will
be created without interoperating with the elisp implementation. For example,
consider Mauro's use case: being able to get colleagues who do not use Emacs
to use Org. I suspect most of the people who would be working on other
implementations
would be starting from Emacs and would be unlikely to leave. Also
unlike markdown,
html export is just one tiny part of Org, whereas markdown was
implemented repeatedly
to allow text input on web pages where people needed to implement
parts of html that
had not already been specified in markdown.

> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.

See some of my other recent emails. In the short term this is impossible
due to the deep dependence on Emacs Lisp. Any outside implementation
that is created today would have to implement elisp. Few have been able
to do this in over 30 years. Moving beyond elisp requires additional machinery
to be added to org to be able to specify other top level langauges. This is
not something that is remotely ready for standardization because no one
even has a single working implementation yet!

> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
No. This is an implementation detail (see below for more).

>  - Org Syntax
This is pretty much done, there are some outstanding points for discussion,
but they are about implementation details, not about the contents of the
syntax. Also extension of the syntax needs to be open and defined entirely
by the elisp implementation, as mentioned by others.

>  - Org Standard Environments
Read https://orgmode.org/worg/dev/org-syntax.html. It will get you up to speed
with the existing terminology that is used in the community.

>
> Org DOM:
> The first thing to specify is the org DOM. (Maybe a different name
> should be used to avoid confusion with the HTML DOM) This is the
> structure of an org-mode document, without the textual
> representation. Many org-related tools operate on org documents
> without needing to use the textual representation. Specifying the DOM
> separately would (a) create a separation of concerns and (b) allow for
> better libraries built around org mode.

Depending on exactly what you mean by DOM this does not need to be standardized.
There are a couple of points that need to be clarified regarding how
to treeify the flat
list of elements that come out of a parse in order to tie things like
associated keywords
to the correct elements, but these are quite minimal. The potential
rats nest that is
trying to standardize a DOM when it is an implementation detail means
that I would
strongly discourage even thinking about Org in that way. I would even
discourage putting
too much emphasis on the org-element api which, while extremely useful
inside Emacs,
is not something that should be standardized because it is a detail
peculiar to the elisp
implementation.

There are cases where certain behaviors, su

Re: Thoughts on the standardization of Org

[Pankaj Jangid]

Asa Zeren  writes:

> In these concerns I see one major flaw. The way they are worded at
> present implies that the Emacs implementation of org is the "one true
> implementation," and that all tools in other environments are
> auxiliary.

At present, that is the truth. Where are other implementations that we
can call true implementations? May be I am not aware of them. Could you
please list a few?

> I believe that if we want org to grow, then it needs to become unbound
> from Emacs. It should become a universal markup format, which just
> happens to have had many tools first implemented for Emacs (even if
> Emacs still will probably remain the best way to edit org files).

I would love to see org becoming a universal markup format. My wish is
that the standard should be under control of Worg. This should be the
only source of truth and implementors may contribute to the standard by
participating in the community or may be some steering committee here.

Re: export select tag behavior and documentation

[Samuel Wales]

huh, in exporting the email, i found a possible export bug.

here is the original.  please reply to this one.

notice that a whole paragraph got deleted in the previous email.  that
is the exported version.  dunno why.


9.4 maint docstring

===
org-export-select-tags is a variable defined in ‘ox.el’.

...

If any such tag is found in a buffer, all trees that do not carry
one of these tags will be ignored during export.
===

1) does this take into account subtree export, region
   export, and narrowed buffers?  if so, perhaps the
   docstring should change.

   my expectation is that just having :export: in a huge
   buffer should not affect exporting a tiny part of it.
   by the principle of least surprise.

   it is sorta surprising enough that something buried in a
   subtree can affect the whole export of that subtree.  :)

2) org-export-with-tasks set to

   - nil will not export a task subtree regardless of :export: tag underneath
   - t will export regardless

   thus basically overriding the export tag feature with the
   export with tasks feature.  is that intended?  i can see
   that it could be the best choice, but could also see it
   being useful to use the export tag even with tasks.

   i wonder when was the last time there was a change to
   this variable or its behavior?  i suspect there might
   have been a change, at least wrt top level todo kw
   stopping export, but i don't know.


On 10/31/20, Samuel Wales  wrote:
> [2020-10-31 Sat 20:59]  =x
>
> bug
> ===
>
>   9.4 maint docstring
>
>   === org-export-select-tags is a variable defined in
>   ‘ox.el’.
>
>   ...
>
>
> 1) does this take into account subtree export, region
>export, and narrowed buffers?  if so, perhaps the
>docstring should change.
>
>my expectation is that just having :export: in a huge
>buffer should not affect exporting a tiny part of it.
>by the principle of least surprise.
>
>it is sorta surprising enough that something buried
>in a subtree can affect the whole export of that
>subtree.  :)
>
> 2) org-export-with-tasks set to
>
>- nil will not export a task subtree regardless of
>  :export: tag underneath
>- t will export regardless
>
>thus basically overriding the export tag feature with
>the export with tasks feature.  is that intended?  i
>can see that it could be the best choice, but could
>also see it being useful to use the export tag even
>with tasks.
>
>i wonder when was the last time there was a change to
>this variable or its behavior?  i suspect there might
>have been a change, at least wrt top level todo kw
>stopping export, but i don't know.
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

export select tag behavior and documentation

[Samuel Wales]

[2020-10-31 Sat 20:59]  =x

bug
===

  9.4 maint docstring

  === org-export-select-tags is a variable defined in
  ‘ox.el’.

  ...


1) does this take into account subtree export, region
   export, and narrowed buffers?  if so, perhaps the
   docstring should change.

   my expectation is that just having :export: in a huge
   buffer should not affect exporting a tiny part of it.
   by the principle of least surprise.

   it is sorta surprising enough that something buried
   in a subtree can affect the whole export of that
   subtree.  :)

2) org-export-with-tasks set to

   - nil will not export a task subtree regardless of
 :export: tag underneath
   - t will export regardless

   thus basically overriding the export tag feature with
   the export with tasks feature.  is that intended?  i
   can see that it could be the best choice, but could
   also see it being useful to use the export tag even
   with tasks.

   i wonder when was the last time there was a change to
   this variable or its behavior?  i suspect there might
   have been a change, at least wrt top level todo kw
   stopping export, but i don't know.

Re: Thoughts on the standardization of Org

[Pankaj Jangid]

Tim Cross  writes:

> I should state up-front, I am somewhat sceptical regarding an org-mode
> which is separate or independent of Emacs. Much of what makes org-mode
> so powerful and useful is due to features of Emacs. While most, if not
> all, of these features could be implemented in other solutions, the
> amount of work and level of maintenance should not be under-estimated.

Not only that. This will really slowdown the evolution of org-mode if
the standardization is outside the control of Emacs Developers.

> Over the last 30+ years involved in technology, I have seen many good
> ideas come undone as the result of a standardisation effort. consider
> for example, the results of the lisp standardisation that resulted in
> common Lisp, what happened with CORBA, xml-rpc and the move to REST
> based APIs. XHTML and the breakdown of standardisation processes
> within the W3C and the development of HTML5.

I am not only skeptical, I totally believe that this sort of
standardization where some other party is giving org-mode a certificate,
will be harmful for the development of org-mode.

> The four areas which I think would provide the greatest benefit would
> be to
>
> - Finalise the draft org syntax document on Worg, possibly adding it
> to
>   the manual once complete. A considerable amount of work has already
>   been put into this document and I think it is a good start.

And this should be the only source of truth. If MIME type registration
changes this then better to avoid that.

> - Define a specification for a property API which compliant org-mode
>   implementation should support. This could be based on the existing
>   ELisp mapping API.
>
> - Define a specification for an element mapping API which compliant
>   org-mode implementation should support. Again, this could be based
>   on the existing ELisp element mapping API.
>
> - Define a set of org reference documents. These would be documents
> that
>   all compliant parsers should be able to process successfully. It
>   might also be worthwhile including some documents with common errors
>   which parses should be able to handle and recover from in a graceful
>   manner.  Those developing external tools can then use these
>   documents as a guide and for testing their implementations.

Agree. And again... the standard place should be Worg only.

-- Pankaj

Re: Thoughts on the standardization of Org

[Asa Zeren]

On Sat, Oct 31, 2020 at 8:40 PM Dr. Arne Babenhauserheide
 wrote:
> The most important point I see here is to avoid hindering the
> development of org-mode within Emacs.

While I definitely support enabling the further development of org-mode, and not
restricting it via a standard, I do see some problems.

> So the most important part of the standard would be areas it doesn’t
> standardize: Reserved for future use in org-mode.

The issue with this is that by picking what areas to reserve, one has
effectively limited the syntactic space that new features can use. This is not a
problem in and of itself, but does make the notion of leaving arbitrary syntax
space reserved impossible, particularly, since in org-mode and similar markup
languages, unadorned text is part of the content, rather than being ill formed,
as in programming languages. This also does not mean that tools can interpret
part of what org-mode considers content as having some domain or implementation
specific meaning. For example, latex blocks. In my opinion, the translation of
these are a language extension by the org-export tool. Even within parts of the
Emacs org implementation, latex blocks should not be considered part of the org
language. For example, the line "* Headline?" in the example below is still
identified as a headline, even though, if the area inside the \begin and \end
commands were supposed to be latex, not org, it should not be.

#+begin_example org
\begin{equation}
* Headline?
\end{equation}
#+end_example

> The most important point I see here is to avoid hindering the development of
> org-mode within Emacs.

> These would then be sections that external tools must handle as
> opaque text so their processing does not break usage within
> org-mode.

In these concerns I see one major flaw. The way they are worded at present
implies that the Emacs implementation of org is the "one true implementation,"
and that all tools in other environments are auxiliary. I believe that if we
want org to grow, then it needs to become unbound from Emacs. It should become a
universal markup format, which just happens to have had many tools first
implemented for Emacs (even if Emacs still will probably remain the best way to
edit org files).

Best,

Asa

On Sat, Oct 31, 2020 at 8:40 PM Dr. Arne Babenhauserheide
 wrote:
>
>
> Asa Zeren  writes:
> > I would appreciate thoughts on these ideas about how to develop and
> > org specification.
>
> The most important point I see here is to avoid hindering the
> development of org-mode within Emacs.
>
> So the most important part of the standard would be areas it doesn’t
> standardize: Reserved for future use in org-mode.
>
> These would then be sections that external tools must handle as opaque
> text so their processing does not break usage within org-mode.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken

Re: Thoughts on the standardization of Org

[Tim Cross]

I think there are a couple of important points to consider in
discussions of this type. I should state up-front, I am somewhat
sceptical regarding an org-mode which is separate or independent of
Emacs. Much of what makes org-mode so powerful and useful is due to
features of Emacs. While most, if not all, of these features could be
implemented in other solutions, the amount of work and level of
maintenance should not be under-estimated.

While standards and formal standardisation of something can be
important, it is often a dual edge sword. Over the last 30+ years
involved in technology, I have seen many good ideas come undone as the
result of a standardisation effort. consider for example, the results of
the lisp standardisation that resulted in common Lisp, what happened
with CORBA, xml-rpc and the move to REST based APIs. XHTML and the
breakdown of standardisation processes within the W3C and the
development of HTML5. Sometimes, we can emerge from a standardisation
process with a clearer, consistent standard that is easy to implement
and use. Other times, we can emerge with a complex, difficult to
implement and confusing standard which can kill or stifle further
development. The trick with standardisation seem to be getting the
balance right between clarity and complexity and focusing on the key
requirements, avoiding the trap of trying to cover everything.

One of the things I like a lot about org-mode is that it is not terribly
prescriptive. It provides a collection of features and functions which
you are able to assemble according to your own needs and preferences.
While I think it is important to have a clear idea of basic syntax for
each of these elements and how they relate to each other, I'm less
convinced we would want to prescribe an overly formal specification for
how an org document should be structured i.e. the org DOM idea. The
existing draft syntax document is probably sufficiently prescriptive
here already.

The four areas which I think would provide the greatest benefit would
be to

- Finalise the draft org syntax document on Worg, possibly adding it to
  the manual once complete. A considerable amount of work has already
  been put into this document and I think it is a good start.

- Define a specification for a property API which compliant org-mode
  implementation should support. This could be based on the existing
  ELisp mapping API.

- Define a specification for an element mapping API which compliant
  org-mode implementation should support. Again, this could be based on
  the existing ELisp element mapping API.

- Define a set of org reference documents. These would be documents that
  all compliant parsers should be able to process successfully. It might
  also be worthwhile including some documents with common errors which
  parses should be able to handle and recover from in a graceful manner.
  Those developing external tools can then use these documents as a
  guide and for testing their implementations.

Tim

Asa Zeren  writes:

> Hi,
>
> Even though I am new to the org-mode community, I would like to share
> some thoughts on the specification of org-mode, especially since I
> have seen some recent discussion of it in relation to registering org
> as a MIME type.
>
> First, I would like to repeat the importance of developing standards
> for org-mode. If we want to expand the influence of org, tooling must
> expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
> convince the entire world to use Emacs and (b) org-mode should be
> integrated into tooling unrelated to text editing, and is outside of
> the Emacs-Lisp environment. Without additional org implementations,
> this is impossible. If org catches on before it is standardized, we
> end up in the situation of Markdown, with many competing standards and
> non-standards. Hence, standardization is essential.
>
> Standardizing org is much harder than standardizing something like
> Markdown, but I think by breaking it down as follows will maximize the
> portability of org while not compromising on development of org.
>
> I see three areas of standardization, which I think should be
> standardized separately:
>  - Org DOM
>  - Org Syntax
>  - Org Standard Environments
>
> Before we get to that, a brief note on /how/ I think that org should
> be specified. I think that org should be specified in terms of an
> /environment/ that defines the properties, etc. that can be used in a
> document. For instance, the org standard would say something to the
> effect of "An environment may specify block bounding keywords that may
> be used like #+\n...#+. and the environment would specify
> "begin_src and end_src are a pair of block bounding keyword that
> indicates a source code block." This is for two reasons. First, this
> allows for development of org tool features independent of the
> standard. Second, this separates the individual features of org mode
> from the overall structure.
>
> Org DOM:
> The first thing to speci

Re: Please help by becoming a maintainer for an Org Babel file

[Corwin Brust]

Hi Bastien!

On Mon, Oct 26, 2020 at 3:07 AM Bastien  wrote:
>
> Dear all,
>
> we are looking for more maintainers of individual Org Babel files.
>
> Jack and Ian are already in, I added myself to ob-clojure.el.
>
> If you feel like proposing yourself for maintaining an Org Babel
> language, that would be super helpful.


I would like to help maintain ob-perl.

(My paperwork isn't completed yet; I do have the form and am working
with my employer.)

Regards,
Corwin

Re: Thoughts on the standardization of Org

[Dr. Arne Babenhauserheide]

Asa Zeren  writes:
> I would appreciate thoughts on these ideas about how to develop and
> org specification.

The most important point I see here is to avoid hindering the
development of org-mode within Emacs.

So the most important part of the standard would be areas it doesn’t
standardize: Reserved for future use in org-mode.

These would then be sections that external tools must handle as opaque
text so their processing does not break usage within org-mode.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken


signature.asc
Description: PGP signature

Thoughts on the standardization of Org

[Asa Zeren]

Hi,

Even though I am new to the org-mode community, I would like to share
some thoughts on the specification of org-mode, especially since I
have seen some recent discussion of it in relation to registering org
as a MIME type.

First, I would like to repeat the importance of developing standards
for org-mode. If we want to expand the influence of org, tooling must
expand beyond Emacs. While Emacs is an amazing tool, (a) we cannot
convince the entire world to use Emacs and (b) org-mode should be
integrated into tooling unrelated to text editing, and is outside of
the Emacs-Lisp environment. Without additional org implementations,
this is impossible. If org catches on before it is standardized, we
end up in the situation of Markdown, with many competing standards and
non-standards. Hence, standardization is essential.

Standardizing org is much harder than standardizing something like
Markdown, but I think by breaking it down as follows will maximize the
portability of org while not compromising on development of org.

I see three areas of standardization, which I think should be
standardized separately:
 - Org DOM
 - Org Syntax
 - Org Standard Environments

Before we get to that, a brief note on /how/ I think that org should
be specified. I think that org should be specified in terms of an
/environment/ that defines the properties, etc. that can be used in a
document. For instance, the org standard would say something to the
effect of "An environment may specify block bounding keywords that may
be used like #+\n...#+. and the environment would specify
"begin_src and end_src are a pair of block bounding keyword that
indicates a source code block." This is for two reasons. First, this
allows for development of org tool features independent of the
standard. Second, this separates the individual features of org mode
from the overall structure.

Org DOM:
The first thing to specify is the org DOM. (Maybe a different name
should be used to avoid confusion with the HTML DOM) This is the
structure of an org-mode document, without the textual
representation. Many org-related tools operate on org documents
without needing to use the textual representation. Specifying the DOM
separately would (a) create a separation of concerns and (b) allow for
better libraries built around org mode.

Org Syntax:
This would be specifying the mapping between the DOM and the textual
representation, specified in terms of an environment.

Org Standard Environments:
This is how I would specify elements such as #+begin_src..#+end_src
would be specified, as standardized elements of the environment. This
would be structured as a number of individual standard environments,
such as "Source Blocks" or "Standard Header Properties" (specifying
#+title, #+author, etc.)

I would appreciate thoughts on these ideas about how to develop and
org specification.

Thanks for reading,
Asa Zeren

Re: Org-Mode as DSL

[Mauro Mandracchia]

Thanks for the warm welcome and the amount of inputs and details that
everybody so fast provides.
I'm really impressed by this community.

Ok seems that delivering a language specification in EBNF has some
challenges that can hardly be addressed due to the nature of org-mode
design, but I read around that often EBNF is not the silver bullet to solve
a grammar representation although it is a de-facto standard.

So I will try to put some orders in my thoughts, when I look at
functionalities of org-mode I will split them in the following categories:

   - Grammar Specification
   - Metadata
  - TODO
  - Code Block

  - Tangling (or waving for the web people)
   - Properties of the file
  - Templating

  - Block Execution
   - Environment (Global variables such as PATH)
  - Language
  - Context for arguments

  - Publishing Functions
  - Target Format
  - Custom Parser

I share a lot with of your ideas and inputs, here some observations and
options that I noticed:

   - Indeed emacs is not that big and could run as a background process to
   deliver org-mode features to other systems.
   I do something very similar to *TEC* for my CI/CD and my static
   generated website, but I'm not sure I would run emacs as a web service.

   - LSP doesn't mention org-mode as available language, but it does for
   markdown
   
https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocumentItem
   maybe because org-mode is not a language?

   - I share the sentiment that some api are a bit obscure and most
   important there isn't for instance an official way to say "this is how you
   create extensions in org".

   - I found that multiple people are already converting org-mode into json
   by using org*-map-entries *which indeed is not very publicly shown (I
   miss something like "this is how you create extensions in org").

   - I relate a lot to the issues that *Tom Gillespie* discusses in his
   post (
   https://lists.gnu.org/archive/html/emacs-orgmode/2020-10/msg00244.html)
   about tangling and executions and I loved to see your efforts in trying to
   solving them.

   - Since gccemacs elisp could compile to C (not sure this is relevant,
   but can be investigated if there are opportunities to be capable to have
   org-mode embendable).


For my personal needs I think I would pragmatically start toying around
with a org-mode to json like here https://github.com/ludios/org-to-json
And once I gain some confidence on the output JSON, I think I will gain
more confidence about the grammar definition.

I think that converting org-mode to JSON could show org-mode as an Abstract
Syntax Tree, I think from there it should be easier to create a
specification and maybe is what already *orgajs
* already did.
I believe creating functions like tangle and execution shouldn't be that
trivial from an AST.

Concluding,
there is overlap between the Interfaces of LSP and org-mode which makes a
lot of sense since both target the interaction for with a document eg:

   - Propose actions, these can be:
  - Execute
  - Tangle
  - Change state TODO
  - Insert date, tag, etc...
   - Go to definition (For linking sections)
   - Support multi language in the same document

This makes me think that creating an org-mode LSP is not that crazy idea
after all, what do you think?
If you would build an LSP for org-mode which setup would you choose? What
do you think would be hard to implement?

Again many thanks 🙏
Mauro.

Re: Bug: headline folding broken by whitespace

[Kyle Meyer]

Russell Adams writes:

> On Sat, Oct 31, 2020 at 01:58:25PM -0400, Kyle Meyer wrote:
>> As I mentioned elsewhere in the thread, my guess is that Russell was
>> using your example as is, rather than adding the space on the line
>> (where X is below).
>
> No. I added a space, and even used highlight-regexp to confirm all
> spaces.
>
> It did not occur for me.

Well, then I'm out of guesses.

In any case, for those who can trigger the issue (using the example from
this thread as well as the examples from the related threads) on
ee3c3b554 up through 47bf6005b (as I can, with a vanilla Emacs 27.1),
e8070d71a should resolve it.

Re: Bug: headline folding broken by whitespace

[Russell Adams]

On Sat, Oct 31, 2020 at 01:58:25PM -0400, Kyle Meyer wrote:
> As I mentioned elsewhere in the thread, my guess is that Russell was
> using your example as is, rather than adding the space on the line
> (where X is below).

No. I added a space, and even used highlight-regexp to confirm all
spaces.

It did not occur for me.


--
Russell Adamsrlad...@adamsinfoserv.com

PGP Key ID: 0x1160DCB3   http://www.adamsinfoserv.com/

Fingerprint:1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Bug: headline folding broken by whitespace

[Kyle Meyer]

Daniele Nicolodi writes:

> This is funny. I have two emacs instances running: one with my
> customizations and an emacs -Q one. The first exhibits the issue, the
> second does not

If your emacs -Q session is just using the Org that comes with Emacs
27.1, it won't have the commit that introduced the regression: ee3c3b554
(org.el: Allow empty subtrees to be folded back, 2020-09-05).  That was
part of the 9.3.8 release.

> matching what Russel reports.

As I mentioned elsewhere in the thread, my guess is that Russell was
using your example as is, rather than adding the space on the line
(where X is below).

--8<---cut here---start->8---
* Foo
** Bar
   bar bar bar+49 1520 7178926

X
* Baz
--8<---cut here---end--->8---

If I use that example (with no custom configuration), the issue
(i.e. Foo won't unfold) starts with ee3c3b554 and goes away when that
commit is reverted in e8070d71a.

Re: Inconsistency between code and manual: org-lowest-priority or org-priority-lowest

[Kyle Meyer]

Daniele Nicolodi writes:

> On 30/10/2020 05:57, Kyle Meyer wrote:

>> The org-X-priority -> org-priority-X rename happened in v9.4, with
>> org-X-priority names retained as aliases.  So, it sounds like there are
>> some leftover bits in the code.
>
> You are absolutely right. This is what you get when you read the manual
> for the latest version but look at the code for an old one...

Quickly grepping, a few instances of the old names remain in the code
base, if you're still interested in sending a patch.

Re: Bug: headline folding broken by whitespace

[Daniele Nicolodi]

On 31/10/2020 17:49, Kyle Meyer wrote:
> Daniele Nicolodi writes:
> 
>> Hello,
>>
>> i saw a few reports of bugs related to folding and unfolding of
>> headlines recently, but I think this is not a duplicate.
> 
> I think you're referring to
> 
> https://orgmode.org/list/87h7qu10ht@eml.cc
> 
> https://orgmode.org/list/CAJj=8EFJXFu9ZQ11cO24AAu=1jykW2Zr+=R4FLuKG=bljfu...@mail.gmail.com
> 
> The cases reported there are resolved by e8070d71a (Revert "org.el:
> Allow empty subtrees to be folded back", 2020-10-26).
> 
>> I just noticed that with the latest release (Org mode version 9.4
>> (9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
>> unfolding of headlines is broken by whitespace. In particular with a
>> buffer with this content:
>>
>> * Foo
>> ** Bar
>>bar bar bar+49 1520 7178926
>>
>>
>> * Baz
>>
>> where the line before "* Baz" contains a single whitespace character,
>> the headline "* Foo" can be folded but cannot be unfolded. Removing the
>> whitespace while the headline is folded fixes the problem.
> 
> Yep, with your test case (making sure to add a space on the line before
> Baz), I can trigger the issue on your reported version (47bf6005b), and
> it goes away with e8070d71a.  So, it looks like the same underlying
> problem and should go away with the next elpa bump.

This is funny. I have two emacs instances running: one with my
customizations and an emacs -Q one. The first exhibits the issue, the
second does not, matching what Russel reports.

By the way, how can I get a dump of org configuration similar to the one
org-submit-bug-report produces?

Cheers,
Dan

Re: Bug: headline folding broken by whitespace

[Kyle Meyer]

Russell Adams writes:

> On Sat, Oct 31, 2020 at 04:06:22PM +0100, Daniele Nicolodi wrote:
>> I just noticed that with the latest release (Org mode version 9.4
>> (9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
>> unfolding of headlines is broken by whitespace. In particular with a
>> buffer with this content:
>>
>> * Foo
>> ** Bar
>>bar bar bar+49 1520 7178926
>>
>>
>> * Baz
>>
>> where the line before "* Baz" contains a single whitespace character,
>> the headline "* Foo" can be folded but cannot be unfolded. Removing the
>> whitespace while the headline is folded fixes the problem.
>
> I'm on Org mode version 9.4 (9.4-27-g47bf60-elpa) and tested, cannot
> reproduce. Do you have something conflicting, or are you running an
> Emacs distribution (ie: Doom, Space)?

If you try again, adding SPC on the line before Baz, you should see it
on the Org version you report.

Re: Bug: headline folding broken by whitespace

[Kyle Meyer]

Daniele Nicolodi writes:

> Hello,
>
> i saw a few reports of bugs related to folding and unfolding of
> headlines recently, but I think this is not a duplicate.

I think you're referring to

https://orgmode.org/list/87h7qu10ht@eml.cc

https://orgmode.org/list/CAJj=8EFJXFu9ZQ11cO24AAu=1jykW2Zr+=R4FLuKG=bljfu...@mail.gmail.com

The cases reported there are resolved by e8070d71a (Revert "org.el:
Allow empty subtrees to be folded back", 2020-10-26).

> I just noticed that with the latest release (Org mode version 9.4
> (9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
> unfolding of headlines is broken by whitespace. In particular with a
> buffer with this content:
>
> * Foo
> ** Bar
>bar bar bar+49 1520 7178926
>
>
> * Baz
>
> where the line before "* Baz" contains a single whitespace character,
> the headline "* Foo" can be folded but cannot be unfolded. Removing the
> whitespace while the headline is folded fixes the problem.

Yep, with your test case (making sure to add a space on the line before
Baz), I can trigger the issue on your reported version (47bf6005b), and
it goes away with e8070d71a.  So, it looks like the same underlying
problem and should go away with the next elpa bump.

Re: Bug: headline folding broken by whitespace

[Daniele Nicolodi]

On 31/10/2020 17:28, Russell Adams wrote:
> On Sat, Oct 31, 2020 at 04:06:22PM +0100, Daniele Nicolodi wrote:
>> I just noticed that with the latest release (Org mode version 9.4
>> (9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
>> unfolding of headlines is broken by whitespace. In particular with a
>> buffer with this content:
>>
>> * Foo
>> ** Bar
>>bar bar bar+49 1520 7178926
>>
>>
>> * Baz
>>
>> where the line before "* Baz" contains a single whitespace character,
>> the headline "* Foo" can be folded but cannot be unfolded. Removing the
>> whitespace while the headline is folded fixes the problem.
> 
> I'm on Org mode version 9.4 (9.4-27-g47bf60-elpa) and tested, cannot
> reproduce. Do you have something conflicting, or are you running an
> Emacs distribution (ie: Doom, Space)?

Thank you for trying to reproduce the issue. Indeed if I try to
reproduce it with an Emacs without my customizations it works just fine.

I'm using Emacs 27.1 from here https://emacsformacosx.com/ I'm surprised
something in my configuration breaks this: I don't use anything fancy.

Cheers,
Dan

Re: Bug: headline folding broken by whitespace

[Russell Adams]

On Sat, Oct 31, 2020 at 04:06:22PM +0100, Daniele Nicolodi wrote:
> I just noticed that with the latest release (Org mode version 9.4
> (9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
> unfolding of headlines is broken by whitespace. In particular with a
> buffer with this content:
>
> * Foo
> ** Bar
>bar bar bar+49 1520 7178926
>
>
> * Baz
>
> where the line before "* Baz" contains a single whitespace character,
> the headline "* Foo" can be folded but cannot be unfolded. Removing the
> whitespace while the headline is folded fixes the problem.

I'm on Org mode version 9.4 (9.4-27-g47bf60-elpa) and tested, cannot
reproduce. Do you have something conflicting, or are you running an
Emacs distribution (ie: Doom, Space)?

--
Russell Adamsrlad...@adamsinfoserv.com

PGP Key ID: 0x1160DCB3   http://www.adamsinfoserv.com/

Fingerprint:1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Bug: headline folding broken by whitespace

[Daniele Nicolodi]

Hello,

i saw a few reports of bugs related to folding and unfolding of
headlines recently, but I think this is not a duplicate.

I just noticed that with the latest release (Org mode version 9.4
(9.4-27-g47bf60-elpa @ /Users/daniele/.emacs.d/elpa/org-20201026/)
unfolding of headlines is broken by whitespace. In particular with a
buffer with this content:

* Foo
** Bar
   bar bar bar+49 1520 7178926


* Baz

where the line before "* Baz" contains a single whitespace character,
the headline "* Foo" can be folded but cannot be unfolded. Removing the
whitespace while the headline is folded fixes the problem.

Cheers,
Dan

Re: Is reading nested simple lists into org-babel code blocks currently supported? [PATCH] - manual.org

[James Boyle]

Hi Tom (orgmode list cc'ed),

> As a result
> each ob-lang will probably have to deal with this itself, or at least
> they will have to define how to consume quoted elisp lists without
> raising errors. 

Thanks -- this is exactly the judgment check I was looking for -- if this type 
of support should fall on individual languages
or not. 

To this end, I've attached a TINYCHANGE patch for manual.org that would have 
helped me in my confusion. The change updates the example to reflect the 
current state of the output, and clarifies the level of support.

Thanks for any help you can provide.

James

On Thu, Oct 29, 2020, at 12:52 PM, Tom Gillespie wrote:
> Hi James,
> I would file this along with other ob-lang dependent features such
> as TRAMP remote execution support. For example, python works as in
> your other examples, but only if you define :var unordered="unordered"
> as another variable before the variable you pass the plain list to
> because python tries to dereference the symbol. A global solution like
> the one you propose breaks the semantics in cases where a language
> does correctly deal with quoted lists (e.g. common lisp). As a result
> each ob-lang will probably have to deal with this itself, or at least
> they will have to define how to consume quoted elisp lists without
> raising errors. That said, I'm not sure that org babel requires that
> ob-lang implementations handle this. Maybe it should? I've added this
> to my growing list of issues related to org babel regularization.
> Best!
> Tom
> 
> On Thu, Oct 29, 2020 at 7:25 AM James Boyle  wrote:
> >
> > Hi all,
> >
> > Since this is a bit of a long post, I've put my questions and summary
> > comment at the top, with the detailed context below.
> >
> > 1. What is the level of support for reading nested simple lists into
> >org-babel code blocks?
> > 2. When the manual (see link below) says that they are not supported,
> >is it known if it meant "not at all supported," or, that support is
> >currently language-dependent?
> > 3. Directionally, does the org-mode project care to support nested
> >simple lists?
> >
> > I'm happy to submit patches for both code and manual in any event, but
> > wanted to align on if there are goals for supporting reading nested
> > lists, and if so, at what level.
> >
> > At a minimum, I think updating the manual to be more precise would be
> > great.
> >
> > Thanks for your any assistance you can provide.
> >
> > Details:
> >
> > I was going through the manual on this page:
> >
> > https://orgmode.org/manual/Environment-of-a-Code-Block.html#Environment-of-a-Code-Block
> >
> > and testing out the example, "A simple named list."
> >
> > Contrary to this line in the manual:
> >
> > > Note that only the top level list items are passed along. Nested list 
> > > items are ignored.
> >
> > It looks like the example org-babel block *does* work for nested
> > lists, depending upon the language. I am using org-mode 9.4.
> >
> > Here is the org block verbatim from the manual, with the
> > results that are supposed to occur:
> >
> > #+NAME: example-list
> > - simple
> >   - not
> >   - nested
> > - list
> >
> > #+BEGIN_SRC emacs-lisp :var x=example-list
> >   (print x)
> > #+END_SRC
> >
> > #+RESULTS:
> > | simple | list |
> >
> >
> > And here is the same block, when I evaluate it locally, with the
> > actual results, showing that unordered, nested lists are not ignored.
> >
> > #+NAME: example-list
> > - simple
> >   - not
> >   - nested
> > - list
> >
> > #+BEGIN_SRC emacs-lisp :var x=example-list
> >   (print x)
> > #+END_SRC
> >
> > #+RESULTS:
> > | simple | (unordered (not) (nested)) |
> > | list   ||
> >
> >
> > I thought this was a nice surprise, until I noticed that whether or
> > not it works is language-dependent.
> >
> > In ruby, I get an error:
> >
> > #+begin_src ruby :var x=example-list
> >   # This is the error I get:
> >
> >   # `main': undefined local variable or method `unordered' for
> >   # main:Object (NameError)
> >   puts x
> > #+end_src
> >
> > #+RESULTS:
> >
> > And here is the equivalent error in js. Presumably other languages
> > have variable support for this feature.
> >
> > #+begin_src js :var x=example-list
> >   // Below is a partial error trace:
> >
> >   // 
> > /private/var/folders/mc/jylgk04j5r1f96hcsd3ywm8hgn/T/babel-shSmEq/js-script-xzngmd:2
> >   // var x=[["simple", [unordered, ["not"], ["nested"]]], ["list"]]; ^
> >   // ReferenceError: unordered is not defined
> >   return x
> > #+end_src
> >
> > #+RESULTS:
> >
> > The error can be trivially "fixed" by some code like this, or making
> > the equivalent change in the actual source line to output a string
> > rather than a symbol. Changing L#2365 linked below to use "unordered"
> > (string) rather than 'unordered (symbol) seemed to fix it, and
> > produced no new test failures for me.
> >
> > #+begin_src emacs-lisp
> >   ; See this line for where 'unordered is being spat out:
> >   ; 

prettify-symbols-mode in org agenda?

[William Xu]

Hi,

Is there any plan to support prettify-symbols-mode in org agenda? With
that mode, it would make it easier to, for example, replace the todo
keywords with some nice looking unicode chars. 

>From reddit, I found yantar92 has posted some elisp changes to enable the 
>support: 
https://www.reddit.com/r/orgmode/comments/i3upt6/prettifysymbolsmode_not_working_with_orgagenda/

-- 
William

Re: [PATCH] ob-java

[ian martins]

As I was trying to decide who is the author of the ob-java docs, I realized
it's not clear how you're defining authorship due to my confusion about
ob-java.  I can think of three ways to determine authorship:

1. the person that wrote it
2. the people who influenced the code
3. the first person to check in the filename

At first I thought I wrote ob-java by rule 1.  I didn't start from the old
ob-java, and I replaced the entire file.  The patch shows only 10 random
lines of over 400 matched the original ob-java.  If we don't count the
lines that also match ob-template.el, there are only 5.

When you said I didn't write it I thought rule 2 was the next most
reasonable, so I made the authors those that wrote the code that I
referenced.  But after thinking about it more it can't be this.  Adding
languages to babel isn't documented well enough for anyone to do it without
looking at an existing implementation, so going by rule 2 all languages
would be authored by whoever wrote the first one, and they're not.

I'm not sure but I think you'd say I wrote ob-haxe, the ob-haxe tests, the
ob-java tests, and the ob-java docs, but not ob-java.  These match up with
rule 3.  I don't think rule 3 is the one anyone would pick from the list,
but maybe most would subconsciously use it as a heuristic for rule 1, since
rule 1 is hard to establish.  I think the change in authorship is clear for
ob-java because it was replaced in one patch, usually changes are
incremental.  Each file is The Ship of Theseus.  Even if we took the
trouble to determine how much any person wrote, it is difficult to decide
for oneself let alone agree on the amount of change required to establish
new authorship.

But rule 3 doesn't work if a file is rewritten.  If Dostoevsky checks in
the text of "Crime and Punishment" as book.txt, and then Dr. Seuss replaces
the content with "The Cat in the Hat," we'd have to say Dostoevsky wrote
"The Cat in the Hat."

So I think either you didn't notice that I'd replaced the file, or you
considered the lines that matched sufficient for continuity, or you're
thinking about authorship in a way I haven't guessed.  Could you clarify?

On Wed, Oct 28, 2020 at 5:13 AM Bastien  wrote:

> ian martins  writes:
>
> > But I want to follow your conventions. I will put the authors of ob-C
> > and ob-python (Eric Schulte and Dan Davison) as the authors of
> > ob-java and ob-haxe. The implementations are nearly the same. it
> > wouldn't make sense for them to have different authors.
>
> Thanks for doing so!
>
> --
>  Bastien
>