[TASK] Making org-src buffers sync with real files to allow LSP and other dev tools integration (was: [PATCH] LSP support in org-src buffers)

[Ihor Radchenko]

Ihor Radchenko  writes:

> In any case, João has a working solution. Once we get some actual patch
> that applies onto Org, we can start to improve it incrementally.

It has been a while since the last update in this thread.

If other people are interested to contribute, please feel free to do so
and ask any questions in the process.
I consider this feature important for Org and encourage contributions.
-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[João Pedro]

On Mon, December 12 2022 13:16, Ihor Radchenko  wrote:

> If an src block has :tangle option, it is probably reasonable to display
> it together with other tangled blocks.
> If an src blocks does not have a :tangle option, :context might be used
> as the means to group blocks together. But then, the value should be
> some kind of string - blocks with the same :context name will be grouped
> together.

That makes much more sense. Though I still think the user should be able
to, even with a :tangle option, be able to choose whether they want to
edit that particular source block with the whole context or not, cause
it might be slow for some huge tangled files, for example.

> And, of course, the whole thing may be behind a defcustom switch. Or
> file option.

Yup! I prefer to have both options, a file option and a defcustom. This
allows me to have a global configuration, but change it to a particular
file without having to deal with the local variables block in Org files.

> We are not in hurry. Any progress will be appreciated as long as there
> is some.

All right. I'll keep you posted on it.

-- 
João Pedro de A. Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)

Re: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

João Pedro  writes:

> Were you thinking of having a
> =:context {yes|no}= option on src blocks?

I am not sure if yes|no are good values.

If an src block has :tangle option, it is probably reasonable to display
it together with other tangled blocks.
If an src blocks does not have a :tangle option, :context might be used
as the means to group blocks together. But then, the value should be
some kind of string - blocks with the same :context name will be grouped
together.

And, of course, the whole thing may be behind a defcustom switch. Or
file option.

> Also, I will have more free time to work on it starting on the second
> week of December. Before that I'll be incredibly busy, unfortunately.

We are not in hurry. Any progress will be appreciated as long as there
is some.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[João Pedro]

On Tue, November 22 2022 02:23, Ihor Radchenko  wrote:

> It looks like you have done something very close to what we are
> discussing. Would you be interested to create a patch against Org core
> instead of relying on advises?

That would be wonderful! That code has some ugly hacks, some of which
gave me a little bit of headache when I was using it, but I think that
it could become something polished and well-behaved, specially taking
inspiration on Karthik's approach to it as well (I can, from a quick
glance, see some improvements that could be made. And who knows maybe we
make it work even with noweb...). Were you thinking of having a
=:context {yes|no}= option on src blocks?

Also, I will have more free time to work on it starting on the second
week of December. Before that I'll be incredibly busy, unfortunately.

Best regards,

-- 
João Pedro de A. Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)

RE: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

"Cook, Malcolm"  writes:

> Coming late to this discussion….
>
> Might there be some value in considering the aims and methods of 
> [Polymode](https://github.com/polymode/polymode#readme) in this regard?  
> There seems to be a significant overlap of concerns.
>
> Just a thought.

I can see why you mentioned polymode, but let's not go there yet.

Polymode is a very fragile and complex interleaving between several
indirect buffers with different active major modes that allows "native"
key bindings, minor modes, and fontification in source blocks.

What we are discussing here is closer to
https://github.com/phillord/lentic, which syncs contents of two or more
separate, not indirect, buffers.

There is really nothing much we can borrow from lentic or polymode. It
is fairly straightforward what to do in general. However, the details
like noweb support and code blocks boundaries are to be discussed and
figured out.

In any case, João has a working solution. Once we get some actual patch
that applies onto Org, we can start to improve it incrementally.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

RE: [PATCH] LSP support in org-src buffers

[Cook, Malcolm]

Coming late to this discussion….

Might there be some value in considering the aims and methods of 
[Polymode](https://github.com/polymode/polymode#readme) in this regard?  There 
seems to be a significant overlap of concerns.

Just a thought.

Cheers,

Malcolm

Re: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

João Pedro  writes:

> I think the contextual src block could be split from the LSP
> functionality, since it could be used in other contexts other than
> dealing with language servers. My iteration is not tied to anything in
> particular and simply creates a source buffer with the whole context for
> that tangled file.
>
> Hope this can help you in some way!
>
> [1] https://gist.github.com/bigodel/15599f3c1da23d1008b7d7d4ff8bff77

It looks like you have done something very close to what we are
discussing. Would you be interested to create a patch against Org core
instead of relying on advises?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[João Pedro]

Hey there, Karthik!

I've had a similar idea, though not tied to LSP or anything. I created a
=org-contextual-src-mode= [1] ir order to have completion working on
Ledger and LaTeX source buffers. It also handles tangling! I have made a
Gist explaining, in a literate style, my process of building such a
mode.

I think the contextual src block could be split from the LSP
functionality, since it could be used in other contexts other than
dealing with language servers. My iteration is not tied to anything in
particular and simply creates a source buffer with the whole context for
that tangled file.

Hope this can help you in some way!

[1] https://gist.github.com/bigodel/15599f3c1da23d1008b7d7d4ff8bff77

Cheers,

-- 
João Pedro de A. Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)

Re: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

[Just following this up as more than one month have passed since the last
activity in this thread.]

Karthic, have you had a chance to work on this further?

If you stumbled upon difficulties, feel free to ask anything. We can try
to help.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

Karthik Chikmagalur  writes:

>> You already implemented a way to associate the org-edit-src buffer with
>> the fully tangled code. Then, why not make it simple and do the real
>> tangling first and then make org-edit-src work directly with a real
>> file buffer associated with the tangled file?
>
> This will drastically simplify the patch, true.  I was working on the 
> assumption that since tangling overwrites the file on disk, it should not be 
> an implicit operation invoked as a side-effect of another action.  It causes 
> other changes that the user might not have intended, like updating timestamps 
> on the tangled file, etc.  What do you suggest?
>
> Moreover, for Eglot to function correctly it is sufficient to (i) associate 
> the buffer with a file -- any file, and (ii) Set the default-directory 
> variable to the correct value.   "Tangling" to a file in /tmp (as I do in the 
> patch) will not work with all the non-Eglot use-cases you describe above.

Then, I suggest to not actually write things on the disk. Instead, we
can augment `org-edit-src-save' to write on disk depending on some
customization (with values t, nil, and 'ask). That customization will,
by default, make `org-edit-src-save' query user if it is desired to
write files on disk.

Also, note that once you associate buffer with a file, various
auto-saving mechanisms may write the file to disk without user consent.
We may also need to look into write-contents-functions to prevent such
scenarios when undesired.

>> The only tricky problem I am seeing with your approach is dealing with
>> noweb references. Care should be taken about editing code blocks
>> containing noweb.
>
> If I reuse the actual tangling machinery in ob-tangle instead of writing my 
> own version reusing only some of the primitives in this library, this should 
> be handled automatically for me.  Is this correct?

No.

`org-edit-special' will _not_ expand noweb.
`org-babel-expand-src-block' will.

In the first case, <> syntax may drive native syntax checkers
crazy. In the second case, care should be taken to prevent editing the
noweb-replaced text. This distinction is something we would not want to
break.

Also, be aware that some <> references may involve code execution,
possibly on remote machines. Remote execution might be undesired during
normal editing. I am not 100% sure how to approach this problem in a
reliable way that will fit all the uses.

> Also: org-src-context-mode works by advising some org-edit-src-* functions.  
> Is it preferable to edit these functions directly instead and add a check for 
> whether org-src-context-mode is enabled?

I am not even sure if a separate org-src-context-mode is needed. It
could be simply a customization for org-edit-special rather than a minor
mode.

So, I am all for editing the functions directly and more complete
integration.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[Karthik Chikmagalur]

> This is not limited to Eglot support. M-x compile, eglot, project.el,
> xrefs, and similar tools all assume that current code buffer is
> associated with a real file in a real project folder, possibly
> containing all kinds of hints like .gitignore, .dir-locals.el, etc.

I hadn't considered this.  This makes context-aware org-src support even more 
important.

> This sounds a bit fragile and full of caveats.
>
> You already implemented a way to associate the org-edit-src buffer with
> the fully tangled code. Then, why not make it simple and do the real
> tangling first and then make org-edit-src work directly with a real
> file buffer associated with the tangled file?
>
> All the tools, including Eglot will then work naturally and as intended.

This will drastically simplify the patch, true.  I was working on the 
assumption that since tangling overwrites the file on disk, it should not be an 
implicit operation invoked as a side-effect of another action.  It causes other 
changes that the user might not have intended, like updating timestamps on the 
tangled file, etc.  What do you suggest?

Moreover, for Eglot to function correctly it is sufficient to (i) associate the 
buffer with a file -- any file, and (ii) Set the default-directory variable to 
the correct value.   "Tangling" to a file in /tmp (as I do in the patch) will 
not work with all the non-Eglot use-cases you describe above.

> The only tricky problem I am seeing with your approach is dealing with
> noweb references. Care should be taken about editing code blocks
> containing noweb.

If I reuse the actual tangling machinery in ob-tangle instead of writing my own 
version reusing only some of the primitives in this library, this should be 
handled automatically for me.  Is this correct?

> Our to-be-released main branch does tangling orders of magnitude faster.
> For example, my config.org with 660 src blocks tangles within 1.3 sec.
>
> This can be made even faster by extra caching if there is a real demand
> for super-fast tangling.

That's a giant improvement over the current implementation. I'll keep this in 
mind when working on the patch.

Also: org-src-context-mode works by advising some org-edit-src-* functions.  Is 
it preferable to edit these functions directly instead and add a check for 
whether org-src-context-mode is enabled?

Karthik

Re: [PATCH] LSP support in org-src buffers

[Ihor Radchenko]

Karthik Chikmagalur  writes:

> I've added limited support for LSP via Eglot in org-src-mode buffers.  I was 
> intending to publish it as a package but it was suggested to me that it could 
> live as part of Org instead, especially now that Eglot is intended to be part 
> of the upcoming Emacs release.  Here are some details:
> ...
> It allows you to run Eglot in org-src buffers opened with org-edit-special 
> (C-c '), giving you context-aware completion, linting, code actions etc.
> ...
> The main problem with reconciling org-src-mode and Language Server (LS) 
> support is that the LSP requires and expects files, not buffers.  By default, 
> org-src buffers are not associated with files.  Even if one were to associate 
> an org-src-mode buffer with a file, set the correct default-directory for a 
> project and start Eglot, it would only contain the small chunk of code from 
> the current code block.  The LS cannot access enough code to form a useful 
> picture of the project.

Thanks for your interest in contributing to Org core!

Transparent code editing in org-src buffers is certainly a feature that
is requested by many. Starting from people complaining about flycheck
giving nonsense errors in org-src buffers extending to any other serious
development work that relies on the existing project development
toolchains.

This is not limited to Eglot support. M-x compile, eglot, project.el,
xrefs, and similar tools all assume that current code buffer is
associated with a real file in a real project folder, possibly
containing all kinds of hints like .gitignore, .dir-locals.el, etc.

> org-src-context-mode reuses the tangling machinery to populate an org-src 
> buffer with code from all blocks associated with the current tangle file, and 
> associates it with a temporary file.  This way, the LS has a better picture 
> -- if still limited to a single file -- of the project.
>
> org-src-context-mode then checks if there's an appropriate Eglot LSP 
> connection active, and reconnects to it.
>
> Only the contents of the current code block are editable.  The other blocks 
> are marked read-only and (by default) only visible by widening the buffer.  
> No actual tangling is done -- the default-directory of the org file is not 
> touched at all.

This sounds a bit fragile and full of caveats.

You already implemented a way to associate the org-edit-src buffer with
the fully tangled code. Then, why not make it simple and do the real
tangling first and then make org-edit-src work directly with a real
file buffer associated with the tangled file?

All the tools, including Eglot will then work naturally and as intended.

The only tricky problem I am seeing with your approach is dealing with
noweb references. Care should be taken about editing code blocks
containing noweb.

> (v) org-src-context-mode does some pseudo-tangling -- this is required to 
> specify what constitutes a "file" for the LS to parse. This adds a 
> performance penalty to org-edit-src-code that can be noticeable if you have 
> many (100+?) code blocks with the same tangle file as the current block.

Our to-be-released main branch does tangling orders of magnitude faster.
For example, my config.org with 660 src blocks tangles within 1.3 sec.

This can be made even faster by extra caching if there is a real demand
for super-fast tangling.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] LSP support in org-src buffers

[Christopher M. Miles]

+1

Good patch, will it consider to also support package "lsp-mode"? Or at
least make it easy to extending for other lsp-related packages?

-- 

[ stardiviner ]
I try to make every word tell the meaning that I want to express without 
misunderstanding.

Blog: https://stardiviner.github.io/
IRC(libera.chat, freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3


signature.asc
Description: PGP signature

[PATCH] LSP support in org-src buffers

[Karthik Chikmagalur]

Hi folks,

I've added limited support for LSP via Eglot in org-src-mode buffers.  I was 
intending to publish it as a package but it was suggested to me that it could 
live as part of Org instead, especially now that Eglot is intended to be part 
of the upcoming Emacs release.  Here are some details:

I.   What it does.
II.  How to use it.
III. How it works.
IV.  Limitations and concerns.

I. What it does
===

It allows you to run Eglot in org-src buffers opened with org-edit-special (C-c 
'), giving you context-aware completion, linting, code actions etc.

II. How to use it


(i) Turn on org-src-context-mode.
(ii) Add :tangle header args to code blocks that you intend the Language Server 
(LS) to parse sequentially, as one file.  To parse all code blocks in the Org 
document using the LS as one file, you can set a global ":tangle yes" header 
argument. (You don't have to tangle anything -- this is to identify which code 
blocks constitute a "file".)
(iii) Use org-edit-special on a code block.
(iv) Run eglot or eglot-connect in the org-src-mode buffer.

>From this point on, you can use org-edit-special and org-edit-src-exit (both 
>bound by default to C-c ') as usual, and you should have LSP support via 
>Eglot.  This LSP connection is persistent across this Emacs session, you can 
>exit the org-src buffer or kill the Org buffer, and come back to it at will.

(v) You can shutdown the LSP connection with eglot-shutdown, as usual.

III. How it works


The main problem with reconciling org-src-mode and Language Server (LS) support 
is that the LSP requires and expects files, not buffers.  By default, org-src 
buffers are not associated with files.  Even if one were to associate an 
org-src-mode buffer with a file, set the correct default-directory for a 
project and start Eglot, it would only contain the small chunk of code from the 
current code block.  The LS cannot access enough code to form a useful picture 
of the project.

org-src-context-mode reuses the tangling machinery to populate an org-src 
buffer with code from all blocks associated with the current tangle file, and 
associates it with a temporary file.  This way, the LS has a better picture -- 
if still limited to a single file -- of the project.

org-src-context-mode then checks if there's an appropriate Eglot LSP connection 
active, and reconnects to it.

Only the contents of the current code block are editable.  The other blocks are 
marked read-only and (by default) only visible by widening the buffer.  No 
actual tangling is done -- the default-directory of the org file is not touched 
at all.

If there's no :tangle header arg, org-src-context-mode does nothing.

IV. Limitations and concerns
=

(i) This creates temporary files with (ostensibly) the contents of code blocks 
in the Org file.

(ii) org-src-context-mode is implemented by advising org-edit-src-code and 
org-edit-src-exit. This is because it was originally intended to be a 
third-party package. These functions will need to be modified a bit otherwise.

(iii) I'm assuming this design will go through revisions, so I haven't updated 
the Org manual yet. (I did update the changelog.)

(iv) It is quite straightforward to add lsp-mode support with a user-option. 
(The LSP-specific part of this package is tiny.) Since lsp-mode is not part of 
Emacs and Eglot will be soon, I decided to focus on Eglot support.

(v) org-src-context-mode does some pseudo-tangling -- this is required to 
specify what constitutes a "file" for the LS to parse. This adds a performance 
penalty to org-edit-src-code that can be noticeable if you have many (100+?) 
code blocks with the same tangle file as the current block.

(vi) Consider this scenario: The code for your entire project resides in one or 
more Org files, and is intended to be tangled to several files under a project 
root directory.  Then the nature of LSP support depends on the state of 
tangling.
- Before tangling anything: LSP support with org-src-context-mode remains 
limited since it can only "see" one file, the one being edited.
- Post-tangling the entire project: You have full and veracious LSP support in 
all org-src buffers.
- Post-tangling and after edits to multiple code blocks: LSP support is now 
*incorrect* since it "sees" a combination of the current state of the "file" 
being edited in the org-src buffer, and the past state of tangled versions of 
other code blocks.

Still, I've found this to be a big improvement over having no LSP support for 
Org code blocks.

Please let me know if you have any feedback.

Karthik
>From 2798a292d293f1d0aeed34bd0014c6bb97079491 Mon Sep 17 00:00:00 2001
From: Karthik Chikmagalur 
Date: Fri, 7 Oct 2022 21:14:42 -0700
Subject: [PATCH] Add org-src-context.el

---
 etc/ORG-NEWS|  27 ++
 lisp/org-src-context.el | 186 
 2 files changed, 213 insertions(+)
 create mode 100644