[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dear Ihor, > Regarding the question about buffer-lens interaction. Let's take even > more complicated example: To run the command, the user hits some key > combination, which happens to be bound to different commands in the main > buffer and in the lense buffer (i.e. the main buffer is in org-mode, the > lense is in mingus-mode, and the key is C-d). What should be the > behaviour in such a case? run the commands in both the buffers? decide > depending on the point position? It is easy to make up similar > complicated examples if you consider some exotic major modes in the > lense buffer. It's basically a question of customization, a client-side decision. In other words, this really depends on what the user wants to happen. This customization is done through the controller of the lens. To your example. If the desirable behavior (for you, as a user) for C-d is to run in the lens, then add "C-d" to the controller of the lens. And then, whenever the point is in the area, C-d runs in the lens unconditionally. (For the sake of terminology, we can say that the keybinding is "redirected".) If you want C-d to work conditionally (sometimes do the org-mode thing and sometimes the mingus-mode thing), I am afraid there is nothing better than to update the controller yourself on the go. And that's fine, because that's what the user wants (to use the same bind for two different things in the same place at different times). (BTW, the controller could be asked to work "in reverse" and redirect all keybindings, except the ones in its black list.) But speaking of the larger picture and integration, a user can define a list of key combinations for any mode and the list will be added to the controller if the lens runs that mode. I think this should cover the vast majority of use-cases. Of course, there is no reason for the logic of key addition not to be flexible enough to cover anything more exotic. > I think that it would be more effective if someone decide on some basic > approach for the low-level implementation of the lense-mode (which > probably involves modifying emacs C-level source code) and continue the > discussion according to the benefits/limitations of that kind of > implementation. I too look forward to hearing from someone about the low-level implementation possibilities :) I especially hope the approach for the border-case issue (as described in my previous message) can work. Best regards, Dmitrii.
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dear Dmitrii, Regarding the question about buffer-lens interaction. Let's take even more complicated example: To run the command, the user hits some key combination, which happens to be bound to different commands in the main buffer and in the lense buffer (i.e. the main buffer is in org-mode, the lense is in mingus-mode, and the key is C-d). What should be the behaviour in such a case? run the commands in both the buffers? decide depending on the point position? It is easy to make up similar complicated examples if you consider some exotic major modes in the lense buffer. I think that it would be more effective if someone decide on some basic approach for the low-level implementation of the lense-mode (which probably involves modifying emacs C-level source code) and continue the discussion according to the benefits/limitations of that kind of implementation. Best, Ihor Dmitrii Korobeinikov writes: > Dear Ihor, > >> Note that indirect buffers always share *all* the contents with the master >> buffer. As a result, it may not be easy to make things like flyspell >> work on code blocks in org-mode, if these code blocks are treated as >> lenses. > > I tried flyspell w/ different dictionaries on 2 buffers. > The dictionary is switched every time I switch into one of the buffers. > You are right, ispell and the like working w/ a file directly would have to > learn to work w/ indirect buffers by managing multiple simultaneous > processes. > Fortunately, that doesn't seem like a big hurdle. > >>> (1) A question: when an indirect buffer is created and some region is >>> narrowed to, is the rest of the buffer duplicated in memory somewhere? If >>> this is so, there could be a useful efficiency-related modification to >>> indirect buffers, which would allow "hard-narrowing": not duplicating the >>> rest of the base buffer. >> >> There is no duplication of the buffer content in indirect buffers. >> Internally, indirect buffer's content is a pointer to the main buffer >> content. If you modify text in any of the indirect buffers or in the >> main buffer, the text is modified in all of them and in the main buffer. >> Only the buffer-local variables are duplicated. >> You can refer to "27.11 Indirect Buffers" in the elisp manual for >> details. > > Bad choice of wording on my side, I didn't mean duplication, but rather > keeping unnecessary info, like text properties in the newly created > indirect buffer, in the regions which were "permanently" chosen to be > narrowed-out. > Anyway, this is a premature optimization for now. > >> > The next immediately outstanding question is: >> > (2) how can "embedding" (of a buffer as a part of another buffer as an >> > area) be done efficiently? This could possibly be approached as two >> > problems: (i) displaying the area and (ii) interacting with it. >> > Any ideas? >> >> These issues have been discussed in >> https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html. >> As I remember, the discussion stopped without a clear conclusion. It was >> not clear how to separate the main buffer contents from the nested >> buffer (I treat them as analogue of the buffer lenses). Another issue >> was how the keymaps and buffer-local variables would interact when the >> point is within a lense. It was not clear what should be the priority. > > The short answer is probably that lens-mode looks at the changes to the > buffer and decides what's what. > Here is my vision for this. > > Say, you have an indirect buffer, call it A, it's base has contents: > >> line 1 >> line 2 >> line 3 > > Also, there is a buffer, call it B, where we want to embed A, with contents: > >> word 1 >> instruction: lens that displays A >> word 2 > > Lens-mode decides to identify the second line as a lens and constructs > layout of the file. > >> [text] >> [lens#A] >> [text] > > Now, construct and display the final buffer as: > >> word 1 >> line 1 >> line 2 >> line 3 >> word 2 > > The core question: how is this "displaying" done. > In part, somewhat like how indirect buffers function. > A displayed piece of text is a pointer/reference to the text in the > indirect buffer. > Of course, this should be done in a way so that the modes running in B > don't change the properties of the text (following the layout constructed > by lens-mode as in the example above). Though, this might better&easier be > done at the display unit level. > > What about interaction? > Well, when the cursor is inside the lens, the controller decides what to do > w/ each keybinding, whether to redirect it to the indirect buffer or not. > And what about the case when borders are crossed? > As I see it, any code that executes - does so as is. > For instance, consider a buffer with a lens (contents: "lens"), which looks > like this: "word1 lens word2". > Place the cursor on the first word, run a command to remove 2 words. > Now there are to possibilities here, depending on the implementation of the > function which does the removal: > 1.
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dear Ihor, > Note that indirect buffers always share *all* the contents with the master > buffer. As a result, it may not be easy to make things like flyspell > work on code blocks in org-mode, if these code blocks are treated as > lenses. I tried flyspell w/ different dictionaries on 2 buffers. The dictionary is switched every time I switch into one of the buffers. You are right, ispell and the like working w/ a file directly would have to learn to work w/ indirect buffers by managing multiple simultaneous processes. Fortunately, that doesn't seem like a big hurdle. >> (1) A question: when an indirect buffer is created and some region is >> narrowed to, is the rest of the buffer duplicated in memory somewhere? If >> this is so, there could be a useful efficiency-related modification to >> indirect buffers, which would allow "hard-narrowing": not duplicating the >> rest of the base buffer. > > There is no duplication of the buffer content in indirect buffers. > Internally, indirect buffer's content is a pointer to the main buffer > content. If you modify text in any of the indirect buffers or in the > main buffer, the text is modified in all of them and in the main buffer. > Only the buffer-local variables are duplicated. > You can refer to "27.11 Indirect Buffers" in the elisp manual for > details. Bad choice of wording on my side, I didn't mean duplication, but rather keeping unnecessary info, like text properties in the newly created indirect buffer, in the regions which were "permanently" chosen to be narrowed-out. Anyway, this is a premature optimization for now. > > The next immediately outstanding question is: > > (2) how can "embedding" (of a buffer as a part of another buffer as an > > area) be done efficiently? This could possibly be approached as two > > problems: (i) displaying the area and (ii) interacting with it. > > Any ideas? > > These issues have been discussed in > https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html. > As I remember, the discussion stopped without a clear conclusion. It was > not clear how to separate the main buffer contents from the nested > buffer (I treat them as analogue of the buffer lenses). Another issue > was how the keymaps and buffer-local variables would interact when the > point is within a lense. It was not clear what should be the priority. The short answer is probably that lens-mode looks at the changes to the buffer and decides what's what. Here is my vision for this. Say, you have an indirect buffer, call it A, it's base has contents: > line 1 > line 2 > line 3 Also, there is a buffer, call it B, where we want to embed A, with contents: > word 1 > instruction: lens that displays A > word 2 Lens-mode decides to identify the second line as a lens and constructs layout of the file. > [text] > [lens#A] > [text] Now, construct and display the final buffer as: > word 1 > line 1 > line 2 > line 3 > word 2 The core question: how is this "displaying" done. In part, somewhat like how indirect buffers function. A displayed piece of text is a pointer/reference to the text in the indirect buffer. Of course, this should be done in a way so that the modes running in B don't change the properties of the text (following the layout constructed by lens-mode as in the example above). Though, this might better&easier be done at the display unit level. What about interaction? Well, when the cursor is inside the lens, the controller decides what to do w/ each keybinding, whether to redirect it to the indirect buffer or not. And what about the case when borders are crossed? As I see it, any code that executes - does so as is. For instance, consider a buffer with a lens (contents: "lens"), which looks like this: "word1 lens word2". Place the cursor on the first word, run a command to remove 2 words. Now there are to possibilities here, depending on the implementation of the function which does the removal: 1. Implementation identifies the boundaries of deletion and removes the words as contents of the main buffer. Lens-mode identifies the removal and deletes the lens altogether. 2. Delete "word1" -> the cursor is on "lens" -> next deletion command is redirected to the indirect buffer, which does the removal. Lens-mode identifies the lens as empty and removes it. The second way might not be always desirable, so whenever a function is called with the cursor outside a lens, as an option, decide to never redirect the input to a lens while the function runs. For another case, consider the first example with lines and, cursor being in the beginning of the file, remove three lines. To make this interesting, assume the first kind of implementation runs, identifying the bounds, never redirecting a command to the lens. Well, if the implementation of the lens is inspired by how indirect buffers work, I imagine, the necessary changes in the embedded buffer take place directly, in which case everything works as desirable. Afterwards, lens-mode processes t
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dear Dmitrii, > Indirect buffers give the answer to the issue of sharing some textual data > between several buffer. Note that indirect buffers always share *all* the contents with the master buffer. As a result, it may not be easy to make things like flyspell work on code blocks in org-mode, if these code blocks are treated as lenses. > (1) A question: when an indirect buffer is created and some region is > narrowed to, is the rest of the buffer duplicated in memory somewhere? If > this is so, there could be a useful efficiency-related modification to > indirect buffers, which would allow "hard-narrowing": not duplicating the > rest of the base buffer. There is no duplication of the buffer content in indirect buffers. Internally, indirect buffer's content is a pointer to the main buffer content. If you modify text in any of the indirect buffers or in the main buffer, the text is modified in all of them and in the main buffer. Only the buffer-local variables are duplicated. You can refer to "27.11 Indirect Buffers" in the elisp manual for details. > The next immediately outstanding question is: > (2) how can "embedding" (of a buffer as a part of another buffer as an > area) be done efficiently? This could possibly be approached as two > problems: (i) displaying the area and (ii) interacting with it. > Any ideas? These issues have been discussed in https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html. As I remember, the discussion stopped without a clear conclusion. It was not clear how to separate the main buffer contents from the nested buffer (I treat them as analogue of the buffer lenses). Another issue was how the keymaps and buffer-local variables would interact when the point is within a lense. It was not clear what should be the priority. Best, Ihor Dmitrii Korobeinikov writes: > I found a clarification on how mmm-mode works. > > https://github.com/polymode/polymode/issues/187 >> mmm-mode also allows having multiple major modes depending on cursor > position in the buffer. However, it does not fully replace major mode > locally. This mode is only taking care about keymap, menu, local variables, > font-lock, and indentation. It does not really take care about the minor > modes and does not run the submode hooks either. > > Just to reiterate, polymode's idea is to switch between indirect buffers, > one for each major mode. > > OK, detail largely disregarded, I now can draw a bird-eye view comparison > between lenses and multi-mode modes. > > - Neither polymode nor mmm-mode treat a region as if it were truly on its > own in a seperate buffer. > > Effects: no stuff like seperate truncation options, implied syntax checking > and so on. > > - Moreover, the region must be a part of the buffer. > > Effects: no data sharing between buffers, no possibility of stitching > different buffers together, etc. > > Now, with these out of the way. > > Indirect buffers give the answer to the issue of sharing some textual data > between several buffer. > (1) A question: when an indirect buffer is created and some region is > narrowed to, is the rest of the buffer duplicated in memory somewhere? If > this is so, there could be a useful efficiency-related modification to > indirect buffers, which would allow "hard-narrowing": not duplicating the > rest of the base buffer. > > The next immediately outstanding question is: > (2) how can "embedding" (of a buffer as a part of another buffer as an > area) be done efficiently? This could possibly be approached as two > problems: (i) displaying the area and (ii) interacting with it. > Any ideas? -- Ihor Radchenko,
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
I found a clarification on how mmm-mode works. https://github.com/polymode/polymode/issues/187 > mmm-mode also allows having multiple major modes depending on cursor position in the buffer. However, it does not fully replace major mode locally. This mode is only taking care about keymap, menu, local variables, font-lock, and indentation. It does not really take care about the minor modes and does not run the submode hooks either. Just to reiterate, polymode's idea is to switch between indirect buffers, one for each major mode. OK, detail largely disregarded, I now can draw a bird-eye view comparison between lenses and multi-mode modes. - Neither polymode nor mmm-mode treat a region as if it were truly on its own in a seperate buffer. Effects: no stuff like seperate truncation options, implied syntax checking and so on. - Moreover, the region must be a part of the buffer. Effects: no data sharing between buffers, no possibility of stitching different buffers together, etc. Now, with these out of the way. Indirect buffers give the answer to the issue of sharing some textual data between several buffer. (1) A question: when an indirect buffer is created and some region is narrowed to, is the rest of the buffer duplicated in memory somewhere? If this is so, there could be a useful efficiency-related modification to indirect buffers, which would allow "hard-narrowing": not duplicating the rest of the base buffer. The next immediately outstanding question is: (2) how can "embedding" (of a buffer as a part of another buffer as an area) be done efficiently? This could possibly be approached as two problems: (i) displaying the area and (ii) interacting with it. Any ideas?
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Am Do., 25. Apr. 2019 um 10:41 Uhr schrieb Dmitrii Korobeinikov : > I have imagined that at the low level there is an actual data structure that > keeps the raw textual data and it could be directly shared by multiple > buffers. That's what indirect buffers do. Maybe the indirect buffer functionality could be beefed up to support what you want?
Re: [O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
I see lens to be useful for the eev mode, too. Roland. Dmitrii Korobeinikov writes: >> Have you looked at Phil Lord's lentic package? I think it implements a >> lot of what you're talking about. > >> https://github.com/phillord/lentic > > This is nice to see! > Indeed, except for embedding, there is a large overlap with what I > described as buffer lenses. > > BTW, judging by this description: "changes percolation now happens > incrementally, so only those parts of the buffer are updated. As a result, > lentic now cope with long files with little noticable delay", the buffers > don't share any data and need to sync with the master [linked] buffer. > Is this the best solution? I have imagined that at the low level there is > an actual data structure that keeps the raw textual data and it could be > directly shared by multiple buffers. I mean, when a buffer is saved to a > file, the text doesn't need to be stripped of properties beforehand, right? > > чт, 25 апр. 2019 г. в 07:37, Noam Postavsky : > >> Dmitrii Korobeinikov writes: >> >> > * Implementation >> > >> > I am not familiar with Emacs internals to say what's feasible of the >> > proposed structure. >> >> Have you looked at Phil Lord's lentic package? I think it implements a >> lot of what you're talking about. >> >> https://github.com/phillord/lentic >> -- Luke, use the FOSS Sent from Emacs
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
чт, 25 апр. 2019 г. в 23:52, Philipp Stephani : > Am Do., 25. Apr. 2019 um 10:41 Uhr schrieb Dmitrii Korobeinikov > : > > I have imagined that at the low level there is an actual data structure > that keeps the raw textual data and it could be directly shared by multiple > buffers. > > That's what indirect buffers do. Maybe the indirect buffer > functionality could be beefed up to support what you want? > https://www.gnu.org/software/emacs/manual/html_node/emacs/Indirect-Buffers.html > The text of the indirect buffer is always identical to the text of its base buffer; changes made by editing either one are visible immediately in the other. But in all other respects, the indirect buffer and its base buffer are completely separate. They can have different names, different values of point, different narrowing, different markers, different major modes, and different local variables. Awesome! Looks like we have some solid rails to drive on. BTW what's the purpose of lentic-mode then? To be "providing multiple persistent views"? https://github.com/phillord/lentic
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dear Ihor, > Another use case for me is to speed up agenda creation. > I usually do not like to split my org files into too many. However, it > results in very large and slow org buffers later. If I can store some > parts of the org files externally and only show them if some condition > is met (say, for certain todo state of the parent entry), it would speed > up my agenda and the buffer navigation quite significantly. That's a good one! > Let me put some historical context to this proposal. > There was a discussion of similar feature in emacs-dev last year. > The idea was to implement nested buffers: > https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html An interesting read, provides another use-case (collect external data in one place to easily view/edit): https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00890.html > There are also several projects, which implement part of the > functionality you described: > - mmm-mode: https://github.com/purcell/mmm-mode > - polymode: https://github.com/polymode/polymode Pretty cool stuff. For thoroughness, let's discuss how these work. I found a comment which mentions polymode's working principle. https://www.reddit.com/r/emacs/comments/50p34n/polymode_is_awesome/?depth=1 >> Polymode doesn't keep its modes in a single emacs buffer but in several indirect buffers, as many as different modes are there in a file. Consequently, polymode is as fast as switching emacs buffers because it never re-installs major modes like other multi-modes do. Dave Love's multi-mode.el gets full credit for this idea. > It looks like it slows emacs to a crawl in my main org config file. It seems to work fairly well in some of my notes files (though with some weird indenting behavior). Basically, simplicity is in place but at the cost of duplication. Lenses could avoid duplication, while yielding increased functionality and speed. (e.g. in polymode, a syntax checker couldn't yield correct results unless narrowing was constantly used, which is inefficient) Now, to MMM-mode. According to the info file: > Within the file, MMM-mode creates /submode regions/ within which other major modes are in effect. > While the point is in a submode region, the following changes occur: > <...> keymap <...> local variables <...> syntax table and indentation <...> font-lock > The submode regions are represented internally by Emacs Lisp objects known as /overlays/. > A lot of the functionality of MMM Mode---that which makes the major mode > appear to change---is implemented by saving and restoring the values of > local variables, or pseudo-variables. What I don't understand is where the modes of the submode region run and when they are turned on. Are necessary modes just allowed to run at the right time for the whole buffer? But then, how are they limited in their effect to just the necessary region? Narrowing? Could, for example, syntax checking be done efficiently that way? Could someone, please, explain? Best regards, Dmitrii.
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
> Have you looked at Phil Lord's lentic package? I think it implements a > lot of what you're talking about. > https://github.com/phillord/lentic This is nice to see! Indeed, except for embedding, there is a large overlap with what I described as buffer lenses. BTW, judging by this description: "changes percolation now happens incrementally, so only those parts of the buffer are updated. As a result, lentic now cope with long files with little noticable delay", the buffers don't share any data and need to sync with the master [linked] buffer. Is this the best solution? I have imagined that at the low level there is an actual data structure that keeps the raw textual data and it could be directly shared by multiple buffers. I mean, when a buffer is saved to a file, the text doesn't need to be stripped of properties beforehand, right? чт, 25 апр. 2019 г. в 07:37, Noam Postavsky : > Dmitrii Korobeinikov writes: > > > * Implementation > > > > I am not familiar with Emacs internals to say what's feasible of the > > proposed structure. > > Have you looked at Phil Lord's lentic package? I think it implements a > lot of what you're talking about. > > https://github.com/phillord/lentic >
[O] bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)
Dmitrii Korobeinikov writes: > * Implementation > > I am not familiar with Emacs internals to say what's feasible of the > proposed structure. Have you looked at Phil Lord's lentic package? I think it implements a lot of what you're talking about. https://github.com/phillord/lentic