Re: A requires/provides approach to linking source code blocks
one thing about org that i think has been making it complex, in addition to number of features, is non-orthogonality. On 7/13/21, Tim Cross wrote: > > Tom Gillespie writes: > > [snip] > >> >> Once I wrap up the formal grammar for org, one of the next things I >> plan to work on is a clear specification for org babel. This is >> critical because so many of the suggestions that come in deal with >> individuals' specific problems and thus fail to account for how such >> features interact with existing features and how the newly proposed >> feature would block some other features in the future, confuse users, >> etc. Such suggestions also often fail to account for increased >> complexity, nor have they been exposed to a sufficient number of >> examples to reveal fundamental ambiguities in how they could be >> interpreted. The issues with variable behavior between ob langs for >> :pre :post :prologue :epilogue etc. are already enough to keep us busy >> for quite some time. >> > > Yes, that clearly summs up my concerns as well. Often, a 'fix' for some > problem can seem straight-forward, possibly even trivial, when > considered only within one person's use case/workflow. Part of what > makes org so powerful is the level of flexibility it supports. However, > this flexibility makes it extremely difficult to consider, or even know > about, all the existing use cases. > > I suspect that once we have some formal specifications for babel, the > next step will be to develop some good unit tests to verify these > specifications. This would at least highlight/alert developers to > unforeseen impact from changes and alert them to things they may > not have considered. > >> With regard to this thread in particular, it is of some interest, but >> there are fundamental issues, including the fact that certain >> languages (e.g. racket) expect module code to exist somewhere on the >> file system. There are ways around many of these issues, in fact there >> are likely many ways around any individual issue, so org babel needs >> to systematically consider the issues and provide a clear >> specification, or at least a guide for how such cases should be >> handled. >> >> To give an example from one of my continual pain points: I start >> writing python or racket in an org src block and then I want it to be >> a library so that it can be reused by other code both inside and >> outside the org file without having to resort to noweb. >> > > This is an interesting point. I think a number of languages have > challenges here. I run into very similar issues with Clojure. > > To some extent, I think this is a grey area within the literate > programming paradigm. The original literate programming model was > developed at a time when most languages were compiled rather then > interpreted. You generated source code, compiled it and then ran it. > > These days, many 'modern' languages are based around an interpreter and > concepts like 'just in time' compilation. In org mode, things become > even more complex because in addition to generation (tangling) of code, > we also want to have evaluation of code blocks, plus we have added the > concept of 'sessions'. > >> What is the best way to handle this? I don't know. Right now I tangle >> things and resort to awful hacks for the reuse-in-this-org-file case, but >> I'm guessing there is a better generic solution which would allow _any_ >> org block to be exported as a library instead of nowebbed in. >> >> Before jumping for any particular suggestion for how to handle this >> we need to explore the diversity of cases that various ob langs >> present, so that we can find a solution that will work for all of >> them. After all, packaging code to a library for reuse is an >> extremely common pattern that org babel should be able to >> abstract, but it is a major undertaking, not just the addition of a >> keyword here and there. >> > > Agree. I'm not convinced we really understand the requirements here and > more exploration and specification is required. The more we add 'simple' > extensions, options, keywords etc the more likely it is we will make > this a much harder process and will likely result in even larger > 'breakage' once we do define a clearer specification. > >> In short I suggest that we issue a general moratorium on new org babel >> feature suggestions until we can stabilize what we already have and >> provide a clear specification for correct behavior. Until we have that >> spec >> we could encourage users to create extensions that implement those >> features. >> > > Yes. While it might sound harsh or overly limiting, I do think such a > moratorium may be required. We may be able to lift it once we have some > core specifications in place. We would still accept bug fixes (though we > may need to provide some clarity on what is a bug fix and what is a > feature enhancement/change - I regularly see posts flagged as bug fixes > which are actually feature enhancements or extensions).
Re: A requires/provides approach to linking source code blocks
Tom Gillespie writes: [snip] > > Once I wrap up the formal grammar for org, one of the next things I > plan to work on is a clear specification for org babel. This is > critical because so many of the suggestions that come in deal with > individuals' specific problems and thus fail to account for how such > features interact with existing features and how the newly proposed > feature would block some other features in the future, confuse users, > etc. Such suggestions also often fail to account for increased > complexity, nor have they been exposed to a sufficient number of > examples to reveal fundamental ambiguities in how they could be > interpreted. The issues with variable behavior between ob langs for > :pre :post :prologue :epilogue etc. are already enough to keep us busy > for quite some time. > Yes, that clearly summs up my concerns as well. Often, a 'fix' for some problem can seem straight-forward, possibly even trivial, when considered only within one person's use case/workflow. Part of what makes org so powerful is the level of flexibility it supports. However, this flexibility makes it extremely difficult to consider, or even know about, all the existing use cases. I suspect that once we have some formal specifications for babel, the next step will be to develop some good unit tests to verify these specifications. This would at least highlight/alert developers to unforeseen impact from changes and alert them to things they may not have considered. > With regard to this thread in particular, it is of some interest, but > there are fundamental issues, including the fact that certain > languages (e.g. racket) expect module code to exist somewhere on the > file system. There are ways around many of these issues, in fact there > are likely many ways around any individual issue, so org babel needs > to systematically consider the issues and provide a clear > specification, or at least a guide for how such cases should be > handled. > > To give an example from one of my continual pain points: I start > writing python or racket in an org src block and then I want it to be > a library so that it can be reused by other code both inside and > outside the org file without having to resort to noweb. > This is an interesting point. I think a number of languages have challenges here. I run into very similar issues with Clojure. To some extent, I think this is a grey area within the literate programming paradigm. The original literate programming model was developed at a time when most languages were compiled rather then interpreted. You generated source code, compiled it and then ran it. These days, many 'modern' languages are based around an interpreter and concepts like 'just in time' compilation. In org mode, things become even more complex because in addition to generation (tangling) of code, we also want to have evaluation of code blocks, plus we have added the concept of 'sessions'. > What is the best way to handle this? I don't know. Right now I tangle > things and resort to awful hacks for the reuse-in-this-org-file case, but > I'm guessing there is a better generic solution which would allow _any_ > org block to be exported as a library instead of nowebbed in. > > Before jumping for any particular suggestion for how to handle this > we need to explore the diversity of cases that various ob langs > present, so that we can find a solution that will work for all of > them. After all, packaging code to a library for reuse is an > extremely common pattern that org babel should be able to > abstract, but it is a major undertaking, not just the addition of a > keyword here and there. > Agree. I'm not convinced we really understand the requirements here and more exploration and specification is required. The more we add 'simple' extensions, options, keywords etc the more likely it is we will make this a much harder process and will likely result in even larger 'breakage' once we do define a clearer specification. > In short I suggest that we issue a general moratorium on new org babel > feature suggestions until we can stabilize what we already have and > provide a clear specification for correct behavior. Until we have that spec > we could encourage users to create extensions that implement those > features. > Yes. While it might sound harsh or overly limiting, I do think such a moratorium may be required. We may be able to lift it once we have some core specifications in place. We would still accept bug fixes (though we may need to provide some clarity on what is a bug fix and what is a feature enhancement/change - I regularly see posts flagged as bug fixes which are actually feature enhancements or extensions). > > PS The other next thing that I am working on might be another way out > for this particular feature request. Namely, it is simplifying and > extending org keyword syntax so that new keywords (with options) and > associated keywords can be specified using keyword syntax
Re: A requires/provides approach to linking source code blocks
In implementing an extension you might consider doing it as a generalized form of what I did with ob-racket (https://github.com/togakangaroo/ob-racket ). I think it is best to rely on the import/require/include mechanism of the language you're using. Pretty much all of them support adjacent files. The problem with code block execution is that it expands the block and writes it to a temporary directory in a non-predictable location. If you want to reference other blocks, you need them to *also* write into files in the same folder. What I do in ob-racket is support a header :adjacent-file which allows you to pass in the name of another block and then writes it into the same folder as the file to be executed with a predictable name (the name of the block) So something like this #+begin_src racket :adjacent-file stacker-reader-expander.rkt #lang reader "./stacker-reader-expander.rkt" Will make sure the block named `stacker-reader-expander.rkt` is written into that same temporary directory with that same name. There's a few rough edges but for the most part it works well. Would love to see someone take this and generalize it into its own extension. On Tue, Jul 13, 2021 at 2:18 PM Tom Gillespie wrote: > We have been receiving many new feature suggestions and requests > coming in for org babel. I think that Tim's suggestion is the right > one. Nearly all of these need to be implemented as an extension first > and tested independently. Further, even if this is done, it should be > clear that there is zero expectation that such extensions will be > incorporated. > > Once I wrap up the formal grammar for org, one of the next things I > plan to work on is a clear specification for org babel. This is > critical because so many of the suggestions that come in deal with > individuals' specific problems and thus fail to account for how such > features interact with existing features and how the newly proposed > feature would block some other features in the future, confuse users, > etc. Such suggestions also often fail to account for increased > complexity, nor have they been exposed to a sufficient number of > examples to reveal fundamental ambiguities in how they could be > interpreted. The issues with variable behavior between ob langs for > :pre :post :prologue :epilogue etc. are already enough to keep us busy > for quite some time. > > With regard to this thread in particular, it is of some interest, but > there are fundamental issues, including the fact that certain > languages (e.g. racket) expect module code to exist somewhere on the > file system. There are ways around many of these issues, in fact there > are likely many ways around any individual issue, so org babel needs > to systematically consider the issues and provide a clear > specification, or at least a guide for how such cases should be > handled. > > To give an example from one of my continual pain points: I start > writing python or racket in an org src block and then I want it to be > a library so that it can be reused by other code both inside and > outside the org file without having to resort to noweb. > > What is the best way to handle this? I don't know. Right now I tangle > things and resort to awful hacks for the reuse-in-this-org-file case, but > I'm guessing there is a better generic solution which would allow _any_ > org block to be exported as a library instead of nowebbed in. > > Before jumping for any particular suggestion for how to handle this > we need to explore the diversity of cases that various ob langs > present, so that we can find a solution that will work for all of > them. After all, packaging code to a library for reuse is an > extremely common pattern that org babel should be able to > abstract, but it is a major undertaking, not just the addition of a > keyword here and there. > > In short I suggest that we issue a general moratorium on new org babel > feature suggestions until we can stabilize what we already have and > provide a clear specification for correct behavior. Until we have that spec > we could encourage users to create extensions that implement those > features. > > Best, > Tom > > > PS The other next thing that I am working on might be another way out > for this particular feature request. Namely, it is simplifying and > extending org keyword syntax so that new keywords (with options) and > associated keywords can be specified using keyword syntax within a > single org file. This would make it possible to get useful high level > keyword behavior in a single file without burdening the core > implementation with more special cases for associated keywords, and it > would allow users to write small elisp functions that could do some of > what is suggested here, all without need to add anything to the core > org implementation. > >
Re: A requires/provides approach to linking source code blocks
We have been receiving many new feature suggestions and requests coming in for org babel. I think that Tim's suggestion is the right one. Nearly all of these need to be implemented as an extension first and tested independently. Further, even if this is done, it should be clear that there is zero expectation that such extensions will be incorporated. Once I wrap up the formal grammar for org, one of the next things I plan to work on is a clear specification for org babel. This is critical because so many of the suggestions that come in deal with individuals' specific problems and thus fail to account for how such features interact with existing features and how the newly proposed feature would block some other features in the future, confuse users, etc. Such suggestions also often fail to account for increased complexity, nor have they been exposed to a sufficient number of examples to reveal fundamental ambiguities in how they could be interpreted. The issues with variable behavior between ob langs for :pre :post :prologue :epilogue etc. are already enough to keep us busy for quite some time. With regard to this thread in particular, it is of some interest, but there are fundamental issues, including the fact that certain languages (e.g. racket) expect module code to exist somewhere on the file system. There are ways around many of these issues, in fact there are likely many ways around any individual issue, so org babel needs to systematically consider the issues and provide a clear specification, or at least a guide for how such cases should be handled. To give an example from one of my continual pain points: I start writing python or racket in an org src block and then I want it to be a library so that it can be reused by other code both inside and outside the org file without having to resort to noweb. What is the best way to handle this? I don't know. Right now I tangle things and resort to awful hacks for the reuse-in-this-org-file case, but I'm guessing there is a better generic solution which would allow _any_ org block to be exported as a library instead of nowebbed in. Before jumping for any particular suggestion for how to handle this we need to explore the diversity of cases that various ob langs present, so that we can find a solution that will work for all of them. After all, packaging code to a library for reuse is an extremely common pattern that org babel should be able to abstract, but it is a major undertaking, not just the addition of a keyword here and there. In short I suggest that we issue a general moratorium on new org babel feature suggestions until we can stabilize what we already have and provide a clear specification for correct behavior. Until we have that spec we could encourage users to create extensions that implement those features. Best, Tom PS The other next thing that I am working on might be another way out for this particular feature request. Namely, it is simplifying and extending org keyword syntax so that new keywords (with options) and associated keywords can be specified using keyword syntax within a single org file. This would make it possible to get useful high level keyword behavior in a single file without burdening the core implementation with more special cases for associated keywords, and it would allow users to write small elisp functions that could do some of what is suggested here, all without need to add anything to the core org implementation.
Re: A requires/provides approach to linking source code blocks
> On 9. Jul 2021, at 19:01, autofrettage wrote: > > Tim wrote: >> This could just be me, but recently, I'm becoming very concerned >> about the growth of additional features and options in org mode. > > Count me in. I have been mostly been hanging around in the shadows, but this > is serious enough for me to wave a flag on the right side. > […] Same concern here. — Alex smime.p7s Description: S/MIME cryptographic signature
Re: A requires/provides approach to linking source code blocks
Tim, > On Jul 8, 2021, at 12:32 PM, Tim Cross wrote: > > My concern here is with the additional complexity. This is already a > somewhat complex aspect of org mode and the behaviour you describe can > effectively be done using noweb, although as you say, not as > declarative in style. This (and the rest of what you said) is very well taken. I share your concern that adding features to an already rich feature suite will make babel very difficult to penetrate. Best, Chuck p.s. Extracting code from src blocks need not depend on `org-babel-tangle' and friends. Custom exporters have to potential to render code using attribute keys and other export features. And such exporters have the virtue of not adding complexity to the org code base.
Re: A requires/provides approach to linking source code blocks
Tim wrote: > This could just be me, but recently, I'm becoming very concerned > about the growth of additional features and options in org mode. Count me in. I have been mostly been hanging around in the shadows, but this is serious enough for me to wave a flag on the right side. I would go as far as saying that several suggestions have been so niche, as to be labeled feature bloats. They can be made available as user-added extensions through melpa, but should stay outside org itself. just my ¢2 Rasmus
Re: A requires/provides approach to linking source code blocks
On 09/07/2021 02:32, Tim Cross wrote: Marko Schuetz-Schmuck writes: I would find it useful to have a more declarative way for specifying sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to capture dependency and then have the exporter compute a sequence satisfying these. I would think that this makes the maintenance of the dependencies more convenient. My concern here is with the additional complexity. This is already a somewhat complex aspect of org mode and the behaviour you describe can effectively be done using noweb, although as you say, not as declarative in style. There was a feature request for dependencies between code blocks to run them in proper order a half of a year ago. I think, it is better to try such feature as an extension of org, as a separate package. I suspect that some non-obvious issue may arise. Likely, to be convenient, it will be desired to autofill requires/provides attributes using some tools related to particular language. The question is whether org code is organized in such way, so extensions of this kind are possible without code duplication. Maybe API could be adjusted. On the other hand, even maintaining of stable semi-internal API sometimes is significant burden. P.S. I am reading the mail list through NNTP news.gmain.io. I have noticed that personal copies sometimes arrives without mail list address. Sorry if you will have to manually add it to reply.
Re: A requires/provides approach to linking source code blocks
Marko Schuetz-Schmuck writes: > I would find it useful to have a more declarative way for specifying > sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to > capture dependency and then have the exporter compute a sequence > satisfying these. I would say that declaring an explicit ordering of blocks via noweb is quite declarative. The tensions seems more to be between implicit and explicit ordering. I'm quite a fan of letting the computer compute things, but in the context of literate programming I tend to prefer explicit over implicit. IMHO that's the main point of literate programming: Be more explicit and document all the details. Try not to hide important aspects. If you have so many blocks that maintaining the order of the concatenation seems like a burden and tiresome I would argue that you are doing it wrong. Because in the end you need quite a good understanding of the final order of the blocks or else maintaining the implicit order via require/provide will also get out of hand. I also second Tims concerns about the additional complexity and feature creep. -- Until the next mail..., Stefan.
Re: A requires/provides approach to linking source code blocks
Marko Schuetz-Schmuck writes: > [[PGP Signed Part:Undecided]] > Dear All, > > AFAIU in the current support for literate programming I can establish > sequence between blocks by either tangling the entire file whereby the > blocks are written to the source code file in the sequence in which they > appear in the org-mode file or I can name the blocks and use noweb > linking to explicitly state the precise sequence in which the blocks > appear in the source code file. > > I would find it useful to have a more declarative way for specifying > sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to > capture dependency and then have the exporter compute a sequence > satisfying these. I would think that this makes the maintenance of the > dependencies more convenient. > > I'd be interested in feedback on the idea. > > Please include my email address in the reply as I am not subscribed to > the list. > My concern here is with the additional complexity. This is already a somewhat complex aspect of org mode and the behaviour you describe can effectively be done using noweb, although as you say, not as declarative in style. I'm not sure the added complexity is worth the additional maintenance overhead and I would be concerned regarding how much confusion this could cause i.e. someone uses both noweb and declarative provide/require. These additions are also likely to hamper any efforts to improve performance - a topic which comes up on a fairly regular basis. This could just be me, but recently, I'm becoming very concerned about the growth of additional features and options in org mode. Many of the new things being requested are for relatively narrow use cases and represent functionality which can largely be achieved with existing features. At the same time, there also seems to be a growing frequency in patches which introduce regression bugs and patches to fix patches. This is all beginning to feel like we are running very close to the tipping point at which time we will have something that is so complex that only a very few people are able to maintain the code base and keep the system stable. New maintainers are discouraged because of the code complexity. We could end up in a position where really important issues cannot be addressed or addressed in a timely manner because of the overall complexity of the code base and time it takes to fix and test and dependence on a very few number of maintainers who are already swamped with work. At some point, I think we may want to consider a temporary freeze on new functionality and spend a few months just focusing on bug fixes and code base improvements or re-factoring. It might also be worthwhile providing some guidelines or criteria/procedures for assessing proposed new features to avoid a perception of new features being accepted/rejected based on personality, loudness of voice or some other real or perceived and irrelevant basis. -- Tim Cross
