Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Evgenii Klimov]

Hi Ihor

On August 23, 2023 12:44:32 PM GMT+01:00, Ihor Radchenko  
wrote:
>Ihor Radchenko  writes:
>
>> What about generalizing the idea and providing a way to set Emacs
>> buffer-local variables in the tangled files?
>>
>> Can be something like:
>>
>> #+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))
>
>Over one month have passed since the last message in this thread.
>
>Evgenii, are you still interested to continue working on this?
>

I am busy for the next month and even after that generalizing the idea would 
take too much effort and time, since I'm not an advanced programmer. So 
probably the answer is no.

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Ihor Radchenko  writes:

> What about generalizing the idea and providing a way to set Emacs
> buffer-local variables in the tangled files?
>
> Can be something like:
>
> #+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))

Over one month have passed since the last message in this thread.

Evgenii, are you still interested to continue working on this?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Timothy  writes:

>> I do not think that setting `lexical-binding’ file-local variable in an
>> Org file makes much sense. I am sure that we can do better.
>
> I suppose part of the question is what sort of way we treat it? In my mind,
> considering the current way lexical scope is seen in Emacs (used everywhere in
> modern elisp, not on-by-default because that would break compat, is my
> impression). I think it would make sense if this eventually ends up with a
> “globally enabled everywhere by default” setting, which is why I think a
> defcustom may be the best fit.

Do you mean to enable :lexical t in all the elisp source blocks?
It is already possible (and done) using 
`org-babel-default-header-args:emacs-lisp'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

"Berry, Charles"  writes:

>> IIRC, there wasn't much discussion.   The current situation doesn't seem 
>> ripe for documentation in the manual.
>
> Agreed. Maybe it would suffice to use enhanced docstrings for the 
> org-babel-execute: functions and point to them in the manual.

IMHO, this is worse than pointing to WORG pages, as we currently do.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Berry, Charles]

> On Jul 15, 2023, at 9:11 AM, Thomas S. Dye  wrote:
> 
> 
> Ihor Radchenko  writes:
> 
>> "Thomas S. Dye"  writes:
>> 
 Am I correct that language-specific header arguments are not yet covered
 in the manual?  I can't find any reference of "lexical" there.
>>> 
>>> Yes, I believe so.  Language-specific header arguments for many languages 
>>> are documented at 
>>> https://orgmode.org/worg/org-contrib/babel/languages/index.html. Several 
>>> languages are not documented there; in these cases the documentation is 
>>> typically in the source code.
>> 
>> It is actually slightly awkward that built-in babel backends are not
>> documented in the manual, while, for example, export backends are.
>> 
> Agreed, but the two differ.  With export, the goal is to produce code 
> recognized as correct by the target software.  With babel, the general goal 
> is to provide language agnostic literate programming, which often produces 
> something that meets standards of correctness not implemented in software.  
> The idea of built-in babel backends, as distinct from contributed backends, 
> or backends distributed by package archive is a function of maintenance 
> burden and distribution channel, rather than something intrinsic to the 
> backend itself and how it contributes to literate programming.  
>> However, unless my memory deceives me, Bastien or Nicolas previously
>> voiced against adding babel documentation. Though I cannot find the
>> relevant discussion now and cannot recall the arguments (if there was
>> such discussion at all).
> 
> IIRC, there wasn't much discussion.   The current situation doesn't seem ripe 
> for documentation in the manual.
> 


Agreed. Maybe it would suffice to use enhanced docstrings for the 
org-babel-execute: functions and point to them in the manual.


The params argument could list the language specific keys and the language 
specific handling of key common to all languages (e.g. `:results graphics' in R 
jumps through a number of hoops that might deserve mention)

Chuck 


> Here are some potential hurdles:
> - there are likely too many built-in backends;
> - the built-in backends are a mixed bag--ob-lua seems half-finished to me, 
> though I don't program in lua and struggled to set up the language to write 
> the documentation stub on Worg;
> - nearly a dozen of the built-in babel backends lack documentation outside 
> the source code (see 
> https://orgmode.org/worg/org-contrib/babel/languages/index.html#orgbc466c5); 
> and
> - language support is inconsistent (see 
> https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html), which 
> introduces complications for language agnostic literate programming.
> 
> One way forward might distinguish between babel backends for GNU software and 
> babel backends for non-GNU software, with the former built in, guaranteed to 
> be consistent to some standard (which needs to be formulated), and documented 
> in the manual and the latter moved to contrib or a package archive, with 
> documentation (if any) on Worg.
> 
> Others might have better ideas.
> 
> All the best,
> Tom
> -- 
> Thomas S. Dye
> https://tsdye.online/tsdye
> 
> 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Thomas S. Dye]

Ihor Radchenko  writes:


"Thomas S. Dye"  writes:

Am I correct that language-specific header arguments are not 
yet 
covered

in the manual?  I can't find any reference of "lexical" there.


Yes, I believe so.  Language-specific header arguments for many 
languages are documented at 
https://orgmode.org/worg/org-contrib/babel/languages/index.html. 
Several languages are not documented there; in these cases the 
documentation is typically in the source code.


It is actually slightly awkward that built-in babel backends are 
not
documented in the manual, while, for example, export backends 
are.


Agreed, but the two differ.  With export, the goal is to produce 
code recognized as correct by the target software.  With babel, 
the general goal is to provide language agnostic literate 
programming, which often produces something that meets standards 
of correctness not implemented in software.  The idea of built-in 
babel backends, as distinct from contributed backends, or backends 
distributed by package archive is a function of maintenance burden 
and distribution channel, rather than something intrinsic to the 
backend itself and how it contributes to literate programming.  

However, unless my memory deceives me, Bastien or Nicolas 
previously
voiced against adding babel documentation. Though I cannot find 
the
relevant discussion now and cannot recall the arguments (if 
there was

such discussion at all).


IIRC, there wasn't much discussion.   The current situation 
doesn't seem ripe for documentation in the manual.


Here are some potential hurdles:
- there are likely too many built-in backends;
- the built-in backends are a mixed bag--ob-lua seems 
half-finished to me, though I don't program in lua and struggled 
to set up the language to write the documentation stub on Worg;
- nearly a dozen of the built-in babel backends lack 
documentation outside the source code (see 
https://orgmode.org/worg/org-contrib/babel/languages/index.html#orgbc466c5); 
and
- language support is inconsistent (see 
https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html), 
which introduces complications for language agnostic literate 
programming.


One way forward might distinguish between babel backends for GNU 
software and babel backends for non-GNU software, with the former 
built in, guaranteed to be consistent to some standard (which 
needs to be formulated), and documented in the manual and the 
latter moved to contrib or a package archive, with documentation 
(if any) on Worg.


Others might have better ideas.

All the best,
Tom
--
Thomas S. Dye
https://tsdye.online/tsdye

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Timothy]

Hi Ihor,

> I do not think that setting `lexical-binding’ file-local variable in an
> Org file makes much sense. I am sure that we can do better.

I suppose part of the question is what sort of way we treat it? In my mind,
considering the current way lexical scope is seen in Emacs (used everywhere in
modern elisp, not on-by-default because that would break compat, is my
impression). I think it would make sense if this eventually ends up with a
“globally enabled everywhere by default” setting, which is why I think a
defcustom may be the best fit.

> I can see that there are people in favour of :lexical feature.
> So, we can probably add it, but it should not be in ob-core.
> Instead, we should provide an infrastructure allowing ob-* backends to
> modify tangling. Maybe some kind of special function that will override
> default `org-babel-spec-to-string’ per backend?

Mmm, something generic for backend-specific modification sounds like a good idea
to me.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at .
Support Org development at ,
or support my work at .

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

"Thomas S. Dye"  writes:

>> Am I correct that language-specific header arguments are not yet 
>> covered
>> in the manual?  I can't find any reference of "lexical" there.
>
> Yes, I believe so.  Language-specific header arguments for many 
> languages are documented at 
> https://orgmode.org/worg/org-contrib/babel/languages/index.html. 
> Several languages are not documented there; in these cases the 
> documentation is typically in the source code.

It is actually slightly awkward that built-in babel backends are not
documented in the manual, while, for example, export backends are.

However, unless my memory deceives me, Bastien or Nicolas previously
voiced against adding babel documentation. Though I cannot find the
relevant discussion now and cannot recall the arguments (if there was
such discussion at all).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Timothy  writes:

> It occurs to me that this use case could also perhaps be satisfied by 
> file-local
> variables? If we presume that mixing tangling to lexically-bound and
> non-lexically bound elisp files is a corner case we don’t care that much 
> about,
> a `org-babel-elisp-lexical' variable could be used to set the behaviour, and
> modified using file-local variable forms as usual.

I do not think that setting `lexical-binding' file-local variable in an
Org file makes much sense. I am sure that we can do better.

I can see that there are people in favour of :lexical feature.
So, we can probably add it, but it should not be in ob-core.
Instead, we should provide an infrastructure allowing ob-* backends to
modify tangling. Maybe some kind of special function that will override
default `org-babel-spec-to-string' per backend?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Thomas S. Dye]

Evgenii Klimov  writes:


No Wayman  writes:

[...]
I, and others, have been surprised to find that the tangled 
file does
not have lexical binding enabled when :lexical blocks are 
tangled.


Am I correct that language-specific header arguments are not yet 
covered

in the manual?  I can't find any reference of "lexical" there.


Yes, I believe so.  Language-specific header arguments for many 
languages are documented at 
https://orgmode.org/worg/org-contrib/babel/languages/index.html. 
Several languages are not documented there; in these cases the 
documentation is typically in the source code.


All the best,
Tom

--
Thomas S. Dye
https://tsdye.online/tsdye

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Evgenii Klimov]

No Wayman  writes:

[...]
> I, and others, have been surprised to find that the tangled file does
> not have lexical binding enabled when :lexical blocks are tangled.

Am I correct that language-specific header arguments are not yet covered
in the manual?  I can't find any reference of "lexical" there.

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[No Wayman]

From: Ihor Radchenko 
I am not sure if I like this approach.
I have 2 problems with the patch:



1. Previous users of :lexical header arg might be surprised.
   Though it is an OK breaking change since people who used 
   :lexical
   argument and expected it to be ignored in the tangled file 
   probably

   did something wrong.


Previous related discussion:

https://list.orgmode.org/87o89184h1@gmail.com/

I, and others, have been surprised to find that the tangled file 
does not have lexical binding enabled when :lexical blocks are 
tangled.

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Timothy]

Hi Ihor,

> 2. More importantly, we are adding ob-emacs-lisp-specific handling into
>generic ob-tangle.
>
> What about generalizing the idea and providing a way to set Emacs
> buffer-local variables in the tangled files?

Looking at this patch earlier, I had the same concern. I think it’s worth being
paranoid about a proliferation of overly-specialised ob-tangle arguments.

It occurs to me that this use case could also perhaps be satisfied by file-local
variables? If we presume that mixing tangling to lexically-bound and
non-lexically bound elisp files is a corner case we don’t care that much about,
a `org-babel-elisp-lexical' variable could be used to set the behaviour, and
modified using file-local variable forms as usual.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at .
Support Org development at ,
or support my work at .

Re: [Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[Ihor Radchenko]

Evgenii Klimov  writes:

> diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
> index a4725ae8c..c632c4814 100644
> --- a/etc/ORG-NEWS
> +++ b/etc/ORG-NEWS
> @@ -284,6 +284,13 @@ setting the ~STYLE~ property for each sub-task.
>  The change is breaking when ~org-use-property-inheritance~ is set to ~t~.
>  
>  ** New and changed options
> +*** =:lexical= header argument now influences tangled files
> +
> +When extracting an Elisp src block, the target's file
> +~lexical-binding~ variable is set according to the src block's
> +=:lexical= parameter.  If at least one block is set to lexical scope,
> +then the whole file will be with lexical scope.

I am not sure if I like this approach.
I have 2 problems with the patch:

1. Previous users of :lexical header arg might be surprised.
   Though it is an OK breaking change since people who used :lexical
   argument and expected it to be ignored in the tangled file probably
   did something wrong.
   
2. More importantly, we are adding ob-emacs-lisp-specific handling into
   generic ob-tangle.

What about generalizing the idea and providing a way to set Emacs
buffer-local variables in the tangled files?

Can be something like:

#+begin_src elisp :file-locals (lexical-binding t eval (line-number-mode))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[Pre-PATCH v2] Add the capability to specify lexical scope in tangled files (was: Add new :lexical header argument)

[General discussions about Org-mode.]

Ihor Radchenko  writes:

> Evgenii Klimov  writes:
>
>> * lisp/ob-core.el (org-babel-common-header-args-w-values): Add new
>> :lexical header argument.
>
> A short note: ob-emacs-lisp already defines :lexical header arg. See
> org-babel-header-args:emacs-lisp.

Thanks for pointing it out. Updated the patch to reflect your note and
documented the change.

>From 7e75b55ccb4e82d9341b6b308b558e5d01df5128 Mon Sep 17 00:00:00 2001
From: Evgenii Klimov 
Date: Thu, 13 Jul 2023 18:16:08 +0100
Subject: [PATCH v2] ob-tangle.el: Add the capability to specify lexical scope
 in tangled files

* lisp/ob-tangle.el (org-babel-tangle): Add the ability to enable
lexical binding in Elisp tangled file.

* etc/ORG-NEWS (=:lexical= header argument now influences tangled
files): Document this change.

Previously one could achieve this manually, but 1) had to keep in mind
to hold this source block top-most, 2) couldn't use comment header
argument, as `lexical-binding' variable must be set in the first line
of a file.
---
 etc/ORG-NEWS  | 7 +++
 lisp/ob-tangle.el | 9 +++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index a4725ae8c..c632c4814 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -284,6 +284,13 @@ setting the ~STYLE~ property for each sub-task.
 The change is breaking when ~org-use-property-inheritance~ is set to ~t~.
 
 ** New and changed options
+*** =:lexical= header argument now influences tangled files
+
+When extracting an Elisp src block, the target's file
+~lexical-binding~ variable is set according to the src block's
+=:lexical= parameter.  If at least one block is set to lexical scope,
+then the whole file will be with lexical scope.
+
 *** Commands affected by ~org-fold-catch-invisible-edits~ can now be customized
 
 New user option ~org-fold-catch-invisible-edits-commands~ controls
diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 1274d0db7..10fe5dcf5 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -262,7 +262,7 @@ expression."
 	 (when file-name
(let ((lspecs (cdr by-fn))
 		 (fnd (file-name-directory file-name))
-		 modes make-dir she-banged lang)
+		 modes make-dir she-banged scoped lang)
 	 ;; drop source-blocks to file
 	 ;; We avoid append-to-file as it does not work with tramp.
 	 (with-temp-buffer
@@ -273,6 +273,8 @@ expression."
 			 (get-spec (lambda (name) (cdr (assq name (nth 4 spec)
 			 (she-bang (let ((sheb (funcall get-spec :shebang)))
  (when (> (length sheb) 0) sheb)))
+ (lexicalp (org-babel-emacs-lisp-lexical
+(funcall get-spec :lexical)))
 			 (tangle-mode (funcall get-spec :tangle-mode)))
 		(unless (string-equal block-lang lang)
 			  (setq lang block-lang)
@@ -294,7 +296,10 @@ expression."
 		(when (and she-bang (not she-banged))
 			  (insert (concat she-bang "\n"))
 			  (setq she-banged t))
-		(org-babel-spec-to-string spec)
+(org-babel-spec-to-string spec)
+		(when (and lexicalp (not scoped))
+			  (save-excursion (elisp-enable-lexical-binding))
+			  (setq scoped t))
 		(setq block-counter (+ 1 block-counter
 		lspecs)
 		   (when make-dir
-- 
2.34.1