Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

 Original Message 
On 8/31/24 10:43 AM, Ihor Radchenko  wrote:

>  Michael Mauger  writes:
>  
>  > Bad news :(  I do not have a reproducible set of steps, but I'll start 
> looking for one. I am not using the fix you sent and am using the latest 
> master of Emacs 30.0.50 and Org 9.7.10

>  Maybe something is setting `print-level' per-buffer?
>  

(Don't have access rn; Holiday weekend here…) 

I will check but I don't set `print-level' anywhere in my code and it's `let' 
bound in the org-persist code. I'll do some digging on where things go bad when 
I get home. My gut feeling is that the damage is done elsewhere and doesn't 
manifest itself until it's persisted.

>  --
>  Ihor Radchenko // yantar92,
>  Org mode contributor,
>  Learn more about Org mode at <https://orgmode.org/>.
>  Support Org development at <https://liberapay.com/org-mode>,
>  or support my work at <https://liberapay.com/yantar92>
>

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger  writes:

> Bad news :(  I do not have a reproducible set of steps, but I'll start 
> looking for one. I am not using the fix you sent and am using the latest 
> master of Emacs 30.0.50 and Org 9.7.10
>
> My init is a set of Org Babel scripts (4.5K org lines; 2.8K el lines) which 
> is the latest incarnation of 30+ years of Emacs. Since this problem started, 
> I start Emacs by first removing the org-persist files and the .el files, so 
> that all the babel code is tangled. After a bunch of debugging and edits to 
> the org files, I saved and went to start Emacs again with a clean slate.
>
> Unfortunately, the `index.eld' file had the following (note the "..." in the 
> file contents):
> ...
> So it was the process of editing the SRC blocks and executing the elisp code 
> to test that resulted in the bad persist file. I'm thinking I may add some 
> code to see if the elisp variable is corrupt or it's just happening on 
> output. My sense is this may be a core bug but I'll see what I can find.

Maybe something is setting `print-level' per-buffer?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

On Sunday, August 25th, 2024 at 3:31 AM, Ihor Radchenko  
wrote:

> Michael Mauger michael.mau...@protonmail.com writes:
> 
> > Sorry for not getting back to you sooner. This change has seemed to work, 
> > but I haven't been playing with my Org Babel config much lately. I'll let 
> > you know if I see it again.
> 
> 
> FYI, we have disabled pretty printing for performance reasons. So, the
> suggested change you may be testing is already included in Org 9.7.10
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?h=bugfix&id=f9351456e70ad08dea74525e4a7d4bd29f98176d
> 

Bad news :(  I do not have a reproducible set of steps, but I'll start looking 
for one. I am not using the fix you sent and am using the latest master of 
Emacs 30.0.50 and Org 9.7.10

My init is a set of Org Babel scripts (4.5K org lines; 2.8K el lines) which is 
the latest incarnation of 30+ years of Emacs. Since this problem started, I 
start Emacs by first removing the org-persist files and the .el files, so that 
all the babel code is tangled. After a bunch of debugging and edits to the org 
files, I saved and went to start Emacs again with a clean slate.

Unfortunately, the `index.eld' file had the following (note the "..." in the 
file contents):

((:container ((elisp org-element--headline-cache) (elisp 
org-element--cache) (version "2.3")) :persist-file 
"ad/c72d74-f821-4190-97c8-41feb7dc8128" :associated (:hash 
"a82d8fdcbf03df59c72ca1df20680b3a" :file 
"/home/michael/Projects/my-config/emacs/my-config.org" :inode 2975404) :expiry 
30 :last-access 1724901536.6903014 ...) (:container ((elisp 
org-element--headline-cache) (elisp org-element--cache) (version "2.3")) 
:persist-file "a6/436395-1c62-4060-8afc-53893e044fd9" :associated (:hash 
"8a94101fb7eb8f1f6b6a4ba89f5405e3" :file 
"/home/michael/Projects/my-config/emacs/my-location.org" :inode 2974753) 
:expiry 30 :last-access 1724809178.237551 ...) (:container ((elisp 
org-element--headline-cache) (elisp org-element--cache) (version "2.3")) 
:persist-file "00/1fd972-5079-4495-ab04-88db0f0a0e6c" :associated (:hash 
"feb07f13f10cab551dc03089e1919594" :file 
"/home/michael/Projects/my-config/emacs/my-eshell.org" :inode 2979957) :expiry 
30 :last-access 1724805634.696495 ...) (:container ((elisp 
org-element--headline-cache) (elisp org-element--cache) (version "2.3")) 
:persist-file "be/57de97-a696-43b4-964c-d9817a54ba0f" :associated (:hash 
"981e2ee1ce9a79044fe39e31e865ff78" :file 
"/home/michael/Projects/my-config/emacs/my-org.org" :inode 2974714) :expiry 30 
:last-access 1724805423.0839553 ...) (:container ((elisp 
org-element--headline-cache) (elisp org-element--cache) (version "2.3")) 
:persist-file "be/052624-e6cf-4034-b29a-dd0d589fec21" :associated (:hash 
"c9f58d9d1fd1f7f289d35af18aadcbd8" :file 
"/home/michael/Projects/my-config/emacs/my-library-of-babel.org" :inode 
2974478) :expiry 30 :last-access 1724805419.3735616 ...) (:container ((index 
"3.2")) :persist-file "63/45cf0a-1428-489e-b247-a4e501140c2c" :associated nil 
:expiry never :last-access 1724908792.4755692 ...))

So it was the process of editing the SRC blocks and executing the elisp code to 
test that resulted in the bad persist file. I'm thinking I may add some code to 
see if the elisp variable is corrupt or it's just happening on output. My sense 
is this may be a core bug but I'll see what I can find.

> --
> Ihor Radchenko // yantar92,
> Org mode contributor,

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger  writes:

> Sorry for not getting back to you sooner. This change has seemed to work, but 
> I haven't been playing with my Org Babel config much lately. I'll let you 
> know if I see it again.

FYI, we have disabled pretty printing for performance reasons. So, the
suggested change you may be testing is already included in Org 9.7.10
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?h=bugfix&id=f9351456e70ad08dea74525e4a7d4bd29f98176d

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] org-persist-write-all is slow because of use of pp [9.8-pre (release_9.7.8-713-g62cbac @ /home/viz/lib/emacs/straight/build/org/)]

[Ihor Radchenko]

Visuwesh  writes:

>> So much for the idea of readable index file.
>
> It is unfortunate, indeed, since the pp version of the file is really
> handy when looking into org-persist issues.  Though, I must mention that
> the slowness reported in OP is compounded by my fairly aggressive CPU
> governor settings (in hopes of improving battery life).

AFAIK, the underlying cause is bad scaling of `pp' algo. So, it will
only get worse (like O(N^2) worse) with increasing number of index
entries. Different CPU settings will just change the threshold when the
problem is triggered.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] org-persist-write-all is slow because of use of pp [9.8-pre (release_9.7.8-713-g62cbac @ /home/viz/lib/emacs/straight/build/org/)]

[Visuwesh]

[சனி ஆகஸ்ட் 10, 2024] Ihor Radchenko wrote:

> Visuwesh  writes:
>
>> I am using the async LaTeX preview branch and org-persist-write-all
>> takes a lot of time to finish executing due to the use of pp when saving
>> the index file.  Changing org-persist-write:index to not pass a non-nil
>> PP argument to org-persist--write-elisp-file changes the execution time
>> of org-persist-write-all from 62 secs to just 8 secs.  For reference,
>>
>> (length org-persist--index) ;; ⇒ 1504
>>
>> I have attached the profiler report for org-persist--index of much
>> smaller length but if required I can reproduce a report for the full one
>> too.
>
> So much for the idea of readable index file.

It is unfortunate, indeed, since the pp version of the file is really
handy when looking into org-persist issues.  Though, I must mention that
the slowness reported in OP is compounded by my fairly aggressive CPU
governor settings (in hopes of improving battery life).

> Fixed, on bugfix.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=f9351456e7

Thanks for the quick fix!

Re: [BUG] org-persist-write-all is slow because of use of pp [9.8-pre (release_9.7.8-713-g62cbac @ /home/viz/lib/emacs/straight/build/org/)]

[Ihor Radchenko]

Visuwesh  writes:

> I am using the async LaTeX preview branch and org-persist-write-all
> takes a lot of time to finish executing due to the use of pp when saving
> the index file.  Changing org-persist-write:index to not pass a non-nil
> PP argument to org-persist--write-elisp-file changes the execution time
> of org-persist-write-all from 62 secs to just 8 secs.  For reference,
>
>     (length org-persist--index) ;; ⇒ 1504
>
> I have attached the profiler report for org-persist--index of much
> smaller length but if required I can reproduce a report for the full one
> too.

So much for the idea of readable index file.
Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=f9351456e7

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Org-persist

[Ihor Radchenko]

Christian Moe  writes:

> I do have a backup of the org-persist directory if there's anything to
> be learned from it about how this happened (but I'm traveling, and don't
> have much time for forensics at the moment).

I don't think so. Not in this case.
I think that we simply need to do format check when reading the index
and refuse to load it when something is off (malformed lists or values).

AFAIU, what you see can happen in peculiar situations when the file is
corrupted, but partially - Emacs is still able to `read' it into Elisp,
but the result of reading is garbage.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Org-persist

[Christian Moe]

Fixed: I killed emacs, deleted ~/.cache/org-persist/, and things look
OK after restart.

Sorry, obvious solution, but I wasn't thinking straight last night.

I do have a backup of the org-persist directory if there's anything to
be learned from it about how this happened (but I'm traveling, and don't
have much time for forensics at the moment).

Yours,
Christian



Christian Moe writes:

> Hi,
>
> It seems my org-persist cache has somehow been corrupted so that opening
> any Org-mode file suddenly causes this odd and severe error:
>
> File mode specification error: (wrong-type-argument listp
> ..6C?.^c)
>
> Retyping the above since the characters may turn out wrong:
> (wrong-type-argument listp \277\243\254\224\3236\214C?\343\222^c)
>
> This essentially disables Org-mode completely. Only the first few lines
> are fontified, or more often none, and org functions like folding don't
> work. The same if I set the buffer to text-mode and then back to
> org-mode.
>
> What is a recommended quick fix?

[BUG] Org-persist

[Christian Moe]

Hi,

It seems my org-persist cache has somehow been corrupted so that opening
any Org-mode file suddenly causes this odd and severe error:

File mode specification error: (wrong-type-argument listp
£”6C?’^c)

Retyping the above since the characters may turn out wrong:
(wrong-type-argument listp \277\243\254\224\3236\214C?\343\222^c)

This essentially disables Org-mode completely. Only the first few lines
are fontified, or more often none, and org functions like folding don't
work. The same if I set the buffer to text-mode and then back to
org-mode.

What is a recommended quick fix?

--

A few extra notes:

Weirdly, after closing all buffers except for *scratch*, I keep getting
the same error when I try to exit Emacs, so I cannot exit.

I can open Org files with emacs -Q.

I don't know how the corruption happened. I have had some irregular
shutdowns, etc., but this appears to have happened in the middle of an
otherwise normal Emacs session.

Yours,
Christian

Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

Sorry for not getting back to you sooner. This change has seemed to work, but I 
haven't been playing with my Org Babel config much lately. I'll let you know if 
I see it again.

Thanks for your help.

--
mich...@mauger.com // FSF and SFConservancy member // GNU Emacs sql.el 
maintainer


On Monday, July 8th, 2024 at 1:39 PM, Ihor Radchenko  
wrote:

> Michael Mauger michael.mau...@protonmail.com writes:
> 
> > > > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
> > > 
> > > This is curious. May it be that you have non-default value of
> > > `pp-default-function'?
> > 
> > I don't touch `pp-default-function' and I've confirmed that it is set to` 
> > pp-fill' in my instance.
> 
> 
> Then, what if you re-define `org-persist-write:index' to
> 
> (defun org-persist-write:index (container _)
> "Write index CONTAINER."
> (org-persist--get-collection container)
> (unless (file-exists-p org-persist-directory)
> (condition-case nil
> (make-directory org-persist-directory 'parent)
> (t
> (warn "Failed to create org-persist storage in %s."
> org-persist-directory)
> (org-persist--check-write-access org-persist-directory
> (when (file-exists-p org-persist-directory)
> (let ((index-file
> (org-file-name-concat org-persist-directory org-persist-index-file)))
> (org-persist--merge-index-with-disk)
> ;; CHANGED: disable pretty-printing
> (org-persist--write-elisp-file index-file org-persist--index t nil)
> (setq org-persist--index-age
> (file-attribute-modification-time (file-attributes index-file)))
> index-file)))
> 
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at https://orgmode.org/.
> 
> Support Org development at https://liberapay.com/org-mode,
> 
> or support my work at https://liberapay.com/yantar92

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger  writes:

>> > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
>> 
>> 
>> This is curious. May it be that you have non-default value of
>> `pp-default-function'?
>> 
>
> I don't touch `pp-default-function' and I've confirmed that it is 
> set to `pp-fill' in my instance.

Then, what if you re-define `org-persist-write:index' to

(defun org-persist-write:index (container _)
  "Write index CONTAINER."
  (org-persist--get-collection container)
  (unless (file-exists-p org-persist-directory)
(condition-case nil
(make-directory org-persist-directory 'parent)
      (t
   (warn "Failed to create org-persist storage in %s."
         org-persist-directory)
   (org-persist--check-write-access org-persist-directory
  (when (file-exists-p org-persist-directory)
(let ((index-file
   (org-file-name-concat org-persist-directory org-persist-index-file)))
  (org-persist--merge-index-with-disk)
  ;; CHANGED: disable pretty-printing
  (org-persist--write-elisp-file index-file org-persist--index t nil)
  (setq org-persist--index-age
(file-attribute-modification-time (file-attributes index-file)))
  index-file)))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

On Monday, July 8th, 2024 at 11:45 AM, Ihor Radchenko  
wrote:

> Michael Mauger michael.mau...@protonmail.com writes:
> 
> > This bug has been difficult to reproduce but several people have
> > encountered it, or symptoms of it. Most recently, there has been this
> > thread on Reddit:
> > https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/
> > 

Just encountered it again. :( The recipe is still not obvious but I'm trying to 
find one... 
(This time got the "\..." version)

> > After some poking around I started getting failure when I tried to
> > reopen an Org file or ran `normal-mode' on it. Shutting down emacs 
> > and restarting cleared the issue. When the issue arose when reopening 
> > the file within a session, the backtrace was pointing to` plistp' failing in
> > `org-persist-read'. When I looked at the persist index there were
> > entries like:
> > 
> > (:container
> > ((elisp org-element--headline-cache) (elisp org-element--cache)
> > (version "2.3"))
> > :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
> > (:hash "aa1176747642fdac45aaf96095a88367" :file
> > "/home/michael/Projects/my-config/emacs/my-org.org" :inode
> > 2554490)
> > :expiry 30 :last-access 1719791738.4428508 :last-access-hr
> > "2024-06-30T19:55:38-0400" ...)
> > 
> > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
> 
> 
> This is curious. May it be that you have non-default value of
> `pp-default-function'?
> 

I don't touch `pp-default-function' and I've confirmed that it is 
set to `pp-fill' in my instance.

> --
> Ihor Radchenko // yantar92,
> Org mode contributor,

Thanks for your attention.

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger  writes:

> This bug has been difficult to reproduce but several people have
> encountered it, or symptoms of it. Most recently, there has been this
> thread on Reddit:
> https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/
>
> Unfortunately, replicating on emacs -Q has proven impossible since -Q
> disables some of the implicated features. My Emacs config is contained 
> in Org Babel files that are invoked with `org-babel-tangle-file' and have
> demonstrated the error multiple times.

Thanks for reporting!

> After some poking around I started getting failure when I tried to
> reopen an Org file or ran `normal-mode' on it. Shutting down emacs and
> restarting cleared the issue. When the issue arose when reopening the
> file within a session, the backtrace was pointing to `plistp' failing in
> `org-persist-read'. When I looked at the persist index there were
> entries like:
>
>  (:container
>   ((elisp org-element--headline-cache) (elisp org-element--cache)
>(version "2.3"))
>   :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
>   (:hash "aa1176747642fdac45aaf96095a88367" :file
>"/home/michael/Projects/my-config/emacs/my-org.org" :inode
>2554490)
>   :expiry 30 :last-access 1719791738.4428508 :last-access-hr
>   "2024-06-30T19:55:38-0400" ...)
>
> (Note: the "..." in the last entry; sometimes it showed up as "\...".)

This is curious. May it be that you have non-default value of
`pp-default-function'?

> The failure during tangling occurred with `org-get-heading' expecting
> `stringp' but getting `nil'. That seemed to be tracked down to
> `org-complex-heading-regexp' not being set yet. If an error had aborted
> out of the `org-persist-read' it appears we could have half of org-mode
> up but not everything properly initialized.

Sounds right.

> Looking at the code in `org-persist.el' in
> `org-persist--write-elisp-file (file data &optional no-circular pp)'
> around line 486 it handles calling `pp' or `prin1'. The code for `pp'
> uses `pp-use-max-width' (as part of a bug fix #58687) which has since
> been superseded in Emacs 30.

Yes, it should be fixed. I did not know that they quickly obsoleted that
variable.

> It looks like the `org-persist-write' code ought to be updated and the
> `org-persist-read' code made more robust so that it does not just shut
> down processing of Org files entirely. That is if the entry is not a
> plist, act as though no entry exists and proceed as if it were a new
> org file. 

Yes. The case when the contents of index file is a valid Elisp data, but
not the data we expect is currently not covered. It should be.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

31.0.50; Org Persist file formatting

[Michael Mauger]

This bug has been difficult to reproduce but several people have
encountered it, or symptoms of it. Most recently, there has been this
thread on Reddit:
https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/

Unfortunately, replicating on emacs -Q has proven impossible since -Q
disables some of the implicated features. My Emacs config is contained 
in Org Babel files that are invoked with `org-babel-tangle-file' and have
demonstrated the error multiple times.

After some poking around I started getting failure when I tried to
reopen an Org file or ran `normal-mode' on it. Shutting down emacs and
restarting cleared the issue. When the issue arose when reopening the
file within a session, the backtrace was pointing to `plistp' failing in
`org-persist-read'. When I looked at the persist index there were
entries like:

 (:container
  ((elisp org-element--headline-cache) (elisp org-element--cache)
   (version "2.3"))
  :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
  (:hash "aa1176747642fdac45aaf96095a88367" :file
 "/home/michael/Projects/my-config/emacs/my-org.org" :inode
 2554490)
  :expiry 30 :last-access 1719791738.4428508 :last-access-hr
  "2024-06-30T19:55:38-0400" ...)

(Note: the "..." in the last entry; sometimes it showed up as "\...".)

The failure during tangling occurred with `org-get-heading' expecting
`stringp' but getting `nil'. That seemed to be tracked down to
`org-complex-heading-regexp' not being set yet. If an error had aborted
out of the `org-persist-read' it appears we could have half of org-mode
up but not everything properly initialized.

Looking at the code in `org-persist.el' in
`org-persist--write-elisp-file (file data &optional no-circular pp)'
around line 486 it handles calling `pp' or `prin1'. The code for `pp'
uses `pp-use-max-width' (as part of a bug fix #58687) which has since
been superseded in Emacs 30.

It looks like the `org-persist-write' code ought to be updated and the
`org-persist-read' code made more robust so that it does not just shut
down processing of Org files entirely. That is if the entry is not a
plist, act as though no entry exists and proceed as if it were a new
org file. 

By deleting the persist index file, the calls to `org-babel-tangle-file'
succeed and I proceed without error. If an entry in the persist index
has the elided entries, then the tangling fails.

I'm not entirely sure that this is the recipe but I have seen some 
weirdness around the `org-persist' feature but fixes other than deleting
persistent storage seem to be inconsistent.

Please feel free to reach out if you'd like me to verify or try some other 
recipe.

In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.42, cairo version 1.18.0) of 2024-06-29 built on michael-laptop
Repository revision: fcd6403d8ebf7ce6e61ba17971d4fdecf06c
Repository branch: master
System Description: Fedora Linux 40 (Workstation Edition)
Package: Org mode version 9.7.5 (release_9.7.5-9-ga091ca @ 
/usr/local/share/emacs/31.0.50/lisp/org/)

Configured using:
 'configure --with-pgtk --with-native-compilation
 --with-file-notification=yes --with-wide-int --with-tree-sitter
 --with-mailutils 'CFLAGS= -O0 -g''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  server-mode: t
  org-popup-posframe-mode: t
  transient-posframe-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: t
  treemacs-fringe-indicator-mode: t
  global-org-modern-mode: t
  repeat-mode: t
  global-dash-fontify-mode: t
  global-diff-hl-mode: t
  global-flycheck-mode: t
  company-posframe-mode: t
  recentf-mode: t
  global-company-mode: t
  company-mode: t
  savehist-mode: t
  vertico-posframe-mode: t
  vertico-mode: t
  which-key-posframe-mode: t
  which-key-mode: t
  which-function-mode: t
  electric-pair-mode: t
  delete-selection-mode: t
  auto-insert-mode: t
  save-place-mode: t
  my-region-size-mode: t
  override-global-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-layout-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  tab-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  isearch-fold-quotes-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  column-number

Re: org-persist-write slowing down kill-buffer

[Ihor Radchenko]

Ihor Radchenko  writes:

> Karthik Chikmagalur  writes:
>
>> - My org-persist-dir is 627MB in size.
>
> What is your (length org-persist--index) ?
>
> Also, it does not look like your `org-element-ast-map' is compiled.

More information has been requested, but none provided within over a
month. Closing.

Canceled.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist-write slowing down kill-buffer

[Ihor Radchenko]

Karthik Chikmagalur  writes:

> I've been noticing that kill-buffer blocks Emacs for a noticeable
> period when killing org buffers.  Here are elp results obtained by
> instrumenting org-persist-* and kill-buffer:
>
> | Function name | Call count | Elapsed time | Average 
> time |
> |---++--+--|
> | kill-buffer   | 38 | 2.5647546329 | 
> 0.0674935429 |
> | org-persist-write-all-buffer  |  1 |  2.562779994 |  
> 2.562779994 |
> | org-persist-write-all |  1 |   2.56277329 |   
> 2.56277329 |
> | org-persist-write |      1 |  1.627834788 |  
> 1.627834788 |
> | org-persist--write-elisp-file |  1 |  0.312172392 |  
> 0.312172392 |

> It blocks Emacs for about 3 seconds each time.  Any Org file with > 500
> lines causes this behavior.
>
> I've also attached the sampling profiler output from killing an Org
> buffer.
>
> Some facts that might be relevant:
>
> - I'm using the WIP LaTeX preview system fork of Org, but there are no
>   LaTeX previews in the Org buffers that I run these tests on.  (There
>   aren't even any LaTeX fragments.)
>
> - My org-persist-dir is 627MB in size.

What is your (length org-persist--index) ?

Also, it does not look like your `org-element-ast-map' is compiled.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

org-persist-write slowing down kill-buffer

[Karthik Chikmagalur]

Hi,

I've been noticing that kill-buffer blocks Emacs for a noticeable
period when killing org buffers.  Here are elp results obtained by
instrumenting org-persist-* and kill-buffer:

| Function name | Call count | Elapsed time | Average time |
|---++--+--|
| kill-buffer   | 38 | 2.5647546329 | 0.0674935429 |
| org-persist-write-all-buffer  |  1 |  2.562779994 |  2.562779994 |
| org-persist-write-all |  1 |   2.56277329 |   2.56277329 |
| org-persist-write |  1 |  1.627834788 |  1.627834788 |
| org-persist--write-elisp-file |  1 |  0.312172392 |  0.312172392 |
| org-persist--find-index   |   4808 | 0.0480629129 | 9.996...e-06 |
| org-persist--normalize-associated |  4 |  0.007245059 | 0.0018112647 |
| org-persist--normalize-container  |  7 | 0.0001262719 | 1.803...e-05 |
| org-persist--get-collection   |  1 |  0.000113982 |  0.000113982 |
| org-persist-write:elisp   |  2 | 2.647...e-05 | 1.323...e-05 |
| org-persist--display-time |  1 |3.772e-06 |3.772e-06 |
| org-persist-write:version |  1 |1.467e-06 |1.467e-06 |

It blocks Emacs for about 3 seconds each time.  Any Org file with > 500
lines causes this behavior.

I've also attached the sampling profiler output from killing an Org
buffer.

Some facts that might be relevant:

- I'm using the WIP LaTeX preview system fork of Org, but there are no
  LaTeX previews in the Org buffers that I run these tests on.  (There
  aren't even any LaTeX fragments.)

- My org-persist-dir is 627MB in size.

This is Emacs 29.3 (pgtk branch), the Org commit that the LaTeX preview
patches are rebased on is 1ae978f940c0f88473f2232177c63a0de7fb7a1c.

Let me know if I can help with more information.

Karthik


kill-buffer-profiler-report.eld
Description: Binary data

Re: org-persist asking for temp-file encoding every time

[Brian Elmegaard]

On 06-04-2024 14:44, Ihor Radchenko wrote:

You need to redirect where Emacs looks for Org mode:
(add-to-list 'load-path "/path/to/org-mode/lisp")

Somewhere early in your init.el.


Thank you for the hint.
It seems to be working without the issue in the development version.

Brian

Re: org-persist asking for temp-file encoding every time

[Ihor Radchenko]

Brian Elmegaard  writes:

> On 20-03-2024 10:14, Ihor Radchenko wrote:
>> org-persist forces encoding in Org 9.7-pre (development version).
>> May you try it and let us know if your problem is still present there?
>>
> I have been looking into this, but it is not clear to me how to override 
> emacs' builtin org-mode, and run another version.
>
> I obtained the version from the repository as explained here: 
> https://orgmode.org/org.html#Installation

You need to redirect where Emacs looks for Org mode:

(add-to-list 'load-path "/path/to/org-mode/lisp")

Somewhere early in your init.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist asking for temp-file encoding every time

[Brian Elmegaard]

On 20-03-2024 10:14, Ihor Radchenko wrote:

org-persist forces encoding in Org 9.7-pre (development version).
May you try it and let us know if your problem is still present there?

I have been looking into this, but it is not clear to me how to override 
emacs' builtin org-mode, and run another version.


I obtained the version from the repository as explained here: 
https://orgmode.org/org.html#Installation


Brian

org-persist asking for temp-file encoding every time

[Brian Elmegaard]

Hi

I am experiencing a small issue with an org-file.
Every time I close it I am prompted with:
Select coding system (default utf-8):
when org-mode is saving a *Temp file*.

In the *Messages* buffer it says:
org-persist: Writing to 
"c:/Users/user/AppData/Roaming/.emacs.d/org-persist/18/673ac1-50b0-4f3c-9cab-58f496875f4f" 
took 56.90 sec.


Can this somehow be avoided?

Best regards,
Brian

Re: Error with org-persist

[Ihor Radchenko]

Al Oomens  writes:

> (file-writable-p "c:/Users/aloom/My Drive/home/.emacs.d/org-persist") - 
> returns "t"
> ...
>> Evaluating (file-writable-p "c:/Users/aloom/My Drive/home/.emacs.d/") 
>> returns "nil".

The problem is that you do not seem to have write access to create files
in "c:/Users/aloom/My Drive/home".

This is a bug in Org mode that it checks for write access there (we
check one parent directory too much).
I fixed the check on the bugfix branch.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=ad0282533
The fix will be available in the next Org bugfix release (likely tomorrow).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Error with org-persist

[Al Oomens]

(file-writable-p "c:/Users/aloom/My Drive/home/.emacs.d/org-persist") - returns 
"t"

From: Ihor Radchenko 
Sent: Wednesday, March 20, 2024 3:07 PM
To: Al Oomens 
Cc: emacs-orgmode@gnu.org 
Subject: Re: Error with org-persist

[ Adding Org mailing list back to CC. Please use "Reply All" to keep the
  conversation public ]

Al Oomens  writes:

> OK, very strange.
> .emacs.d/org-persist/ does not exist. It should be automatically created, 
> correct?

Yup.

> Evaluating (file-writable-p "c:/Users/aloom/My Drive/home/.emacs.d/") returns 
> "nil".
> However, I was able to visit a new file and save it to that directory:
> ...

What about (file-writable-p "c:/Users/aloom/My 
Drive/home/.emacs.d/org-persist") ?

--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Error with org-persist

[Ihor Radchenko]

[ Adding Org mailing list back to CC. Please use "Reply All" to keep the
  conversation public ]

Al Oomens  writes:

> OK, very strange.
> .emacs.d/org-persist/ does not exist. It should be automatically created, 
> correct?

Yup.

> Evaluating (file-writable-p "c:/Users/aloom/My Drive/home/.emacs.d/") returns 
> "nil".
> However, I was able to visit a new file and save it to that directory:
> ...

What about (file-writable-p "c:/Users/aloom/My 
Drive/home/.emacs.d/org-persist") ?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Error with org-persist

[Ihor Radchenko]

Al Oomens  writes:

> Greetings, I have never posted to this list before.

Welcome!

> ... recently started getting this
> error when I first open a .org file:
>
> 'Missing write access rights to org-persist-directory: "c:/Users/aloom/My 
> Drive/home/.emacs.d/org-persist/"'.
>
> I am using emacs v29.2 on Windows 11. Org-mode is 9.6.21
>
> This started happening fairly recently, perhaps when I upgraded Emacs? Also 
> note, that I have my $HOME in a Goggle Drive synced folder. Perhaps that has 
> something to do with this? Or perhaps the space in the PATH to my HOME 
> directory? Although, as I stated, I did not have this error until fairly 
> recently.

The message you are seeing is displayed when Emacs has no write access
to your .emacs.d.
Check your FS write access settings along the path you see in the
message.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Error with org-persist

[Al Oomens]

Greetings, I have never posted to this list before. I searched the email 
archive, and didn't see anything related. I have been using Emacs and org-more 
for several years and recently started getting this error when I first open a 
.org file:

'Missing write access rights to org-persist-directory: "c:/Users/aloom/My 
Drive/home/.emacs.d/org-persist/"'.

I am using emacs v29.2 on Windows 11. Org-mode is 9.6.21

This started happening fairly recently, perhaps when I upgraded Emacs? Also 
note, that I have my $HOME in a Goggle Drive synced folder. Perhaps that has 
something to do with this? Or perhaps the space in the PATH to my HOME 
directory? Although, as I stated, I did not have this error until fairly 
recently.

Let me know if you need any further information.
Thanks!

Re: org-persist asking for temp-file encoding every time

[Ihor Radchenko]

Brian Elmegaard  writes:

> I am experiencing a small issue with an org-file.
> Every time I close it I am prompted with:
> Select coding system (default utf-8):
> when org-mode is saving a *Temp file*.
>
> In the *Messages* buffer it says:
> org-persist: Writing to
> "c:/Users/user/AppData/Roaming/.emacs.d/org-persist/18/673ac1-50b0-4f3c-9cab-58f496875f4f"
> took 56.90 sec.
>
> Can this somehow be avoided?

May you share your Org version? (M-x org-version)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

org-persist asking for temp-file encoding every time

[Brian Elmegaard]

Hi

I am experiencing a small issue with an org-file.
Every time I close it I am prompted with:
Select coding system (default utf-8):
when org-mode is saving a *Temp file*.

In the *Messages* buffer it says:
org-persist: Writing to
"c:/Users/user/AppData/Roaming/.emacs.d/org-persist/18/673ac1-50b0-4f3c-9cab-58f496875f4f"
took 56.90 sec.

Can this somehow be avoided?

Best regards,
Brian

Re: Disable time messages from org-persist

[Colin Baxter]

>>>>> Ihor Radchenko  writes:

> Colin Baxter  writes:
>> Ho do I disable messages from org-persist telling me how long it
>> took to write to gc-lock.eld? For example,
>> 
>> --8<---cut here-----------start->8---
>> org-persist: Writing to "~/path/to/org-persist/gc-lock.eld" took
>> 3.08 sec --8<---cut
>> here---end--->8---
>> 
>> I find them a distraction and the information is of no use to me.

> You can set `org-persist--report-time' to nil.

Thank you.

Re: Disable time messages from org-persist

[Ihor Radchenko]

Colin Baxter  writes:

> Ho do I disable messages from org-persist telling me how long it took to
> write to gc-lock.eld? For example,
>
> --8<---cut here---start--------->8---
> org-persist: Writing to "~/path/to/org-persist/gc-lock.eld" took 3.08 sec
> --8<---cut here---end--->8---
>
> I find them a distraction and the information is of no use to me.

You can set `org-persist--report-time' to nil.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Disable time messages from org-persist

[Colin Baxter]

Ho do I disable messages from org-persist telling me how long it took to
write to gc-lock.eld? For example,

--8<---cut here---start->8---
org-persist: Writing to "~/path/to/org-persist/gc-lock.eld" took 3.08 sec
--8<---cut here---end--->8---

I find them a distraction and the information is of no use to me.

Colin Baxter.

Re: org-persist - not wanted.

[Ihor Radchenko]

Colin Baxter  writes:

> I find a file called "gc-lock.eld" has been deposited un-requested and
> un-wanted in ~/.emacs.d/org-persist.
>
> I had thought I had disabled org-persist with the following
>
> --8<---cut here---start----->8---
> (setq org-persist-disable-when-emacs-Q t)
> (defun me/advice--org-persist (old-fn &rest args)
>  (let (user-init-file)
>   (apply old-fn args)))
> (advice-add 'org-persist-write :around #'me/advice--org-persist)
> (advice-add 'org-persist-read :around #'me/advice--org-persist)
> (advice-add 'org-persist-gc :around #'me/advice--org-persist)
> --8<---cut here-------end--->8---
>
> This had worked fine until today. How do I now stop org-persist
> depositing this - and maybe other -files. I do not want them.

Please note that disabling org-persist is not officially supported and
can be broken any moment. Like it appeared to happen for you after
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5a5ec1b320512f5e97f72668abc13447567fdd35

In future, I may implement some options to, say, set org-persist save
the data alongside with .org file. For now, it is a centralized place.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist - not wanted.

[Colin Baxter]

>>>>> Colin Baxter  writes:

> I find a file called "gc-lock.eld" has been deposited un-requested
> and un-wanted in ~/.emacs.d/org-persist.

> I had thought I had disabled org-persist with the following

> (setq org-persist-disable-when-emacs-Q t) (defun
> me/advice--org-persist (old-fn &rest args) (let (user-init-file)
    > (apply old-fn args))) (advice-add 'org-persist-write :around
> #'me/advice--org-persist) (advice-add 'org-persist-read :around
> #'me/advice--org-persist) (advice-add 'org-persist-gc :around
> #'me/advice--org-persist)

> This had worked fine until today. How do I now stop org-persist
> depositing this - and maybe other -files. I do not want them.

> I am using emacs-30.0.50 and Org mode version 9.7-pre
> (release_9.6.13-998-g571186).

 Ignore my rant. I cannot reproduce this so it looks to be spurious. 

org-persist - not wanted.

[Colin Baxter]

I find a file called "gc-lock.eld" has been deposited un-requested and
un-wanted in ~/.emacs.d/org-persist.

I had thought I had disabled org-persist with the following

--8<---cut here---start->8---
(setq org-persist-disable-when-emacs-Q t)
(defun me/advice--org-persist (old-fn &rest args)
 (let (user-init-file)
  (apply old-fn args)))
(advice-add 'org-persist-write :around #'me/advice--org-persist)
(advice-add 'org-persist-read :around #'me/advice--org-persist)
(advice-add 'org-persist-gc :around #'me/advice--org-persist)
--8<---cut here---end--->8---

This had worked fine until today. How do I now stop org-persist
depositing this - and maybe other -files. I do not want them.

I am using emacs-30.0.50 and Org mode version 9.7-pre
(release_9.6.13-998-g571186).

Best wishes, Colin Baxter.

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-11, 11:06 +, Ihor Radchenko  wrote:
> Fixed, on bugfix.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=866e638c2

Super, thanks for helping with this, Ihor. 🙏

> You may consider customizing `remote-file-name-inhibit-locks'.

Ah! Good to know, thanks.

Cheers, F.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali  writes:

> Sorry for the delay. My results below, after testing a couple of
> different cases.
>
> If the remote file had been saved before losing connection, then the
> patch below will fix the problem. TRAMP will still hang for a while, as
> it tries to reach the server but the buffer will be closed immediately
> when using `tramp-cleanup-all-buffers' (or a variant thereof).
> ...
> (Apologies, the patch is badly formatted and not meant for the
> repository.) The gist is that I replaced an occurrence of `associated'
> with `file' in your latest patch, Ihor. It seems to work, what do you
> think?

I made the same change in my v2 patch.
Applied.
Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=866e638c2

> However, if the remote file hadn't been saved, then TRAMP will still
> hang indefinitely. Quitting will reveal the following backtrace:
>
> Debugger entered--Lisp error: (quit "")
> ...
>   tramp-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
>   kill-buffer("test.org")
>   funcall-interactively(kill-buffer "test.org")
>   command-execute(kill-buffer)
>
> This seems to be on the TRAMP side?

Yes.
You may consider customizing `remote-file-name-inhibit-locks'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 11:26 +, Fabio Natali  wrote:
> On 2023-11-10, 11:13 +, Fabio Natali  wrote:
>> Brilliant. I think the second branch should read `(not (file-remote-p
>> file))', i.e. `file' instead of `associated'? It seems to work with that
>> micro amendment.
>
> Hm, I stand corrected, I retested it end-to-end and no, it doesn't seem
> to work. I'll post more details later. Cheers, Fabio.

Hi Ihor,

Sorry for the delay. My results below, after testing a couple of
different cases.

If the remote file had been saved before losing connection, then the
patch below will fix the problem. TRAMP will still hang for a while, as
it tries to reach the server but the buffer will be closed immediately
when using `tramp-cleanup-all-buffers' (or a variant thereof).

--- /tmp/test-0.el  2023-11-11 09:27:33.477117451 +
+++ /tmp/test-1.el  2023-11-11 09:27:14.217129703 +
@@ -5,7 +5,8 @@
  (unless (stringp associated)
(setq associated (cadr associated)))
  (let* ((rtn `(:file ,associated))
-(inode (and (fboundp 'file-attribute-inode-number)
+(inode (and (not (file-remote-p associated))
+   (fboundp 'file-attribute-inode-number)
 (file-attribute-inode-number
  (file-attributes associated)
(when inode (plist-put rtn :inode inode))
@@ -25,6 +26,7 @@
  (or (buffer-base-buffer associated)
  associated)))
  (setq inode (when (and file
+   (not (file-remote-p file))
 (fboundp 'file-attribute-inode-number))
(file-attribute-inode-number
 (file-attributes file

(Apologies, the patch is badly formatted and not meant for the
repository.) The gist is that I replaced an occurrence of `associated'
with `file' in your latest patch, Ihor. It seems to work, what do you
think?

However, if the remote file hadn't been saved, then TRAMP will still
hang indefinitely. Quitting will reveal the following backtrace:

Debugger entered--Lisp error: (quit "")
  signal(quit (""))
  tramp-error(nil quit "")
  tramp-signal-hook-function(quit (""))
  signal(quit (""))
  tramp-maybe-open-connection((tramp-file-name "ssh" nil nil "192.168.1.100" 
nil "~/.#test.org" nil))
  tramp-send-command((tramp-file-name "ssh" nil nil "192.168.1.100" nil 
"~/.#test.org" nil) "echo ~ 2>/dev/null; echo tramp_exit_status $?")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil "192.168.1.100" 
nil "~/.#test.org" nil) "echo ~")
  tramp-sh-handle-get-home-directory((tramp-file-name "ssh" nil nil 
"192.168.1.100" nil "~/.#test.org" nil) "")
  tramp-sh-file-name-handler(tramp-get-home-directory (tramp-file-name "ssh" 
nil nil "192.168.1.100" nil "~/.#test.org" nil) "")
  apply(tramp-sh-file-name-handler tramp-get-home-directory ((tramp-file-name 
"ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) ""))
  tramp-file-name-handler(tramp-get-home-directory (tramp-file-name "ssh" nil 
nil "192.168.1.100" nil "~/.#test.org" nil) "")
  tramp-get-home-directory((tramp-file-name "ssh" nil nil "192.168.1.100" nil 
"~/.#test.org" nil) "")
  tramp-sh-handle-expand-file-name("/ssh:192.168.1.100:~/.#test.org" nil)
  tramp-sh-file-name-handler(expand-file-name "/ssh:192.168.1.100:~/.#test.org" 
nil)
  apply(tramp-sh-file-name-handler expand-file-name 
("/ssh:192.168.1.100:~/.#test.org" nil))
  tramp-file-name-handler(expand-file-name "/ssh:192.168.1.100:~/.#test.org" 
nil)
  files--transform-file-name("/ssh:192.168.1.100:~/test.org" nil ".#" "")
  make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-run-real-handler(make-lock-file-name ("/ssh:192.168.1.100:~/test.org"))
  tramp-handle-make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-sh-file-name-handler(make-lock-file-name 
"/ssh:192.168.1.100:~/test.org")
  apply(tramp-sh-file-name-handler make-lock-file-name 
"/ssh:192.168.1.100:~/test.org")
  tramp-file-name-handler(make-lock-file-name "/ssh:192.168.1.100:~/test.org")
  tramp-compat-make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-handle-unlock-file("/ssh:192.168.1.100:~/test.org")
  tramp-sh-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
  apply(tramp-sh-file-name-handler unlock-file "/ssh:192.168.1.100:~/test.org")
  tramp-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
  kill-buffer("test.org")
  funcall-interactively(kill-buffer "test.org")
  command-execute(kill-buffer)

This seems to be on the TRAMP side?

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 11:13 +, Fabio Natali  wrote:
> Brilliant. I think the second branch should read `(not (file-remote-p
> file))', i.e. `file' instead of `associated'? It seems to work with that
> micro amendment.

Hm, I stand corrected, I retested it end-to-end and no, it doesn't seem
to work. I'll post more details later. Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 09:26 +, Ihor Radchenko  wrote:
> What about the attached patch?

Hi Ihor,

Brilliant. I think the second branch should read `(not (file-remote-p
file))', i.e. `file' instead of `associated'? It seems to work with that
micro amendment.

Anything else I should be testing or does this look reasonable enough to
be pushed to the repo at some point?

Thank you very much for this!

🙏🙏🙏

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali  writes:

> On 2023-11-09, 12:17 +, Ihor Radchenko  wrote:
>> May I know what
>>(file-remote-p "/ssh::/home/user/test.org")
>> (with appropriate ) returns on your side?
>
> Hi Ihor,
>
> Sure, no particular reason to obfuscate the remote address, it's
> actually a LAN IP. The above function returns "/ssh:192.168.1.100:". Is
> that the kind of output one should be expecting? Anything wrong about
> it?

Nothing wrong. Just me missing one code branch.
What about the attached patch?

>From 73e2fc687b966f3c12b87dd40c90cdc6b9f440fc Mon Sep 17 00:00:00 2001
Message-ID: <73e2fc687b966f3c12b87dd40c90cdc6b9f440fc.1699608346.git.yanta...@posteo.net>
From: Ihor Radchenko 
Date: Wed, 8 Nov 2023 19:58:42 +0200
Subject: [PATCH v2] org-persist--normalize-associated: Avoid TRAMP connection
 for remote files

* lisp/org-persist.el (org-persist--normalize-associated): Never try
to store inode association for remote TRAMP files.

Reported-by: Fabio Natali 
Link: https://orgmode.org/list/87jzqthdge@fabionatali.com
---
 lisp/org-persist.el | 15 ---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 81edcf6f2..bdc6006b6 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -614,9 +614,14 @@ (defun org-persist--normalize-associated (associated)
  (unless (stringp associated)
(setq associated (cadr associated)))
  (let* ((rtn `(:file ,associated))
-(inode (and (fboundp 'file-attribute-inode-number)
-(file-attribute-inode-number
- (file-attributes associated)
+(inode (and
+;; Do not store :inode for remote files - it may
+;; be time-consuming on slow connections or even
+;; fail completely when ssh connection is closed.
+(not (file-remote-p associated))
+(fboundp 'file-attribute-inode-number)
+(file-attribute-inode-number
+ (file-attributes associated)
(when inode (plist-put rtn :inode inode))
    rtn))
 ((or (pred bufferp) `(:buffer ,_))
@@ -634,6 +639,10 @@ (defun org-persist--normalize-associated (associated)
  (or (buffer-base-buffer associated)
  associated)))
  (setq inode (when (and file
+;; Do not store :inode for remote files - it may
+;; be time-consuming on slow connections or even
+;; fail completely when ssh connection is closed.
+(not (file-remote-p associated))
 (fboundp 'file-attribute-inode-number))
(file-attribute-inode-number
 (file-attributes file
-- 
2.42.0


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-09, 12:17 +, Ihor Radchenko  wrote:
> May I know what
>(file-remote-p "/ssh::/home/user/test.org")
> (with appropriate ) returns on your side?

Hi Ihor,

Sure, no particular reason to obfuscate the remote address, it's
actually a LAN IP. The above function returns "/ssh:192.168.1.100:". Is
that the kind of output one should be expecting? Anything wrong about
it?

Thanks, cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali  writes:

> On 2023-11-08, 18:01 +, Ihor Radchenko  wrote:
>> I see the problem now.
>> Does the attached patch solve the "freeze"?
>
> Hey, thanks for sending this, really appreciate it.
>
> Unfortunately that didn't seem to fix it though. Here's what I did.
>
> - Added `(not (file-remote-p associated))' to the right place
> - Revaluated the function `org-persist--normalize-associated'
> - Tried to close the stale buffer (without repeating the entire
>   workflow, just trying with the buffer from the previous test)
>
> Result: the problem is still there, same backtrace.

Then I need to ask about
tramp-file-name-handler(file-attributes 
"/ssh::/home/user/test.org")

"/ssh::/home/user/test.org" is obfuscated to mask your
personal data, but it apparently fails to be filtered by `file-remote-p'
check. May I know what
   (file-remote-p "/ssh::/home/user/test.org")
(with appropriate ) returns on your side?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-08, 18:01 +, Ihor Radchenko  wrote:
> I see the problem now.
> Does the attached patch solve the "freeze"?

Hey, thanks for sending this, really appreciate it.

Unfortunately that didn't seem to fix it though. Here's what I did.

- Added `(not (file-remote-p associated))' to the right place
- Revaluated the function `org-persist--normalize-associated'
- Tried to close the stale buffer (without repeating the entire
  workflow, just trying with the buffer from the previous test)

Result: the problem is still there, same backtrace.

If anything else comes to mind, happy to test it.

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali  writes:

> Thanks for your email. Sure, glad to share the backtrace with you.
> ...
>   tramp-file-name-handler(file-attributes 
> "/ssh::/home/user/test.org")
>   org-persist--normalize-associated(#)

I see the problem now.
Does the attached patch solve the "freeze"?

>From ac571f9654ef5de8cef7157e216beeb0b91f6125 Mon Sep 17 00:00:00 2001
Message-ID: 
From: Ihor Radchenko 
Date: Wed, 8 Nov 2023 19:58:42 +0200
Subject: [PATCH] org-persist--normalize-associated: Avoid TRAMP connection for
 remote files

* lisp/org-persist.el (org-persist--normalize-associated): Never try
to store inode association for remote TRAMP files.

Reported-by: Fabio Natali 
Link: https://orgmode.org/list/87jzqthdge@fabionatali.com
---
 lisp/org-persist.el | 11 ---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 01078f459..f97e1d7a4 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -481,9 +481,14 @@ (defun org-persist--normalize-associated (associated)
  (unless (stringp associated)
(setq associated (cadr associated)))
  (let* ((rtn `(:file ,associated))
-(inode (and (fboundp 'file-attribute-inode-number)
-(file-attribute-inode-number
- (file-attributes associated)
+(inode (and
+;; Do not store :inode for remote files - it may
+;; be time-consuming on slow connections or even
+;; fail completely when ssh connection is closed.
+(not (file-remote-p associated))
+(fboundp 'file-attribute-inode-number)
+(file-attribute-inode-number
+ (file-attributes associated)
(when inode (plist-put rtn :inode inode))
rtn))
 ((or (pred bufferp) `(:buffer ,_))
-- 
2.42.0


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-08, 09:25 +, Ihor Radchenko  wrote:
> May you please share the backtrace?
> In particular, may you (1) M-x toggle-debug-on-quit (2) try to close the
> problematic file; observe Emacs "freeze" (3) C-g and post the obtained
> backtrace.

Hi Ihor,

Thanks for your email. Sure, glad to share the backtrace with you.

This is how the issue can be reproduced on my system.

- Enable debug, `toggle-debug-on-quit'
- Open a remote file, `(find-file "/ssh::~/test.org")'
- Bring the remote machine offline
- Try to close the remote file, `kill-buffer'
- The system hangs for a while as it tries to reach the remote machine
- Quitting (`keyboard-quit') produces the following backtrace

#+begin_export ascii
Debugger entered--Lisp error: (quit "")
  signal(quit (""))
  tramp-error(nil quit "")
  tramp-signal-hook-function(quit (""))
  signal(quit (""))
  tramp-maybe-open-connection((tramp-file-name "ssh" nil nil "" 
nil "/home/user/test.org" nil))
  tramp-send-command((tramp-file-name "ssh" nil nil "" nil 
"/home/user/test.org" nil) "echo \\\"`getconf PATH 2>/dev/null`\\\" 
2>/dev/null; e...")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil 
"" nil "/home/user/test.org" nil) "echo \\\"`getconf PATH 
2>/dev/null`\\\"")
  tramp-send-command-and-read((tramp-file-name "ssh" nil nil "" 
nil "/home/user/test.org" nil) "echo \\\"`getconf PATH 2>/dev/null`\\\"" 
noerror)
  tramp-get-remote-path((tramp-file-name "ssh" nil nil "" nil 
"/home/user/test.org" nil))
  tramp-get-remote-stat((tramp-file-name "ssh" nil nil "" nil 
"/home/user/test.org" nil))
  tramp-sh-handle-file-attributes("/ssh::/home/user/test.org")
  tramp-sh-file-name-handler(file-attributes 
"/ssh::/home/user/test.org")
  apply(tramp-sh-file-name-handler file-attributes 
"/ssh::/home/user/test.org")
  tramp-file-name-handler(file-attributes 
"/ssh::/home/user/test.org")
  org-persist--normalize-associated(#)
  org-persist-write-all(#)
  org-persist-write-all-buffer()
  kill-buffer("test.org")
  funcall-interactively(kill-buffer "test.org")
  command-execute(kill-buffer)
#+end_export

Glad to share other info if helpful.

Thanks, cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali  writes:

> I seem to be having an issue with org-persist and Tramp which is close
> to what discussed in this thread:
>
> https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html

After that thread, Org should not (by default) examine remote files.

> - I open a Org file on a remote machine, via Tramp.
> - The SSH connection dies.
> - I try to kill the stale buffer with kill-buffer and from IBuffer.
> - The buffer is not killed, instead Emacs hangs for a while while still
>   trying to connect to the old SSH connection.
> - The buffer becomes sentient and gains immortality... either that or I
>   restart Emacs, which is clearly a big no-no. :D
>
> Looking at the backtrace, it looks like the kill operation insists on
> calling org-persist-write-all-buffer, which in turn seems to be calling
> Tramp and therefore SSH to the now unreachable machine.

May you please share the backtrace?
In particular, may you (1) M-x toggle-debug-on-quit (2) try to close the
problematic file; observe Emacs "freeze" (3) C-g and post the obtained
backtrace.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-07, 22:08 +0100, Antonio Carlos Padoan Junior 
 wrote:
> My workaround is to save the buffer locally (perhaps in /tmp). This
> action releases the buffer.

Hi Antonio,

Thanks for getting back to me.

Unfortunately your work-around doesn't seem to work in my case. Just to
make sure I'm not missing anything, when you say you save the buffer
locally, you mean with `write-file' (C-x C-w), right?

If I use `write-file', the system lags for a while, then it prompts me
to insert a file name. Whatever path I insert, Emacs will try to use the
old SSH connection and therefore freeze. If I `keyboard-quit' (C-g), the
system "unfreezes" but the file won't be saved.

Thanks, best wishes, Fabio.

Re: Issue with org-persist and Tramp

[Antonio Carlos Padoan Junior]

Hello Fabio,

Fabio Natali  writes:

> Hi,
>
> I seem to be having an issue with org-persist and Tramp which is close
> to what discussed in this thread:
>
> https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html
>
> - I open a Org file on a remote machine, via Tramp.
> - The SSH connection dies.
> - I try to kill the stale buffer with kill-buffer and from IBuffer.
> - The buffer is not killed, instead Emacs hangs for a while while still
>   trying to connect to the old SSH connection.
> - The buffer becomes sentient and gains immortality... either that or I

I was intrigued by the same issue. My workaround is to save the buffer
locally (perhaps in /tmp). This action releases the buffer. I think this
behavior is probably intended to avoid loosing "local" work after the ssh
connection dies, but who knows...  

Anyway, if there is any other workaround I would be interested as well.

BR,
-- 
Antonio Carlos PADOAN JUNIOR
GPG fingerprint:
243F 237F 2DD3 4DCA 4EA3  1341 2481 90F9 B421 A6C9

Issue with org-persist and Tramp

[Fabio Natali]

Hi,

I seem to be having an issue with org-persist and Tramp which is close
to what discussed in this thread:

https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html

- I open a Org file on a remote machine, via Tramp.
- The SSH connection dies.
- I try to kill the stale buffer with kill-buffer and from IBuffer.
- The buffer is not killed, instead Emacs hangs for a while while still
  trying to connect to the old SSH connection.
- The buffer becomes sentient and gains immortality... either that or I
  restart Emacs, which is clearly a big no-no. :D

Looking at the backtrace, it looks like the kill operation insists on
calling org-persist-write-all-buffer, which in turn seems to be calling
Tramp and therefore SSH to the now unreachable machine.

More context:

- The SSH process is now gone, no trace of its original process.
- I've tried all combinations of tramp-cleanup-* commands, with no luck.
- I'm on Emacs 29.1 and Org 9.6.9.
- The default value of org-persist-remote-files seems to be 100.

Is there any org-persist-related setting that I should include in my
init.el? Anything that I might be missing here?

Thanks for your help, cheers, Fabio.

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Wed, 3 May 2023 at 07:55, Ihor Radchenko  wrote:

> This was actually just an omission in `org-persist-gc'.
> Now fixed on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e11073d17

Thank you!

> But do not create so much sense of urgency here. As I explained, this is
> not something that much critical to fix ASAP.

Fair, I won't anymore.

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros  writes:

> For the record, even ".org.gpg" files generate an entry in the cache
> index. (True, not the `:persist-file' itself though).
>
> My ~/.cache/org-persist/index contains:
>
> (:container
>  ((elisp org-element--headline-cache)
>   (elisp org-element--cache))
>  :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
>  (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
> "/path/to/myfile.org.gpg" :inode 41551881)
>  :expiry 30)

This was actually just an omission in `org-persist-gc'.
Now fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e11073d17

> Please, please, be reasonable about this. Please, do not store
> information about known encrypted files in other places. Please, allow
> users to disable the feature cleanly and safely for arbitrary files if
> they choose to.

But do not create so much sense of urgency here. As I explained, this is
not something that much critical to fix ASAP.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 2 May 2023 at 17:54, Ihor Radchenko  wrote:

> I think `recentf-save-file' for example is no different. And
> org-id-locations-file. And custom-file, if you happen to save safe
> buffer-local variables by answering "!" in Emacs prompt. And many many
> other places.

Those two are easy to avoid, if one wishes. But granted, there are
plenty of places, some keeping file names, some worse. Still, this is
arguably something to avoid.

> I do not think that file name, even from encrypted volume, is something
> we need to worry about.

Granted too. What I'm worried about is being able to disable the
feature. Besides, why store it in the index if the persist file does
not exist?

> I even suspect that, for example, browser cache often contains all kinds
> of secrets, like files associated with web pages were you logged in. And
> they can be read by anyone familiar with the layout! (like
> https://www.nirsoft.net/utils/chrome_cache_view.html)

I really hope that's not Org's benchmark.  ;-)

> That said, do not worry about this issue being forgotten. But it is not
> easy to design cleanly. I am thinking about it.
> Of course, if you have good ideas or patches, they are welcome.

Thank you!

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros  writes:

> For the record, even ".org.gpg" files generate an entry in the cache
> index. (True, not the `:persist-file' itself though).
>
> My ~/.cache/org-persist/index contains:
>
> (:container
>  ((elisp org-element--headline-cache)
>   (elisp org-element--cache))
>  :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
>  (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
> "/path/to/myfile.org.gpg" :inode 41551881)
>  :expiry 30)

I think `recentf-save-file' for example is no different. And
org-id-locations-file. And custom-file, if you happen to save safe
buffer-local variables by answering "!" in Emacs prompt. And many many
other places.

I do not think that file name, even from encrypted volume, is something
we need to worry about.

I even suspect that, for example, browser cache often contains all kinds
of secrets, like files associated with web pages were you logged in. And
they can be read by anyone familiar with the layout! (like
https://www.nirsoft.net/utils/chrome_cache_view.html)

> Please, please, be reasonable about this. Please, do not store
> information about known encrypted files in other places. Please, allow
> users to disable the feature cleanly and safely for arbitrary files if
> they choose to.

That said, do not worry about this issue being forgotten. But it is not
easy to design cleanly. I am thinking about it.
Of course, if you have good ideas or patches, they are welcome.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 25 Apr 2023 at 07:52, Gustavo Barros  wrote:

> My view is that there really should be a way of doing it. Because this
> has relevant security implications. And because users should be able
> to have control of their data in the first place. And then if one such
> a feature, necessarily requiring persistence (as opposed to the
> convenience of "just" speeding things up), a user-error could be
> signaled saying "sorry, can't do that because you disabled
> persistence". And the two cases you mentioned seem quite the exception
> to me. Of course, I'm speaking here about writing information about /
> of the org file itself to an external file, naturally internet
> downloads must go somewhere and that should be easy. The thing is Org
> is used for a lot of things. You can rest assured I have no intention
> of exporting my encrypted files alongside some remote content. But I
> do wish to be able to make sure my passwords are safe.

For the record, even ".org.gpg" files generate an entry in the cache
index. (True, not the `:persist-file' itself though).

My ~/.cache/org-persist/index contains:

(:container
 ((elisp org-element--headline-cache)
  (elisp org-element--cache))
 :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
 (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
"/path/to/myfile.org.gpg" :inode 41551881)
 :expiry 30)

Please, please, be reasonable about this. Please, do not store
information about known encrypted files in other places. Please, allow
users to disable the feature cleanly and safely for arbitrary files if
they choose to.

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 25 Apr 2023 at 07:21, Ihor Radchenko  wrote:

> > I don't understand that. Isn't persistence about keeping data across
> > Emacs sessions?
>
> Not only. We also use org-persist to cache downloaded images from
> internet and, in future, to cache image previews (currently, they just
> sit in `org-preview-latex-image-directory'). These scenarios make use of
> org-persist during a single Emacs session, not just across several
> sessions.

I see. But, for uses where the data is only required during a session,
does the cache have to be written to an external file? But I think I
get why write it for the previews, and obviously, for downloads too.
I'm not sure this should be called "persistence" though.

> Even if we allow completely disabling persistence for certain files,
> previews, and internet downloads should happen _somewhere_ in file system
> and might inevitably cause leakage. But what is the way around?

My view is that there really should be a way of doing it. Because this
has relevant security implications. And because users should be able
to have control of their data in the first place. And then if one such
a feature, necessarily requiring persistence (as opposed to the
convenience of "just" speeding things up), a user-error could be
signaled saying "sorry, can't do that because you disabled
persistence". And the two cases you mentioned seem quite the exception
to me. Of course, I'm speaking here about writing information about /
of the org file itself to an external file, naturally internet
downloads must go somewhere and that should be easy. The thing is Org
is used for a lot of things. You can rest assured I have no intention
of exporting my encrypted files alongside some remote content. But I
do wish to be able to make sure my passwords are safe.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros  writes:

>> We cannot just disable persistence completely.
>> For example, remote image export relies on persistence to be working
>> _during_ Emacs session.
>
> I don't understand that. Isn't persistence about keeping data across
> Emacs sessions?

Not only. We also use org-persist to cache downloaded images from
internet and, in future, to cache image previews (currently, they just
sit in `org-preview-latex-image-directory'). These scenarios make use of
org-persist during a single Emacs session, not just across several
sessions.

Even if we allow completely disabling persistence for certain files,
previews, and internet downloads should happen _somewhere_ in file system
and might inevitably cause leakage. But what is the way around?

CCing Timothy as he might have further thoughts.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Sun, 23 Apr 2023 at 11:19, Ruijie Yu  wrote:

> Side track: remember that you have `always' and `ignore' in your
> collection of functions. :)

Somehow I wasn't aware of `always'. Thanks!

Gustavo.

Re: How to disable org-persist in a given file?

[General discussions about Org-mode.]

Gustavo Barros  writes:
>> You can use `org-persist-before-write-hook' to disable writing
>> selectively.
>
> Thanks! That's the one. Though it would be nice if a variable existed
> for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
> (&rest _args) t) nil t)' is not our average file local variable. :)

(lambda (&rest _args) t)

Side track: remember that you have `always' and `ignore' in your
collection of functions. :)

-- 
Best,


RY

[Please note that this mail might go to spam due to some
misconfiguration in my mail server -- still investigating.]

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Sun, 23 Apr 2023 at 10:51, Ihor Radchenko  wrote:

> We can try (string-match mounted-file-systems default-directory).
> Will it work with your setup?

Wouldn't that exclude a lot of legitimate use cases?
Personally, I don't see an issue in this scenario of mine for Org to
handle. It's an exception I must handle.

> We cannot just disable persistence completely.
> For example, remote image export relies on persistence to be working
> _during_ Emacs session.
>
> What about file/directory-local variable that will redirect where to
> save cache?

I don't understand that. Isn't persistence about keeping data across
Emacs sessions?
Redirecting the cache is useful too. But, imho, there should be a way
to disable persistence altogether for sensitive data (or, at least,
any storage of data in other files, since I don't get what you mean by
persistence during a session). If `org-persist-before-write-hook' does
this, as I understand it does, adding a function there which just
returns the value of a variable defaulting to nil and safe-local if
booleanp, would be enough, wouldn't it? Or what am I missing?

> I think we can add a section near "Code Evaluation and Security Issues".

That would be appreciated, thank you.

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros  writes:

> On Sun, 23 Apr 2023 at 07:55, Ihor Radchenko  wrote:
>
>> Thanks for letting us know about this scenario!
>
> Yes, but there's little Emacs/Org can do there, I think. Once the
> volume is mounted, there's no way to tell it is meant to be treated as
> an encrypted file.

We can try (string-match mounted-file-systems default-directory).
Will it work with your setup?

>> You can use `org-persist-before-write-hook' to disable writing
>> selectively.
>
> Thanks! That's the one. Though it would be nice if a variable existed
> for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
> (&rest _args) t) nil t)' is not our average file local variable. :)

We cannot just disable persistence completely.
For example, remote image export relies on persistence to be working
_during_ Emacs session.

What about file/directory-local variable that will redirect where to
save cache? 

> ... I think this would deserve an
> entry in the manual (as far as I can tell, currently there isn't one,
> but I'm running built-in on 29, so I might be out of date), letting
> people know it is enabled by default, and how to opt out, if they want
> to. As my scenario above shows, there's little hope of being able to
> cover "all cases", and people must take care of that for themselves
> within reason. All Org can do is let people know, and on security
> related issues, better be outspoken than shy.

I think we can add a section near "Code Evaluation and Security Issues".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

Hi Ihor,

On Sun, 23 Apr 2023 at 07:55, Ihor Radchenko  wrote:

> Thanks for letting us know about this scenario!

Yes, but there's little Emacs/Org can do there, I think. Once the
volume is mounted, there's no way to tell it is meant to be treated as
an encrypted file.

With time I'm learning that non-standard setups, while they may seem
attractive at first, come and bite you sooner or later. Not
necessarily because they are technically flawed, but because it breaks
people's expectations.

I do have some unusual partitioning scheme, and sometimes it becomes
an uphill battle to avoid leakages. On Emacs alone, I have to take
special care of backups, bookmarks (that's now fixed on 29), trash,
and now cache.

> You can use `org-persist-before-write-hook' to disable writing
> selectively.

Thanks! That's the one. Though it would be nice if a variable existed
for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
(&rest _args) t) nil t)' is not our average file local variable. :)

> You can refer to the comment in org-persist.el for explanation about the
> core concepts about CONTAINER and ASSOCIATED terms.
>
> Let us know if you have difficulties understanding the commentary or if
> you think that things can be improved.

I took a closer look at it now, and I think it is clear. But I do have
a suggestion. I've seen the use of `org-persist' by `org-element' by
default announced in the news file. But I think this would deserve an
entry in the manual (as far as I can tell, currently there isn't one,
but I'm running built-in on 29, so I might be out of date), letting
people know it is enabled by default, and how to opt out, if they want
to. As my scenario above shows, there's little hope of being able to
cover "all cases", and people must take care of that for themselves
within reason. All Org can do is let people know, and on security
related issues, better be outspoken than shy.

And thank you very much for this nice feature!

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros  writes:

> However, I do have some Org files which contain sensitive data. I'd
> like to be able to disable persistence in those files. Now, some of
> them are ".org.gpg" files, and I've seen the sources of `org-persist'
> and noticed persistence is inhibited for them. But I also do have some
> encrypted loop devices which, once opened, have a "plain" .org file
> there.

Thanks for letting us know about this scenario!

> However, taking a look at `org-persist's API, I couldn't find any
> obvious handle to disable persistence there. Everything I've seen
> presumes I have knowledge of registered "containers", which I don't.
>
> Is currently `org-element' the only part of Org using persistence by
> default? Is there some way to inhibit `org-persist' altogether in a
> given file? (As opposed to just inhibiting `org-element' use of it
> with `org-element-cache-persistent').

You can use `org-persist-before-write-hook' to disable writing
selectively.

You can refer to the comment in org-persist.el for explanation about the
core concepts about CONTAINER and ASSOCIATED terms.

Let us know if you have difficulties understanding the commentary or if
you think that things can be improved.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

How to disable org-persist in a given file?

[Gustavo Barros]

Hi All,

this is more of a question really. I moved recently to the new pretest
version of Emacs and thus got the 9.6 version of Org. And after that I
started to notice some saving by `org-persist' going on, and indeed I
have a new associated "~/.cache/org-persist/" directory.

However, I do have some Org files which contain sensitive data. I'd
like to be able to disable persistence in those files. Now, some of
them are ".org.gpg" files, and I've seen the sources of `org-persist'
and noticed persistence is inhibited for them. But I also do have some
encrypted loop devices which, once opened, have a "plain" .org file
there.

I've greped the sources, and I see that `org-element' is making use of
`org-persist'. I've also seen the release notes, and that
`org-element' makes available `org-element-cache-persistent'.

However, taking a look at `org-persist's API, I couldn't find any
obvious handle to disable persistence there. Everything I've seen
presumes I have knowledge of registered "containers", which I don't.

Is currently `org-element' the only part of Org using persistence by
default? Is there some way to inhibit `org-persist' altogether in a
given file? (As opposed to just inhibiting `org-element' use of it
with `org-element-cache-persistent').

Best regards,
Gustavo.

Re: Broken org-persist-storage probably leads to ‘org-open-at-point’ ask for TAGS file

[Ihor Radchenko]

Göktuğ Kayaalp  writes:

> I have discovered a weird behaviour that I have tracked back to
> org-persist and org-element-cache, where after some time, when I try to
> follow a document internal link in org mode, whether to a headline or to
> a =<>=, it would ask me to load a TAGS file, I think somewhat
> like how xref might ask you to do when you call say
> ‘xref-find-definitions’.
> ...

It appears to me that you had an issue with cache at some point, and then
it got propagated to disk, and triggered later on unrelated command.

May you set `org-element--cache-self-verify-frequency' to a larger value
and later let us know if you start seeing warnings from
org-element-cache?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Broken org-persist-storage probably leads to ‘org-open-at-point’ ask for TAGS file

[Göktuğ Kayaalp]

Hello,

I have discovered a weird behaviour that I have tracked back to
org-persist and org-element-cache, where after some time, when I try to
follow a document internal link in org mode, whether to a headline or to
a =<>=, it would ask me to load a TAGS file, I think somewhat
like how xref might ask you to do when you call say
‘xref-find-definitions’.

This has been going on for a few weeks I think and I finally had a
chance to debug this, and it appears that some problem in the persistent
cache might be triggering this bug. I haven’t been able to fully debug
the problem because by the time I had discovered it had anything to do
with org-persist and where the storage files for the latter were, I had
reset the database. I do not have a backup of ‘org-persist-directory’ to
compare thus, or, for now, to reproduce the bug.

It appears that one thing that can trigger the error is Org mode parser
errors. For example, while trying to poke around ‘org-open-at-point’ a
bit, I noticed that eval’ing the call to ‘org-element-lineage’ in its
definition

(let* ((context
;; Only consider supported types, even if they are not the
;; closest one.
(org-element-lineage
 (org-element-context)
 '(citation citation-reference clock comment comment-block
footnote-definition footnote-reference headline
inline-src-block inlinetask keyword link node-property
planning src-block timestamp)
 t))

resulted in the following warning, which then implored me to send a bug
report:

⛔ Warning (org-element-cache): org-element--cache: Org parser error in 
org.el.gz::333063. Resetting.
 The error was: (error "rx ‘**’ range error")
 Backtrace:
"  (backtrace-to-string nil)
  (org-element-at-point)
  (org-element-context)
  (org-element-lineage (org-element-context) '(citation citation-reference 
clock comment comment-block footnote-definition footnote-reference headline 
inline-src-block inlinetask keyword link node-property planning src-block 
timestamp) t)
  (progn (org-element-lineage (org-element-context) '(citation 
citation-reference clock comment comment-block footnote-definition 
footnote-reference headline inline-src-block inlinetask keyword link 
node-property planning src-block timestamp) t))
  (eval (progn (org-element-lineage (org-element-context) '(citation 
citation-reference clock comment comment-block footnote-definition 
footnote-reference headline inline-src-block inlinetask keyword link 
node-property planning src-block timestamp) t)) t)
  (elisp--eval-last-sexp nil)

after which the bug was triggered and I had become unable to use file
internal links in org mode.

In another instance an args out of range error in a ‘progn’ was
encountered while trying to store a link in a test buffer which only had
"\n\n\n<>" in it, which again triggered the bug.

I then tried to disable the cache which didn’t help, and to reset the
cache and then some other things, listed below, which didn’t either. But
after restarting Emacs (with the org element cache enabled), I was no
longer able to reproduce the bug.

  (setf org-element-use-cache t)
  M-x org-element-cache-reset
  (org-element-cache-reset t t)
  (org-element-cache-refresh (point))
  (setq-local org-element--cache nil)
  (setq-local org-element--headline-cache nil)
  (setq-local org-element--cache-hash-left nil
  org-element--cache-hash-right nil)

It is after this point that I debugged further and discovered
org-persist and its role. Sadly the default location of
‘org-persist-directory’ is not a location I include in my backups, so I
cannot reproduce the bug unless I accidentally recreate whatever
triggered it.

All this debugging occurred in the org mode package as it comes with
upstream emacs, built by me from commit
6c4abbab7999f55792a323e4bb1eb55ef5a7b990. In case it is helpful, below
are links to my org mode config and Emacs build script

https://codeberg.org/cadadr/personal-computing/src/commit/748803bb63d756e0a2ec75b00c34dfbd1f40cf0f/emacs.d/gk/gk-org.el
https://codeberg.org/cadadr/personal-computing/src/commit/748803bb63d756e0a2ec75b00c34dfbd1f40cf0f/emacs.d/bin/build-emacs.sh

‘org-version’ reports

Org mode version 9.6.1 (release_9.6.1-34-geea8da @ 
/home/cadadr/local/emacs/share/emacs/30.0.50/lisp/org/)

The org-persist storage verion currently seems to be 3.1 (exceprt from
=/home/cadadr/.cache/org-persist/index= below). I don’t know what it was
beforehand, but I usually build Emacs from git tip every Sunday, so that
might give a tip

 (:container
  ((index "3.1"))

Sadly I cannot do any further debugging, as I can’t reproduce the bug
anymore.

- Göktuğ.

P.S. Please keep me in CC if you want me to see your replies, I am not
subscribed to the list.

Re: org-persist-write slow when pp-use-max-width is t

[Ihor Radchenko]

Michael Eliachevitch  writes:

> I submitted a bug report where I attached the index file and a minimal 
> example file which benchmarks running pp on it, with which I could reproduce 
> my issue from emacs -Q:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58687
>
> Let's see what the emacs devs have to say about it.

Since the slowdown has been acknowledged as a known problem to be fixed
in future, I have pushed a workaround for now.

Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=ecdb44204

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: How to disable completely org-persist

[Jean Louis]

* to...@tuxteam.de  [2022-12-31 10:12]:
> Perhaps it would make sense to state the goals in more
> detail. I can see many "aspects":
> 
>  - "I don't want my ~/.emacs.d/ littered beyond my explicit
>control"

Isn't that happening anyway by various packages and functions? I see
there "auto-install", "auto-save-list" directories, I did not create
them myself. I see "desktop", though I used "desktop" package nobody
warned me it will be there. There is "erc-log" which I did not know it
will be there, there is "eshell", "games", "icons", "image-dired",
"multisession", "newsticker", "todo" and so on, too many.

I never had explicit control over creation of those files.

Using a package alone is not what I consider under "explicit", as I
was not asked directly about any of those directories.

>  - "I don't have (nor want to have) XDG directories"

Me too, I do not want to have, and I do not have, but some XDG
environment variables are there.

>  - "I don't want things in /tmp" [1]

Me too, but I keep it in my private $TMPDIR, as it is unsafe to keep
it in /tmp

I do keep /tmp encrypted for that reason as it is not safe. And my
/home is also encrypted on different partition. 

In general it is unsafe to choose /tmp for private information as then
maybe /tmp is not encrypted.

>  - "I don't want my Emacs's session state to spill to
>persistent media"

Me too, unless I use "desktop" and that is saved on encrypted
partition.

But I would not like using Org files on remote servers and that
information is anywhere else stored unencrypted about which file I
opened and what I wrote.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: How to disable completely org-persist

[Angelo Graziosi]

> Il 30/12/2022 18:42 Colin Baxter ha scritto:
> 
>  
> >>>>> Angelo Graziosi writes:
> 
> > tomas wrote:
> >> there is now a defvar `org-element-cache-persistent'
> 
> > Ah! didn't knew that..
> 
> > Setting
> 
> > (setq org-element-cache-persistent nil)
> 
> > seems to work..
> 
> Indeed, although I'm not yet convinced that org-persist directories wont
> eventually start showing up in /tmp!

...or in ~/.cache..

Re: How to disable completely org-persist

[tomas]

On Sat, Dec 31, 2022 at 02:29:43AM +0100, Angelo Graziosi wrote:
> 
> > Il 30/12/2022 18:42 Colin Baxter ha scritto:
> > 
> >  
> > >>>>> Angelo Graziosi writes:
> > 
> > > tomas wrote:
> > >> there is now a defvar `org-element-cache-persistent'
> > 
> > > Ah! didn't knew that..
> > 
> > > Setting
> > 
> > > (setq org-element-cache-persistent nil)
> > 
> > > seems to work..
> > 
> > Indeed, although I'm not yet convinced that org-persist directories wont
> > eventually start showing up in /tmp!
> 
> I fear that sooner or later it will reappear somewhere.. 

Hm. I'm sure people are accepting patches?

Having persistence "centralised" in an org-persist facility
should make it easier for you (or me) to control what happens.

Otherwise, each exporter might be creating files everywhere.

Perhaps it would make sense to state the goals in more
detail. I can see many "aspects":

 - "I don't want my ~/.emacs.d/ littered beyond my explicit
   control"
 - "I don't have (nor want to have) XDG directories"
 - "I don't want things in /tmp" [1]
 - "I don't want my Emacs's session state to spill to
   persistent media"

What is it what you (or me, or anyone) wants to achieve?

Cheers

[1] Currently I see two directories in /tmp made by my emacs
   session: some `babel-...', which seems to be made at Org mode
   load (before any babel things are invoked), and `emacs1000'
   (I start Emacs as a server, so that's where the socket goes)
   
-- 
t


signature.asc
Description: PGP signature

Re: How to disable completely org-persist

[Ihor Radchenko]

Angelo Graziosi  writes:

>> Indeed, although I'm not yet convinced that org-persist directories wont
>> eventually start showing up in /tmp!
>
> I fear that sooner or later it will reappear somewhere.. 
>
> When it showed up few week ago, I found that setting and added it to the init 
> file but that time it didn't solved the issue. That folder showed up even 
> with that setting.. Instead now it seems to fix the issue hmm.. I wouldn't 
> want it to be a Pyrrhic victory!

I may indeed reappear somewhere.
As I stated in my last message and the commits, we now have the
following:

1. For emacs -Q, the folder, if created, is removed upon closing Emacs
   It may still stay if you do something like kill -9 (I can't do much
   about this scenario).
   Note that this situation is rather unlikely scenario. You need to do
   something manually with emacs -Q.
2. For normal emacs usage, the folder is created only when it is needed
   If you set org-element-cache-persistent to nil, Org may still use
   org-persist for other purposes, like LaTeX previews or caching remote
   images during export.
3. As I suggested in my other message, if you don't like (2), set
   `org-persist-directory' to the location that does not bother you.

   Alternatively, you can use
   https://github.com/emacscollective/no-littering that will organize
   .emacs.d closer to XDG schema.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable completely org-persist

[Angelo Graziosi]

> Il 30/12/2022 18:42 Colin Baxter ha scritto:
> 
>  
> >>>>> Angelo Graziosi writes:
> 
> > tomas wrote:
> >> there is now a defvar `org-element-cache-persistent'
> 
> > Ah! didn't knew that..
> 
> > Setting
> 
> > (setq org-element-cache-persistent nil)
> 
> > seems to work..
> 
> Indeed, although I'm not yet convinced that org-persist directories wont
> eventually start showing up in /tmp!

I fear that sooner or later it will reappear somewhere.. 

When it showed up few week ago, I found that setting and added it to the init 
file but that time it didn't solved the issue. That folder showed up even with 
that setting.. Instead now it seems to fix the issue hmm.. I wouldn't want it 
to be a Pyrrhic victory!

Re: How to disable completely org-persist

[Colin Baxter]

>>>>> Angelo Graziosi  writes:

> tomas wrote:
>> there is now a defvar `org-element-cache-persistent'

> Ah! didn't knew that..

> Setting

> (setq org-element-cache-persistent nil)

> seems to work..

Indeed, although I'm not yet convinced that org-persist directories wont
eventually start showing up in /tmp!

Best wishes,

Re: How to disable completely org-persist

[Ihor Radchenko]

to...@tuxteam.de writes:

>> The variable has been there since the beginning.
>
> But you made me aware of it, and thanks for it :)

https://orgmode.org/Changes.html ;)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: How to disable completely org-persist

[tomas]

On Fri, Dec 30, 2022 at 01:35:39PM +, Ihor Radchenko wrote:
>  writes:
> 
> > AFAIK there is now a defvar `org-element-cache-persistent'. Maybe it
> > works for you?
> 
> The variable has been there since the beginning.

But you made me aware of it, and thanks for it :)

> Now, org-persist is just more careful not to create the directory when
> nothing needs to be stored there.

Understood.

Cheers & may the new year...
-- 
t


signature.asc
Description: PGP signature

Re: How to disable completely org-persist

[Ihor Radchenko]

 writes:

> AFAIK there is now a defvar `org-element-cache-persistent'. Maybe it
> works for you?

The variable has been there since the beginning.
Now, org-persist is just more careful not to create the directory when
nothing needs to be stored there.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable completely org-persist

[Angelo Graziosi]

tomas wrote:

> there is now a defvar `org-element-cache-persistent'

Ah! didn't knew that..

Setting 

(setq org-element-cache-persistent nil)

seems to work..

Thanks,
 Angelo.

Re: How to disable completely org-persist

[tomas]

On Fri, Dec 30, 2022 at 01:11:12PM +0100, Angelo Graziosi wrote:
> > Il 05/12/2022 18:58 Colin Baxter ha scritto:
> > 
> >  
> > >>>>> Angelo Graziosi writes:
> > 
> > > Colin Baxter wrote:
> > >> I too would like a means to disable org-persist.
> > 
> > > On emacs-devel there was this suggestion:
> > 
> > > https://lists.gnu.org/archive/html/emacs-devel/2022-12/msg00125.html
> > 
> > It appears to work, after removing a couple of surplus brackets from the
> > end of the last line.
> 
> After the last Emacs master build, the what suggested there, i.e.
> 
> ;; As suggested here:
> ;; https://lists.gnu.org/archive/html/emacs-devel/2022-12/msg00125.html
> (setq org-persist-disable-when-emacs-Q t)
> 
> (defun tv/advice--org-persist (old-fn &rest args)
>   (let (user-init-file)
>     (apply old-fn args)))
> 
> (advice-add 'org-persist-write :around #'tv/advice--org-persist)
> (advice-add 'org-persist-read :around #'tv/advice--org-persist)
> (advice-add 'org-persist-gc :around #'tv/advice--org-persist)
> 
> does not work any more: that directory is recreated even with that cod in the 
> init.el
> 
> Someone knows how to avoid that?

AFAIK there is now a defvar `org-element-cache-persistent'. Maybe it
works for you?

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: How to disable completely org-persist

[Angelo Graziosi]

> Il 05/12/2022 18:58 Colin Baxter ha scritto:
> 
>  
> >>>>> Angelo Graziosi writes:
> 
> > Colin Baxter wrote:
> >> I too would like a means to disable org-persist.
> 
> > On emacs-devel there was this suggestion:
> 
> > https://lists.gnu.org/archive/html/emacs-devel/2022-12/msg00125.html
> 
> It appears to work, after removing a couple of surplus brackets from the
> end of the last line.

After the last Emacs master build, the what suggested there, i.e.

;; As suggested here:
;; https://lists.gnu.org/archive/html/emacs-devel/2022-12/msg00125.html
(setq org-persist-disable-when-emacs-Q t)

(defun tv/advice--org-persist (old-fn &rest args)
  (let (user-init-file)
    (apply old-fn args)))

(advice-add 'org-persist-write :around #'tv/advice--org-persist)
(advice-add 'org-persist-read :around #'tv/advice--org-persist)
(advice-add 'org-persist-gc :around #'tv/advice--org-persist)

does not work any more: that directory is recreated even with that cod in the 
init.el

Someone knows how to avoid that?

TIA,
 Angelo.

Re: org-persist files in /tmp

[tomas]

On Sun, Dec 25, 2022 at 09:30:00AM +, Ihor Radchenko wrote:
>  writes:
> 
> >> Another idea is to avoid caching of parse result for small files.
> >
> > I haven't been following along very closely, but seeing the
> > involved complexities, I'd appreciate a knob to disable caching
> > altogether.
> 
> org-element-cache-persistent

Thanks, Ihor. You made me a Christmas present *<:-)~

> > My usage of Org hasn't triggered any slowdowns which would be
> > worth all that complexity (and yes, my biggest Org file so-far
> > is 36k lines, and my box isn't the fastest around).
> 
> The main reason we cannot disable org-persist is _not_ Org AST cache.
> The reason is mostly caching some downloaded files.

I see. That latency weighs significantly more, yes.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: org-persist files in /tmp

[Ihor Radchenko]

Max Nikulin  writes:

> If you demonstrate that e.g., when working with encrypted files, their 
> sensitive content leaks to the cache then it will raise the severity of 
> the issue.

Nothing related to encrypted files is ever written by org-persist.
Some edge cases might exist for org-crypt, but that will involve manual
calls to writing cache.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist files in /tmp

[Ihor Radchenko]

 writes:

>> Another idea is to avoid caching of parse result for small files.
>
> I haven't been following along very closely, but seeing the
> involved complexities, I'd appreciate a knob to disable caching
> altogether.

org-element-cache-persistent

> My usage of Org hasn't triggered any slowdowns which would be
> worth all that complexity (and yes, my biggest Org file so-far
> is 36k lines, and my box isn't the fastest around).

The main reason we cannot disable org-persist is _not_ Org AST cache.
The reason is mostly caching some downloaded files.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist files in /tmp

[Ihor Radchenko]

Max Nikulin  writes:

> Ihor, I do not like that after your latest changes temporary directory 
> became world readable.

This is the only sane way for emacs -Q, AFAIK. And the cache will now
only exist while Emacs is running (for -Q cmd arg). Removed
unconditionally upon exiting.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: org-persist files in /tmp

[Ihor Radchenko]

Greg Minshall  writes:

>> Do we need to care about cleaning up /tmp?
>
> my two cents is that maybe one should not care so much about cleaning up
> /tmp, but i think it's worthwhile trying not to clutter it up too much.

I improved things a tiny bit more.
Now, whatever is created by emacs -Q will be removed as long as
`kill-emacs-hook' is executed. It is the most aggressive cleaning I can
imagine.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: org-persist files in /tmp

[tomas]

On Fri, Dec 23, 2022 at 09:12:52PM +0700, Max Nikulin wrote:

[...]

> If you demonstrate that e.g., when working with encrypted files, their
> sensitive content leaks to the cache then it will raise the severity of the
> issue. Of course, the always appreciated option is to provide a patch that
> consistently makes org-persist optional.

Understood that hint, and you are, of course, right :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: org-persist files in /tmp

[Max Nikulin]

On 23/12/2022 13:35, to...@tuxteam.de wrote:

On Thu, Dec 22, 2022 at 11:07:52PM +0700, Max Nikulin wrote:

Another idea is to avoid caching of parse result for small files.


I haven't been following along very closely, but seeing the
involved complexities, I'd appreciate a knob to disable caching
altogether.


Ihor wrote that org-persist usage is not limited to org-element cache. 
Another case is referenced remote files. Perhaps e.g. LaTeX preview 
still uses old approach for generated image, but it is rather close case.


I do not mind to have a setting that disables org-persist, but it 
requires some amount of work. Its priority unlikely will be high.


If you demonstrate that e.g., when working with encrypted files, their 
sensitive content leaks to the cache then it will raise the severity of 
the issue. Of course, the always appreciated option is to provide a 
patch that consistently makes org-persist optional.

Re: org-persist files in /tmp

[tomas]

On Thu, Dec 22, 2022 at 11:07:52PM +0700, Max Nikulin wrote:
> On 22/12/2022 22:45, Tim Cross wrote:
> > Could some of the issues people are concerned about regarding use of
> > /tmp be avoided if instead the temporary files were put into ~/.cache?

[...]

> Another idea is to avoid caching of parse result for small files.

I haven't been following along very closely, but seeing the
involved complexities, I'd appreciate a knob to disable caching
altogether.

My usage of Org hasn't triggered any slowdowns which would be
worth all that complexity (and yes, my biggest Org file so-far
is 36k lines, and my box isn't the fastest around).

But, of course, it's your call :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: org-persist files in /tmp

[Max Nikulin]

On 22/12/2022 22:45, Tim Cross wrote:

Could some of the issues people are concerned about regarding use of
/tmp be avoided if instead the temporary files were put into ~/.cache?


There is no ~/.cache on Windows, the fallback is ~/.emacs.d. org-persist 
files in ~/.emacs.d was the original source of complains. Moreover, I do 
not think emacs -q should write to the same places as emacs initialized 
in the regular way.


I see a couple of options:
- Remove the directory on exit (taking some care to not prevent quit in 
the cases of errors)
- Create the directory lazily. The variable should not be accessed 
directly, when some code is going to write a file it should call a 
function that creates the directory if it is first call.


Another idea is to avoid caching of parse result for small files.

Re: org-persist files in /tmp

[Tim Cross]

Max Nikulin  writes:

> On 22/12/2022 19:34, Ruijie Yu wrote:
>> One possible approach to this is to have all org-persist related
>> temporary directories into an overall "$TMPDIR/org-persist" directory.
>
> Predictable name in a "world" writable directory generally is not a good 
> idea. Multiple
> users may try to run Org on the same machine. There are some kernel 
> parameters to prevent
> certain type of attacks, however I am unsure concerning their default values 
> in various
> Linux distributions and what will happen if one user creates a symlink to 
> somewhere the
> under home directory of another one. So unfortunately a directory reusable by 
> different
> emacs sessions should be avoided.
>
> Ihor, I do not like that after your latest changes temporary directory became 
> world
> readable.
>
> Another point is that creating temporary files and directories must be an 
> atomic
> operation. In between of removing and recreating it an attacker might manage 
> to create a
> file with the same name.

Could some of the issues people are concerned about regarding use of
/tmp be avoided if instead the temporary files were put into ~/.cache?
To me, that would seem to be the appropriate location for such files. It
would mean that org would need to 'manage' or clean out old files, but
that shouldn't be a big issue.

Re: org-persist files in /tmp

[Max Nikulin]

On 22/12/2022 19:34, Ruijie Yu wrote:

One possible approach to this is to have all org-persist related
temporary directories into an overall "$TMPDIR/org-persist" directory.


Predictable name in a "world" writable directory generally is not a good 
idea. Multiple users may try to run Org on the same machine. There are 
some kernel parameters to prevent certain type of attacks, however I am 
unsure concerning their default values in various Linux distributions 
and what will happen if one user creates a symlink to somewhere the 
under home directory of another one. So unfortunately a directory 
reusable by different emacs sessions should be avoided.


Ihor, I do not like that after your latest changes temporary directory 
became world readable.


Another point is that creating temporary files and directories must be 
an atomic operation. In between of removing and recreating it an 
attacker might manage to create a file with the same name.

Re: org-persist files in /tmp

[General discussions about Org-mode.]

Greg Minshall  writes:

> hi, Ihor,
>
>> Do we need to care about cleaning up /tmp?
>
> my two cents is that maybe one should not care so much about cleaning up
> /tmp, but i think it's worthwhile trying not to clutter it up too much.
>
> cheers, Greg

One possible approach to this is to have all org-persist related
temporary directories into an overall "$TMPDIR/org-persist" directory.
That is, assuming that the parent directory exists, we create
org-persist temporary directories as "$TMPDIR/org-persist/XX" and
everything else would remain the same.

The downside for this approach is that, since `make-tempfile' only makes
a mkdir() call in its underlying function try_dir() assuming the
existence of its parents [1, 2], we would probably have to create the
parent via `(mkdir DIR t)' before every `make-tempfile' call within
org-persist.

[1] emacs/src/fileio.c
[2] emacs/lib/tempname.c

Best,


RY

Re: org-persist files in /tmp

[Tim Cross]

Ihor Radchenko  writes:

> "Fraga, Eric"  writes:
>
>> for some reason, I am now getting many (tens) directories of the form
>> org-persist-NN in /tmp.  These seem to include an index file and a
>> cache type sub-directory structure.  Why are these there and does
>> anything clean them up?
>>
>> I have nothing related to org-persist in my configuration that I can
>> see.
>
> If you run something like make test or emacs -Q + org, it is expected.
> These are throwaway directories used by org-persist for emacs -Q.
>
> Do we need to care about cleaning up /tmp?

Probably not - at least not on most modern Linux systems as these tend
to have a systemd task which will clean up the temp directories on
reboot. You can usually tweak the settings for systemd-tempfiles if you
want to modify when and how often temporary files are cleaned up.

Re: org-persist files in /tmp

[Greg Minshall]

hi, Ihor,

> Do we need to care about cleaning up /tmp?

my two cents is that maybe one should not care so much about cleaning up
/tmp, but i think it's worthwhile trying not to clutter it up too much.

cheers, Greg

Re: org-persist files in /tmp

[Ihor Radchenko]

William Denton  writes:

>> You should _not_ see something like
>>
>> ((:container
>>  ((index "2.7"))
>>  :persist-file "d0/5078fe-5e31-4ddb-95a0-93ceae58df0c" :associated nil 
>> :expiry never :last-access 1671637032.483552 :last-access-hr 
>> "2022-12-21T18:37:12+0300"))
>>
>> as the only contents of "index" file.
>
> I just checked my /tmp/ and I have 32 org-persist directories, and I must 
> sadly 
> report they all have index files like I shouldn't see:

Thanks for the heads-up!
Fixed on bugfix now.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e2366ac28

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist files in /tmp

[William Denton]

On 21 December 2022, Ihor Radchenko wrote:


Could be. If so, it may indicate some issue with logic. The index should
not be written if the only entry inside index file is index version
itself.

You should _not_ see something like

((:container
 ((index "2.7"))
 :persist-file "d0/5078fe-5e31-4ddb-95a0-93ceae58df0c" :associated nil :expiry never 
:last-access 1671637032.483552 :last-access-hr "2022-12-21T18:37:12+0300"))

as the only contents of "index" file.


I just checked my /tmp/ and I have 32 org-persist directories, and I must sadly 
report they all have index files like I shouldn't see:


 ((:container
   ((index "2.7"))
   :persist-file "2f/859289-dcc1-4d55-8c91-e22b4ccc92dd" :associated nil :expiry never 
:last-access 1671429856.980864 :last-access-hr  "2022-12-19T01:04:16-0500"))

Nineteen of them were made on Monday ago, and I may have rebuilt Emacs and Org 
then, and maybe restarted Emacs, but not nineteen times.


Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada

Re: org-persist files in /tmp

[Ihor Radchenko]

"Fraga, Eric"  writes:

> On Wednesday, 21 Dec 2022 at 15:06, Ihor Radchenko wrote:
>> If you run something like make test or emacs -Q + org, it is expected.
>> These are throwaway directories used by org-persist for emacs -Q.
>
> I am not explicitly running either of those (at the moment).
>
> I wonder if this is somehow related to native compilation or some other
> post-build processes as I have built Emacs twice from git today (and I
> always update org when I build Emacs) and these directories have
> appeared soon thereafter (I think).

Could be. If so, it may indicate some issue with logic. The index should
not be written if the only entry inside index file is index version
itself.

You should _not_ see something like

((:container
  ((index "2.7"))
  :persist-file "d0/5078fe-5e31-4ddb-95a0-93ceae58df0c" :associated nil :expiry 
never :last-access 1671637032.483552 :last-access-hr 
"2022-12-21T18:37:12+0300"))

as the only contents of "index" file.

>> Do we need to care about cleaning up /tmp?
>
> Maybe not but I do use /tmp for some things and having tens of these
> directories does clutter things up a bit.  I'm happy to ignore but was
> curious as to why they were there at all.

This has been introduced in

2944a2152d491452697cc4db538d6b2344c0e37d
Author: Ihor Radchenko 
org-persist: Use temporary index for emacs -Q

* lisp/org-persist.el (org-persist--disable-when-emacs-Q): Rename
`org-persist-disable-when-emacs-Q' to internal variable.  Update the
docstring.
(org-persist-read):
(org-persist-write):
(org-persist-gc): Do not disable persistence.  Persistence is
necessary for remote file caching to work within a single Emacs
session.  Instead, use temporary directory as index for emacs -Q.

Also, see
https://orgmode.org/list/1158097067.265983.1670026787...@mail1.libero.it

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-persist files in /tmp

[Fraga, Eric]

On Wednesday, 21 Dec 2022 at 15:06, Ihor Radchenko wrote:
> If you run something like make test or emacs -Q + org, it is expected.
> These are throwaway directories used by org-persist for emacs -Q.

I am not explicitly running either of those (at the moment).

I wonder if this is somehow related to native compilation or some other
post-build processes as I have built Emacs twice from git today (and I
always update org when I build Emacs) and these directories have
appeared soon thereafter (I think).

> Do we need to care about cleaning up /tmp?

Maybe not but I do use /tmp for some things and having tens of these
directories does clutter things up a bit.  I'm happy to ignore but was
curious as to why they were there at all.

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.6-124-g036cc0 in Emacs 30.0.50

Re: org-persist files in /tmp

[Ihor Radchenko]

"Fraga, Eric"  writes:

> for some reason, I am now getting many (tens) directories of the form
> org-persist-NN in /tmp.  These seem to include an index file and a
> cache type sub-directory structure.  Why are these there and does
> anything clean them up?
>
> I have nothing related to org-persist in my configuration that I can
> see.

If you run something like make test or emacs -Q + org, it is expected.
These are throwaway directories used by org-persist for emacs -Q.

Do we need to care about cleaning up /tmp?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

org-persist files in /tmp

[Fraga, Eric]

Dear all,

for some reason, I am now getting many (tens) directories of the form
org-persist-NN in /tmp.  These seem to include an index file and a
cache type sub-directory structure.  Why are these there and does
anything clean them up?

I have nothing related to org-persist in my configuration that I can
see.

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.6-120-gd9e258 in Emacs 30.0.50

Re: How to disable completely org-persist

[Ihor Radchenko]

Ihor Radchenko  writes:

> Ihor Radchenko  writes:
>
>> Upon further investigation, it looks like we need Org persist all the
>> time. As temporary storage, at least. In particular, downloading remote
>> images requires org-persist to store files somewhere. All the times.
>> ...
>> So, we cannot disable org-persist and should not even do it with Emacs
>> -Q. Instead, we may redirect org-persist-directory to /tmp in some
>> scenarios. I will look into doing this soon.
>
> I am going to do the following in order to reduce the issue with
> littering `user-emacs-directory':
> ...

Done on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=aa86ed534
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=2944a2152

Fixed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>