Re: Apptainer/singularity container connection

[Michael Albinus]

Nasser Alkmim  writes:

Hi Nasser,

> I tested here and it works.
>
> I'm only having problems with the debugger (dape), but I think it is a 
> mistake with the configuration.
>
> Compilation and the LSP with eglot work nicely.

Thanks for the feedback. I've pushed an extended version of the patch
into the repositories. I've decided to make "apptainer" an optional
Tramp method, which means that you have to enable it when you want to
use it. Drop the following line into your .emacs:

--8<---cut here---start->8---
(with-eval-after-load 'tramp (tramp-enable-method "apptainer"))
--8<---cut here---end--->8---

Please report if there are problems with this method we could solve in Tramp.

> Best regards,

Best regards, Michael.

Re: Apptainer/singularity container connection

[Michael Albinus]

Nasser Alkmim  writes:

Hi,

> Thanks Michael.
> I can not find the patch, is it attached to the email?

Oops, forgotten. See below.

> Best regards,

Best regards, Michael.

diff --git a/lisp/tramp-container.el b/lisp/tramp-container.el
index 30639cbe..52c78189 100644
--- a/lisp/tramp-container.el
+++ b/lisp/tramp-container.el
@@ -83,6 +83,15 @@
 ;; Where:
 ;; SANDBOX	is the running sandbox to connect to.
 ;;		It could be an application ID, an instance ID, or a PID.
+;;
+;;
+;;
+;; Open a file on a running Apptainer instance:
+;;
+;; C-x C-f /apptainer:INSTANCE:/path/to/file
+;;
+;; Where:
+;; INSTANCE	is the running instance to connect to.

 ;;; Code:

@@ -142,6 +151,14 @@ If it is nil, the default context will be used."
   :type '(choice (const "flatpak")
  (string)))

+;;;###tramp-autoload
+(defcustom tramp-apptainer-program "apptainer"
+  "Name of the Apptainer client program."
+  :group 'tramp
+  :version "30.1"
+  :type '(choice (const "apptainer")
+ (string)))
+
 ;;;###tramp-autoload
 (defconst tramp-docker-method "docker"
   "Tramp method name to use to connect to Docker containers.")
@@ -172,6 +189,10 @@ This is for out-of-band connections.")
 (defconst tramp-flatpak-method "flatpak"
   "Tramp method name to use to connect to Flatpak sandboxes.")

+;;;###tramp-autoload
+(defconst tramp-apptainer-method "apptainer"
+  "Tramp method name to use to connect to Apptainer instances.")
+
 ;;;###tramp-autoload
 (defmacro tramp-skeleton-completion-function (method  body)
   "Skeleton for `tramp-*-completion-function' with multi-hop support.
@@ -373,6 +394,28 @@ see its function help for a description of the format."
lines)))
   (mapcar (lambda (name) (list nil name)) (delq nil names)

+;;;###tramp-autoload
+(defun tramp-apptainer--completion-function (method)
+  "List Apptainer instances available for connection.
+
+This function is used by `tramp-set-completion-function', please
+see its function help for a description of the format."
+  (tramp-skeleton-completion-function method
+(when-let ((raw-list
+		(shell-command-to-string (concat program " instance list")))
+	   ;; Ignore header line.
+   (lines (cdr (split-string raw-list "\n" 'omit)))
+   (names (mapcar
+		   (lambda (line)
+			 (when (string-match
+(rx bol (group (1+ (not space)))
+(1+ space) (1+ (not space))
+(1+ space) (1+ (not space)))
+line)
+			   (match-string 1 line)))
+   lines)))
+  (mapcar (lambda (name) (list nil name)) (delq nil names)
+
 ;;;###tramp-autoload
 (defvar tramp-default-remote-shell) ;; Silence byte compiler.

@@ -477,6 +520,17 @@ see its function help for a description of the format."
 		(tramp-remote-shell-login ("-l"))
 		(tramp-remote-shell-args ("-c"

+ (add-to-list 'tramp-methods
+	  `(,tramp-apptainer-method
+		(tramp-login-program ,tramp-apptainer-program)
+		(tramp-login-args (("shell")
+   ("instance://%h")
+   ("%h"))) ; Needed for multi-hop check.
+		;; (tramp-direct-async (,tramp-default-remote-shell "-c"))
+		(tramp-remote-shell ,tramp-default-remote-shell)
+		(tramp-remote-shell-login ("-l"))
+		(tramp-remote-shell-args ("-c"
+
  (tramp-set-completion-function
   tramp-docker-method
   `((tramp-container--completion-function ,tramp-docker-method)))
@@ -505,11 +559,16 @@ see its function help for a description of the format."
   tramp-flatpak-method
   `((tramp-flatpak--completion-function ,tramp-flatpak-method)))

+ (tramp-set-completion-function
+  tramp-apptainer-method
+  `((tramp-apptainer--completion-function ,tramp-apptainer-method)))
+
  (add-to-list 'tramp-completion-multi-hop-methods tramp-docker-method)
  (add-to-list 'tramp-completion-multi-hop-methods tramp-podman-method)
  (add-to-list 'tramp-completion-multi-hop-methods tramp-kubernetes-method)
  (add-to-list 'tramp-completion-multi-hop-methods tramp-toolbox-method)
  (add-to-list 'tramp-completion-multi-hop-methods tramp-flatpak-method)
+ (add-to-list 'tramp-completion-multi-hop-methods tramp-apptainer-method)

  ;; Default connection-local variables for Tramp.

Re: Apptainer/singularity container connection

[Michael Albinus]

Nasser Alkmim  writes:

> Hi,

Hi Nasser,

> Does anyone use apptainer (former singularity) containers with TRAMP?
>
> I would like to develop in emacs, but make the container libraries
> available so LSP server and debuger can recognize them.
>
> I tried already adding a tramp-method:
>
> (add-to-list 'tramp-methods
>  '("apptainer" (tramp-login-program "apptainer")
>(tramp-login-args (("shell") ("instance://%h")
>
> Then I start an apptainer instance named "mycontainer".
> So I can start an interactive shell in the container with,
>
> apptainer shell instance://mycontainer
>
> I would like to use TRAMP with "/apptainer:mycontainer:/"?
>
> Any ideas?

Well, it is a good idea to add Apptainer to Tramp. I have made a small
proof of concept, see the appended patch for tramp-container.el. You can
apply it on top of this file, either from the Tramp or Emacs git repo,
master branch.

> Thanks

Best regards, Michael.

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

> Hi Michael,

Hi Tijs,

> I've tested the patch and now it's possible again to edit the remote
> files with tramp after changing the option
> tramp-warn-only-if-setting-file-attributes-fail to t.

Thanks for the feedback. I've pushed the patch to the repositories, will
appear with Tramp 2.7.1.

Finally, I've changed the name of the user option to
tramp-inhibit-errors-if-setting-file-attributes-fail.

> Thank you very much!
> Tijs

Best regards, Michael.

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

> Hi Michael,

Hi Tijs,

> Ok I see, a user option would be perfect in that case.

Please try the appended patch. Since it changes a macro, you must
recompile both tramp.el and tramp-sh.el (in that order).

Then try your test, again, setting the new user option
tramp-warn-only-if-setting-file-attributes-fail to t in advance.

> Thank you!
> Tijs

Best regards, Michael.

diff --git a/lisp/tramp.el b/lisp/tramp.el
index 5b101000..e4c8c1aa 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -3533,6 +3533,11 @@ on the same host.  Otherwise, TARGET is quoted."

,@body)))

+(defcustom tramp-warn-only-if-setting-file-attributes-fail nil
+  "Whether to warn only if `tramp-*-set-file-{modes,times,uid-gid}' fails."
+  :version "30.1"
+  :type 'boolean)
+
 (defmacro tramp-skeleton-set-file-modes-times-uid-gid
 (filename  body)
   "Skeleton for `tramp-*-set-file-{modes,times,uid-gid}'.
@@ -3548,7 +3553,11 @@ BODY is the backend specific code."
 	 ;; "file-writable-p".
 	 '("file-directory-p" "file-exists-p" "file-symlinkp" "file-truename")
(tramp-flush-file-properties v localname))
- ,@body))
+ (condition-case err
+	 (progn ,@body)
+   (error (if tramp-warn-only-if-setting-file-attributes-fail
+		  (display-warning 'tramp (error-message-string err))
+		(signal (car err) (cdr err)))

 (defmacro tramp-skeleton-write-region
   (start end filename append visit lockname mustbenew  body)

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

> Hi Michael,

Hi Tijs,

> Would it be an idea to check first if the user can run the chmod
> command before calling the tramp-sh-handle-set-file-modes function?

Too much hassle, because this would be checked for everybody, with
performance degradation.

What about a user option, which (when non-nil) transforms the error into
a warning?

> Thank you,
> Tijs

Best regards, Michael.

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

> Hi Michael,

Hi Tijs,

> This is the output of the command:
>
> chmod: changing permissions of ‘/var/www/aaa/index.htm’: Operation not
> permitted
> tramp_exit_status 1

This doesn't tell us much. Hmm.

Could you try to find out, why chmod on that machine does not have
proper permissions for that file? And are you sure, that it is ACL and
not the SELinux security context which is in the way? Perhaps you can
play with chmod's --role and --type arguments, if it supports them?

> Thank you!
> Tijs

Best regards, Michael.

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

> Hi Michael,

Hi Tijs,

[Pls keep keep tramp-devel@gnu.org in Cc, for the archives]

> Here is the output of the tramp debug buffer:

Well, let's see. (Next time pls send it as attachment, sending it as
plain text adds line formatting, which makes it hard to read)

> 18:37:21.563273 tramp-do-copy-or-rename-file (0) # Copying
> /scp:user@aaa:/var/www/aaa/index.htm to
> /mnt/encrypted1/tmp/emacs-auto-save/!scp:user@aaa:!var!www!aaa!index.htm~...done

This is the auto-save file, which succeeds.

> 18:37:21.565837 tramp-send-command (6) # setfacl --set-file=-
> /var/www/aaa/index.htm <<'5facc8e89585e10211c9db6e2745622c'
> user::rw-
> user:apache:rwx
> user:user:rwx
> group::r-x
> mask::rwx
> other::r--
>
> 5facc8e89585e10211c9db6e2745622c
> 18:37:21.620609 tramp-wait-for-regexp (6) #
> ///9d7bf6c80cc21776a3aafba01457c424#$
> 18:37:21.620812 tramp-send-command (6) # echo tramp_exit_status $?
> 18:37:21.656831 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///9d7bf6c80cc21776a3aafba01457c424#$

setfacl sets the ACL's for the given file. It returns exit code 0,
meaning it has succeeded. And the ACL's look normal.

> 18:37:21.657544 tramp-send-command (6) # chcon --user=unconfined_u -
> -role=object_r --type=httpd_sys_content_t --range=s0
> /var/www/aaa/index.htm 2>/dev/null; echo tramp_exit_status $?
> 18:37:21.688826 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///9d7bf6c80cc21776a3aafba01457c424#$

chcon sets the SELinux security context, also successful.

> 18:37:21.690019 tramp-send-command (6) # chmod  674
> /var/www/aaa/index.htm 2>/dev/null; echo tramp_exit_status $?
> 18:37:21.815967 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///9d7bf6c80cc21776a3aafba01457c424#$

And now chmod 674 fails. Unfortunately, we don't see the reson (stderr
is redirected to /dev/null).

Could you please try on the remote machine user@aaa in a shell the
command:

--8<---cut here---start->8---
# chmod  674 /var/www/aaa/index.htm; echo tramp_exit_status $?
--8<---cut here---end--->8---

What do you see?

> Thank you for checking!
> Tijs

Best regards, Michael.

Re: error while changing file’s mode when saving file with ACL permissions

[Michael Albinus]

Tijs Mallaerts  writes:

Hi Tijs,

> Is it possible to configure tramp so it becomes possible to edit files
> that are set up with ACL permissions? Currently when trying to save
> such a file the following error is shown:
>
> tramp-error: Error while changing file’s mode

Tramp doesn't do something special wrt ACLs. Emacs has the functions
file-acl and set-file-acl, and Tramp provides an own implementation for
them in some of its backends (tramp-sh.el, tramp-smb.el, tramp-sudoedit.el).

Tramp doesn't call these functions itself. How they are used is out of
Tramp control. Should Tramp care about?

How does the error look like in detail? Could you, pls, set
tramp-verbose to 6, run your test, and show the resulting Tramp debug buffer?

> Thank you!
> Tijs

Best regards, Michael.

Tramp 2.6.3

[Michael Albinus]

Hello,

The Tramp team is happy to announce the release of Tramp 2.6.3.

Tramp is a remote file editing package for Emacs.  Tramp uses different
methods like ssh or scp to access files on remote hosts as if they were
local files.  Access to the remote file system for editing files,
version control, directory editing and running processes on the remote
host are transparently enabled.

Tramp 2.6.3 supports Emacs since version 26.1.  This is the version
integrated with Emacs 29.3.

The download files are located at .

Tramp 2.6.3 is also available via GNU ELPA.

Best regards, Michael.

Re: tramp readlink slowdown on find-file

[Michael Albinus]

Prashant Tak  writes:

Hi Prashant,

> Is there no other way to disable the amount of checking done by tramp?

Tramp tries already to minimize the number of checks. It has been
optimized over the releases; Tramp 2.7 is the current state.

In order to apply improvements, I need to know which Emacs / Tramp
versions you are using. Furthermore, please rerun a full test (starting
with "emacs -Q --eval '(setq tramp-verbose 6)', send the resulting debug
buffer, and show the superfluous checks.

Furthermore, we should investigate why readlink is such slow
remotely. But for that I need first to know your Tramp version.

What I have seen is that you configure tramp-remote-path as
'(tramp-own-remote-path))). That means, you trust only your own
settings, and we have no idea which version of readlink is used. Could
you, pls, just for test deactivate this setting? (This could include
removing the connection property "remote-path").

> Prashant

Best regards, Michael.

Re: tramp readlink slowdown on find-file

[Michael Albinus]

Prashant Tak  writes:

Hi Prashant,

Pls keep tramp-devel@gnu.org in Cc, for the archives.

>> When you connect to the remote host directly via ssh from a shell, does it
>> take the same time to call '\readlink --canonicalize-missing /a/b/c/xyz.md'?
>
> No, that works instantly. Is there a way to disable multiple file existence
> checks that tramp does?

Tramp caches the result of a check for remote-file-name-inhibit-cache
seconds. Increase that value (it defaults to 10), or set it to nil.

> Prashant

Best regards, Michael.

Re: tramp readlink slowdown on find-file

[Michael Albinus]

Prashant Tak  writes:

> Hi

Hi Prashant,

> while opening a file in tramp, I'm experiencing immense slowdown,
> 20+ seconds to open a file.
> In the debug buffer I found readlink status check to be consuming ~12
> seconds. Any idea what could be causing this?
>
> 21:22:47.919689 tramp-sh-handle-file-truename (4) # Finding true name
> for ‘/ssh:host:/a/b/c/xyz.md’
> 21:22:47.920152 tramp-send-command (6) # \readlink
> --canonicalize-missing /a/b/c/xyz.md 2>/dev/null; echo
> tramp_exit_status $?
> 21:22:59.722106 tramp-wait-for-regexp (6) #
> /a/b/c/xyz.md
> tramp_exit_status 0

When you connect to the remote host directly via ssh from a shell, does it
take the same time to call '\readlink --canonicalize-missing /a/b/c/xyz.md'?

Please keep the backslash when calling readlink. And note that I have
omitted the stderr redirection in order to see possible output.

> Thanks for your wonderful work Michael, the emacs community appreciates it!
>
> Prashant

Best regards, Michael.

Re: Review wanted for method for accessing Mock chroots

[Michael Albinus]

Tim Landscheidt  writes:

Hi Tim,

>> However: I don't know Mock chroots in detail, but it looks to me like it
>> is something what could live in tramp-container.el. What do you think a bout?
>
> It probably fits quite nicely, but due to my preference for
> sticking to distribution packages, I'd need an interim solu-
> tion until Emacs 30 makes its way to my system. :-)

You don't. Tramp is distributed and updated also via GNU ELPA. The
current release on ELPA is 2.6.2.2. Later this year, likely end of June,
this will be Tramp 2.7, which could contain your package.

>> Why "2.7.1-pre"? This isn't a released version; I would depend on "2.7.1".
>
> Because I copied the version string from the wrong file; I
> changed it to "2.5.4-pre" (as that is what was and is
> shipped on Fedora 38).

In this case, I would say at least "2.5.3", which is a stable
release. "2.5.4-pre" is nothing one could determine correctly.

Fedora 38 comes with Emacs 28.3 (which is an unofficial Emacs
release). This is compatible with Tramp 2.7, which says

--8<---cut here---start->8---
;; Package-Requires: ((emacs "27.1"))
--8<---cut here---end--->8---

So you can install Tramp from GNU ELPA without pain.

Well, it is up to you to decide, whether you want to inregrate into
Tramp proper, or distribute a standalone package. But in the former, you
would profit from fixes in Tramp automatically.

> Tim

Best regards, Michael.

Re: Connection local profile does not change variables

[Michael Albinus]

Simon Chevolleau  writes:

> Hi Michael,

Hi Simon,

> It looks like my connection profile is indeed set correctly when I
> connect to my remote server.
> The output looks like this, so I guess this is working at intended, now
> I have to figure out why my packages aren't correctly working with those
> variables set now:

Don't forget that connection-local variables are implemented as
buffer-local variables. If you want to profit from them in a given
buffer, you must call *in that buffer* one of the functions or macros
mentioned in (info "(elisp) Applying Connection Local Variables"), for
example `hack-connection-local-variables-apply' or
`with-connection-local-variables'.

> Thank you a lot!
> Best regards,
> Simon

Best regards, Michael.

Re: Connection local profile does not change variables

[Michael Albinus]

Simon Chevolleau  writes:

> Hi Michael,

Hi Simon,

> After looking to local variables using `apropos-local-variable`, it
> appears that variables from my connection local profile are not set.
> Even by settings this:
>   (connection-local-set-profiles
>nil 'remote-bird)
>
> What are my best ways to dig into this problem ?

Run the settings you have shown in your previous message:

--8<---cut here---start->8---
(connection-local-set-profile-variables
 'remote-bird
 '((detached-shell-program . "~/bin/dtach")
   (detached-session-directory . "~/tmp")
   (detached-dtach-program . "dtach")
   (shell-file-name . "/usr/bin/bash")
   (shell-command-switch . "-c")
   (conda-anaconda-home . "/CONDAS/users/schevolleau/anaconda3")
   (conda--executable-path . 
"/CONDAS/users/schevolleau/anaconda3/condabin/conda")
   (shell-interactive-switch . "-i")))

(connection-local-set-profiles
 '(:application tramp :protocol "ssh") 'remote-bird)
--8<---cut here---end--->8---

Then check, whether they are set:

--8<---cut here---start->8---
(let ((default-directory "/ssh:localhost:"))
  (with-connection-local-variables
   (message "%S" (pp (buffer-local-variables)
--8<---cut here---end--->8---

You should see your settings.

> Best regards,
> Simon

Best regards, Michael.

Re: Connection local profile does not change variables

[Michael Albinus]

Simon Chevolleau  writes:

> Hi TRAMP community,

Hi Simon,

> I'm trying to set connection local profile work.
> I connect to a remote directory through SSH using bookmark.
> But when I check the variables, none have changed and
> connection-local-criteria is set to nil (indicating that no profile has
> been set ?).
> Can someone help me understand what's going on ?
>
> Best regards,
>
> Config:
> (after! tramp
>   (setq connection-local-criteria-alist nil)
>   (setq connection-local-profile-alist nil)
>   (connection-local-set-profile-variables
>'remote-bird
>'((detached-shell-program . "~/bin/dtach")
>  (detached-session-directory . "~/tmp")
>  (detached-dtach-program . "dtach")
>  (shell-file-name . "/usr/bin/bash")
>  (shell-command-switch . "-c")
>  (conda-anaconda-home . "/CONDAS/users/schevolleau/anaconda3")
>  (conda--executable-path . 
> "/CONDAS/users/schevolleau/anaconda3/condabin/conda")
>  (shell-interactive-switch . "-i")))
>
>   (connection-local-set-profiles
>'(:application tramp :protocol "ssh") 'remote-bird)
>   )

connection-local-criteria is a global variable, which is let-bound while
executing with-connection-local-variables. Afterwards, it is nil,
again. So this doesn't mean that connection-local variables aren't set.

You might check instead the values of the buffer-local variables you
have declared, whether they are set accordingly in the remote buffer.

> Simon CHEVOLLEAU

Best regards, Michael.

Re: Review wanted for method for accessing Mock chroots

[Michael Albinus]

Tim Landscheidt  writes:

> Hi,

Hi Tim,

> Is there anything blatantly wrong/not backward-compatible/
> not foreward-compatible about it?  I saved on (customizable)
> variables as it is hard to predict what other users might
> want/need to change.  Any advice is appreciated.

Some few comments below.

However: I don't know Mock chroots in detail, but it looks to me like it
is something what could live in tramp-container.el. What do you think a bout?

Here are some few comments from roughly scanning the code:

> ;;; mock-tramp.el --- TRAMP integration for Mock chroots  -*- 
> lexical-binding: t; -*-

Tramp is spelled out “Tramp”. See the manual.

> ;; Package-Requires: ((tramp "2.7.1-pre"))

Why "2.7.1-pre"? This isn't a released version; I would depend on "2.7.1".

> ;;;###autoload
> (defcustom mock-tramp-method "mock"
>   "TRAMP method to connect to Mock chroots."
>   :type 'string
>   :group 'mock-tramp)

You can keep the :group out; the last declared defgroup is taken by
default.

> ;;;###autoload
> (defun mock-tramp--list-chroots (directory)
>   "Return a list of chroots defined in DIRECTORY.

Why ;;;###autoload?

> ;;;###autoload
> (with-eval-after-load 'tramp
>   (add-to-list 'tramp-methods

Same here.

>  (tramp-login-program "mock")

I would declare and use `mock-tramp-program'.

>  (tramp-login-args (("-r") ("%h")
> ("--shell"
>  "--"
>  "/usr/bin/env"
>  "PROMPT_COMMAND="
>  "/bin/sh"

I would use `tramp-default-remote-shell'.

>  "-l")))
>  (tramp-remote-shell "/bin/sh")

Same here.

> TIA,
> Tim

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]

coder.nov...@mailbox.org writes:

Hi,

> I have a Yubikey 5 NFC with the USBA port. And the error message now
> as I was trying to test for it again suddenly is just gone
> now.. unsure why but it works fine now! And no problem, glad I was
> able to help get the bug fixed!

Good! As said in the other message, the yubikey and Tramp have different
timeouts. Likely you've entered a Tramp timeout, which is not an error
but intended.

Best regards, Michael.

Re: [PATCH] Fix typo in the documentation

[Michael Albinus]

Daniel Martín  writes:

Hi Daniel,

> The attached patch fixes a small typo in the TRAMP documentation.

Thanks for the patch, I've pushed it to the repositories.

Best regards, Michael.

Re: Remote LSP unable to find texlab server

[Michael Albinus]

Riccardo Romoli  writes:

> Michael,

Hi Riccardo,

> I've been able to solve the issue...even if I do not
> understand why...I removed and re-installed some packages, cleared all
> the connections and rebooted both my local machine than the severe a
> couple of times... Thanks for helping me to understand the log file of
> tramp, and to better understand how it works.

Good, so there's nothing left to do for me.

> All the best, Riccardo

Best regards, Michael.

Re: Remote LSP unable to find texlab server

[Michael Albinus]

Riccardo Romoli  writes:

> Good morning Michael, 

Hi Riccardo,

> Did you really start with emacs -Q, and have sent
>   the *whole* log?
>
> I'm not an expert emacs user so it has been quite difficult to be able
> to run the command you suggested without errors. Due to the spacemacs
> layers config, to loar lsp-mode I wrote a specific init.el file and
> ran emacs -Q -l my-specific-init-el.

I've adapted your my-specific-init-el to my needs, It is almost
identical to your version except some load paths:

--8<---cut here---start->8---
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/lsp-mode-20240204.835")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/dash-20240103.1301")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/f-20231219.750")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/s-20220902.1511")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/ht-20230703.558")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/lv-0.15.0")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/markdown-mode-2.6")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/spinner-1.7.4")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/lsp-latex-20231104.313")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/consult-1.2")
(add-to-list 'load-path "/home/albinus/.emacs.d/elpa/compat-29.1.4.4")
--8<---cut here---end--->8---

This shouldn't matter. Then I've opened Emacs as specified by you,
'emacs -Q -l my-specific-init-el'.

> First of all I try to open a local latex file to check if lsp works
> using this configuration file.

I haven't done so, because I don't know whether lsp-mode caches
something. Please do the same, and test directly with the remote file.

> Then I open the remote latex file and manually load lsp (M-x
> lsp-mode).

I've done also via 'C-x C-f /ssh:ubuntu-2204:Downloads/paper.tex'. I
didn't need to activate lsp-mode, it started automatically. From the
*Messages* buffer:

--8<---cut here---start->8---
Tramp: Inserting ‘/ssh:ubuntu-2204:/home/admin/Downloads/paper.tex’...done
LSP :: The following servers support current file but do not have automatic 
installation: texlab-tramp digestif-tramp
You may find the installation instructions at 
https://emacs-lsp.github.io/lsp-mode/page/languages.
(If you have already installed the server check *lsp-log*).
--8<---cut here---end--->8---

And the *lsp-log* buffer contains

--8<---cut here---start->8---
Command "texlab" is not present on the path.
Command "digestif" is not present on the path.
Command "texlab" is not present on the path.
Command "digestif" is not present on the path.
--8<---cut here---end--->8---

No surprise, because I haven't installed them on the remote side.

However, the *debug tramp/ssh ubuntu-2204* buffer contains the whole
Tramp history, starting with establishing the connection and visiting my
LaTeX example file /ssh:ubuntu-2204:/home/admin/Downloads/paper.tex. It
also tries to set the remote environment properly, including the
directory ~/.cargo/bin (which doesn't exist, of course). 

--8<---cut here---start->8---
12:53:23.996234 tramp-send-command (6) # tramp_bundle_read_file_names 
<<'c63857ea2e20ce04703cfe09646c9fbe'
~/.cargo/bin
/bin
/usr/bin
/sbin
/usr/sbin
/usr/local/bin
/usr/local/sbin
/local/bin
/local/freeware/bin
/local/gnu/bin
/usr/freeware/bin
/usr/pkg/bin
/usr/contrib/bin
/opt/bin
/opt/sbin
/opt/local/bin
c63857ea2e20ce04703cfe09646c9fbe
--8<---cut here---end--->8---

And it tries to find the texlab program, which doesn't exist also:

--8<---cut here---start->8---
12:53:27.189448 tramp-send-command (6) # test -x /bin/texlab 2>/dev/null; echo 
tramp_exit_status $?
12:53:27.194125 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///fb9a9b64230e6c3eb974787de9dfacf6#$
12:53:27.194641 tramp-send-command (6) # test -x /usr/bin/texlab 2>/dev/null; 
echo tramp_exit_status $?
12:53:27.199189 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///fb9a9b64230e6c3eb974787de9dfacf6#$
12:53:27.199697 tramp-send-command (6) # test -x /sbin/texlab 2>/dev/null; echo 
tramp_exit_status $?
12:53:27.202264 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///fb9a9b64230e6c3eb974787de9dfacf6#$
12:53:27.202781 tramp-send-command (6) # test -x /usr/sbin/texlab 2>/dev/null; 
echo tramp_exit_status $?
12:53:27.204749 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///fb9a9b64230e6c3eb974787de9dfacf6#$
12:53:27.205256 tramp-send-command (6) # test -x /usr/local/bin/texlab 
2>/dev/null; echo tramp_exit_status $?
12:53:27.207197 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///fb9a9b64230e6c3eb974787de9dfacf6#$
12:53:27.207725 tramp-send-command (6) # test -x /usr/local/sbin/texlab 
2>/dev/null; 

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]

coder.nov...@mailbox.org writes:

Hi,

> I tested it today and it does let me authenticate the key! But while it's 
> asking me to touch the key, it does display the following text in the buffer
> ```
> File error: [[Regexp ‘^^M*User presence confirmed[
> ^M]*\|^^M*\(Enter PIN for .*\)[
> ^M]*\|^^M*sign_and_send_pubkey: signing failed for .*[
> ^M]*’ not found in 0 secs]]
> ```
> However, once I touch the key, it does let me access the files on the remote 
> machine. So it does seem to be functional other than that!

Thanks for the feedback. Since you are able to access the remote machine
with you PIN-protected yubikey, I've pushed the patch to the
repositories. Will be available on GNU ELPA later this month, as Tramp 2.6.2.2.

The error message ... hmm, it isn't clear to me. One possible reason is
that Tramp has reached a timeout, that means you have waited for a while
until entering the PIN.

Another possibility is that your yubikey shows slightly different
messages while using it. Which yubikey do you use? I have taken a
"YubiKey 5C NFC" for my tests.

Anyway, if the error message persists, pls run the test with
tramp-verbose set to 10, and show the Tramp debug buffer.
 
Best regards, Michael.

Re: Remote LSP unable to find texlab server

[Michael Albinus]

Riccardo Romoli  writes:

> Hi Michael,

Hi Riccardo,

> I tried your suggestion but nothing changed. I started a new emacs,
> cleared the tramp cache but again the same issue. I attached the log
> as you requested, hoping this could help. Let me know.

The log looks strange.

- It is incomplete. Did you really start with emacs -Q, and have sent
  the *whole* log?

- It contains helm backtraces. This makes it harder to analyze, so pls
  don't use helm for the test.

- It contains file notification activities. Pls don't activate them while
  testing.

- It doesn't show any remote process invocation. Did you start the
  remote lsp server?

- You didn't send me the steps how to produce the log. I've asked for
  this, in order to reproduce locally.

> Best regards, Riccardo

Best regards, Michael.

Remote LSP unable to find texlab server

[Michael Albinus]

Riccardo Romoli  writes:

> Hi,

Hi Riccardo,

in the future, pls don't use an empty Subject. I've found your message
in my junk folder.

> I need to setup lsp to work on LaTeX file in a remote machine. I
> installed texlab both in local and in remote machine and add the path
> to the remote texlab in my emacs, spacemacs, init file as follow:
>
> (with-eval-after-load "tramp" (add-to-list 'tramp-remote-path "~/.cargo/bin"))
>   (setenv "PATH" (concat (getenv "PATH") ":~/.cargo/bin/"))
>   (setq exec-path (append exec-path '("~/.cargo/bin/")))
>   (require 'lsp-latex)
>   (setq lsp-latex-texlab-executable "~/.cargo/bin/")
>
>   (with-eval-after-load "tex-mode"
> (add-hook 'tex-mode-hook 'lsp)
> (add-hook 'latex-mode-hook 'lsp))
>
> I still get this from lsp: LSP: The following servers support current
> file but do not have automatic installation: texlab-tramp. Command
> "texlab" is not present on the path.
>
> Do you have any suggestions?

Have you tried my recommendations on SX (cleaning up the Tramp cache
etc)? If it still doesn't work, please start a new Emacs session like

--8<---cut here---start->8---
emacs -Q -l tramp --eval '(setq tramp-verbose 6)'
--8<---cut here---end--->8---

Setup your lsp environment, and rerun the test. If it fails, send the
result of the Tramp debug buffer.

Please report all steps you have performed after starting Emacs. I don't
use lsp myself, so I need the steps to reproduce.

> Best Regards
>
> Riccardo

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]

Michael Albinus  writes:

Hi,

>> Also here's the traces attached as a file.
>
> Thanks. Unfortunately, you haven't send the full traces, so I cannot see
> whether the messages "Confirm user presence for key ...:" and "Enter
> PIN:" disturb each other. Hmm.

That was the problem, indeed. Please remove the patch I've sent
before. The appended patch works in my local environment, could you pls check?

Best regards, Michael.

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 44c0bdc7..3e6fb384 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -544,6 +544,7 @@ shell from reading its init file."
 (tramp-terminal-prompt-regexp tramp-action-terminal)
 (tramp-antispoof-regexp tramp-action-confirm-message)
 (tramp-security-key-confirm-regexp tramp-action-show-and-confirm-message)
+(tramp-security-key-pin-regexp tramp-action-otp-password)
 (tramp-process-alive-regexp tramp-action-process-alive))
   "List of pattern/action pairs.
 Whenever a pattern matches, the corresponding action is performed.
@@ -563,6 +564,7 @@ corresponding PATTERN matches, the ACTION function is called.")
 (tramp-wrong-passwd-regexp tramp-action-permission-denied)
 (tramp-copy-failed-regexp tramp-action-permission-denied)
 (tramp-security-key-confirm-regexp tramp-action-show-and-confirm-message)
+(tramp-security-key-pin-regexp tramp-action-otp-password)
 (tramp-process-alive-regexp tramp-action-out-of-band))
   "List of pattern/action pairs.
 This list is used for copying/renaming with out-of-band methods.
diff --git a/lisp/tramp.el b/lisp/tramp.el
index 61f51272..f3da56e7 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -788,6 +788,13 @@ The regexp should match at end of buffer."
   :version "28.1"
   :type 'regexp)

+(defcustom tramp-security-key-pin-regexp
+  (rx bol (* "\r") (group "Enter PIN for " (* nonl)) (* (any "\r\n")))
+  "Regular expression matching security key PIN prompt.
+The regexp should match at end of buffer."
+  :version "29.3"
+  :type 'regexp)
+
 (defcustom tramp-operation-not-permitted-regexp
   (rx (| (: "preserving times" (* nonl)) "set mode") ":" (* blank)
   "Operation not permitted")
@@ -5589,7 +5596,7 @@ of."
 	  prompt)
   (goto-char (point-min))
   (tramp-check-for-regexp proc tramp-process-action-regexp)
-  (setq prompt (concat (match-string 1) " "))
+  (setq prompt (concat (string-trim (match-string 1)) " "))
   (tramp-message vec 3 "Sending %s" (match-string 1))
   ;; We don't call `tramp-send-string' in order to hide the
   ;; password from the debug buffer and the traces.
@@ -5665,14 +5672,17 @@ Wait, until the connection buffer changes."
   (ignore set-message-function clear-message-function)
   (tramp-message vec 6 "\n%s" (buffer-string))
   (tramp-check-for-regexp proc tramp-process-action-regexp)
-  (with-temp-message
-	  (replace-regexp-in-string (rx (any "\r\n")) "" (match-string 0))
+  (with-temp-message (concat (string-trim (match-string 0)) " ")
 	;; Hide message in buffer.
 	(narrow-to-region (point-max) (point-max))
 	;; Wait for new output.
 	(while (not (tramp-compat-ignore-error file-error
 		  (tramp-wait-for-regexp
-		   proc 0.1 tramp-security-key-confirmed-regexp)))
+		   proc 0.1
+		   (tramp-compat-rx
+			(| (regexp tramp-security-key-confirmed-regexp)
+			   (regexp tramp-security-key-pin-regexp)
+			   (regexp tramp-security-key-timeout-regexp))
 	  (when (tramp-check-for-regexp proc tramp-security-key-timeout-regexp)
 	(throw 'tramp-action 'timeout))
 	  (redisplay 'force)))
@@ -6726,12 +6736,13 @@ Consults the auth-source package."
 		   (tramp-get-connection-property key "login-as")))
 	 (host (tramp-file-name-host-port vec))
 	 (pw-prompt
-	  (or prompt
-	  (with-current-buffer (process-buffer proc)
-		(tramp-check-for-regexp proc tramp-password-prompt-regexp)
-		(if (string-match-p "passphrase" (match-string 1))
-		(match-string 0)
-		  (format "%s for %s " (capitalize (match-string 1)) key)
+	  (string-trim-left
+	   (or prompt
+	   (with-current-buffer (process-buffer proc)
+		 (tramp-check-for-regexp proc tramp-password-prompt-regexp)
+		 (if (string-match-p "passphrase" (match-string 1))
+		 (match-string 0)
+		   (format "%s for %s " (capitalize (match-string 1)) key))
 	 (auth-source-creation-prompts `((secret . ,pw-prompt)))
 	 ;; Use connection-local value.
 	 (auth-sources (buffer-local-value 'auth-sources (process-buffer proc)))

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]


Von meinem iPad gesendet

> Am 30.01.2024 um 16:08 schrieb coder.nov...@mailbox.org:
> Also to add, if I disable x11-sshaskpass, because of that error still 
> showing in the buffer, I am unable to enter the pin and it makes Tramp and 
> Emacs lock up
> 
>> On 01/30/2024 8:21 AM CST coder.nov...@mailbox.org wrote:
>> Hey, I'm so sorry for the delay again! It took me a bit to get the patch 
>> added (since I'm using NixOS along with Doom Emacs, had to update Emacs to 
>> 29.2 and such). I gave it a test and it still says that error in the 
>> messenger buffer; but it allows me to enter the PIN (I have x11-sshaskpass) 
>> and then after I enter the pin and touch the key, Tramp is able to enter 
>> into the remote filesystem like it's suppose to!

Please send me traces with verbosity 10. It might take time to reply, I‘m ill  
these days.

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]

coder.nov...@mailbox.org writes:

> Sorry for the delay! Here's the output
>
> ssh server ...

Thanks, that helps.

Tramp 2.6.2 (Emacs 29.2) supports already OTP input, as it would be
appropriate for the PIN. However, it doesn't know the PIN prompt as used
by Yubikey yet.

Could you pls install first Tramp 2.6.2.0 from GNU ELPA? This is a
little bit problematic, because Emacs 29.1 has a bug which prevents
installation of Tramp from GNU ELPA. See 

for mitigation instructions.

On top, pls install this patch:

diff --git a/lisp/tramp.el b/lisp/tramp.el
index bd556753..a97d8e37 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -688,7 +688,7 @@ The `sudo' program appears to insert a `^@' character into the prompt."
 (defcustom tramp-otp-password-prompt-regexp
   (rx bol (* nonl)
   ;; JumpCloud.
-  (group (| "Verification code"))
+  (group (| "Verification code" "Enter PIN"))
   (* nonl) (any ":：៖") (* blank))
   "Regexp matching one-time password prompts.
 The regexp should match at end of buffer."

After recompilation of tramp.el, it should work. Unfortunately, I cannot
test locally, so I'm waiting for your response.

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Tramp crashes when trying to authenticate ssh with yubikey resident-key that requires user to enter FIDO2 PIN

[Michael Albinus]

coder.nov...@mailbox.org writes:

Hi,

> I'm using my Yubikey 5 to authenticate ssh on my server. The resident
> key I created for it requires both the FIDO2 PIN and touching the key
> to authenticate it. When I try to log into the server with Tramp, the
> program immediately crashes with `File error: [[Regexp '^^M*User
> presence confirmed [^M]*' not found in 0 secs]]`; and Emacs also
> completely locks up as a result of Tramps failing. My computer's
> running NixOS and I have Doom Emacs installed.

Tramp supports already the Yubikey presence message, but it doesn't
support the Yubikey PIN input. Must be added.

Could you, pls, run ssh in a terminal, and show all messages which
happen there (except the PIN value, of course)?

Best regards, Michael.

Re: Unexpected "recover" from non-existing tramp-login-program

[Michael Albinus]

Tim Landscheidt  writes:

Hi Tim,

>>> while working with docker-tramp 0.1.1 and not setting
>>> docker-tramp-docker-executable correctly on a Fedora system,
>
>> First of all, docker-tramp from M(?)ELPA is deprecated. Since Tramp 2.6
>> (Eamcs 29), Tramp has an own implementation for docker containers.
>
>> This offers also the method "podman" which you might need on Fedora.
>
> I tend to stick with distribution packages which for Fedo-
> ra 38 means Emacs 28.3.  docker-tramp works very nicely with
> that and covers all of my needs.

Sure.

>> Well, we could do something like this. Does this work for you?
>
> Yes, this gives a clear error indication.  Thanks!

Thanks for the feedback, I've pushed this to the repositories. You'll
see it with the next Emacs release, either 29.3 or 30.1.

> Tim

Best regards, Michael.

Re: Multi-hop completion with TRAMP 2.5.4-pre/Emacs 28.3?

[Michael Albinus]

Tim Landscheidt  writes:

> Hi,

Hi Tim,

> lisp/net/tramp-container.el appears to have code introduced
> in Emacs 29.1 (?) that allows for multi-hop completion.

No, it is added to Tramp 2.7 (Emacs 30). See new user option
tramp-completion-multi-hop-methods, which tells for which methods this
is implemented. ATM, its value is '("flatpak" "toolbox" "kubernetes" "podman" 
"docker")

> Is multi-hop completion not possible with TRAMP
> 2.5.4-pre/Emacs 28.3?

Yes, it isn't. But you can install Tramp 2.7 and use it with of Emacs 27
or later.

> Tim

Best regards, Michael.

Re: Unexpected "recover" from non-existing tramp-login-program

[Michael Albinus]

Tim Landscheidt  writes:

> Hi,

Hi Tim,

> while working with docker-tramp 0.1.1 and not setting
> docker-tramp-docker-executable correctly on a Fedora system,

First of all, docker-tramp from M(?)ELPA is deprecated. Since Tramp 2.6
(Eamcs 29), Tramp has an own implementation for docker containers.

This offers also the method "podman" which you might need on Fedora.

> I noticed this odd behaviour:
>
> | ELISP> (add-to-list 'tramp-methods
> | '("test-dne"
> |   (tramp-login-program "does-not-exist")
> |   (tramp-login-args (nil ("exec" "-it") ("-u" "%u") 
> ("%h") ("sh")))
> |   (tramp-remote-shell "/bin/sh")
> |   (tramp-remote-shell-args ("-i" "-c"
> | (("test-dne"
> |   (tramp-login-program "does-not-exist")
> |   (tramp-login-args
> |(nil
> | ("exec" "-it")
> | ("-u" "%u")
> | ("%h")
> | ("sh")))
> |   (tramp-remote-shell "/bin/sh")
> |   (tramp-remote-shell-args
> |("-i" "-c")))
> |  […])
>
> | ELISP> (find-file "/test-dne:does-not-exist-either:/etc/passwd")
> | #
> | ELISP>
>
> The resulting buffer will contain the contents of
> /etc/passwd on the primary system (well, there are no
> others :-)).

I've tried this with Emacs 29.2 (Tramp 2.6.2), and the result is
slightly better: it is an empty buffer #, and there is
the error message "Use M-x make-directory RET RET to create the
directory and its parents". Default directory is
"/test-dne:does-not-exist-either:/etc/", which is correct.

> My assumption is that technically this is not an error as
> with the great power that Emacs gives to the user comes
> great responsibility to set one's variables properly :-).

Indeed, as always with Emacs :-)

> I have not debugged where this behaviour is rooted and
> whether it can be "fixed" easily, but if TRAMP can detect
> that the call to tramp-login-program fails, it would be nice
> if it could croak.

Well, we could do something like this. Does this work for you?

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 38925652..1301cd63 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -5340,7 +5340,7 @@ connection if a previous connection has died for some reason."
 			"2>" (tramp-get-remote-null-device previous-hop))
 			?l (concat remote-shell " " extra-args " -i"))
 		   ;; A restricted shell does not allow "exec".
-		   (when r-shell '("&&" "exit" "||" "exit")))
+		   (when r-shell '("&&" "exit")) '("||" "exit"))
 		  " "))

 		;; Send the command.

> Tim

Best regards, Michael.

Re: tramp-parse-passwd does not seem to work if getent is available?

[Michael Albinus]

Tim Landscheidt  writes:

> Hi,

Hi Tim,

Thanks for the report!

> Looking at the code:
>
> | (defun tramp-parse-passwd (filename)
> |   "Return a list of (user host) tuples allowed to access.
> | Host is always \"localhost\"."
> |   (with-tramp-connection-property nil "parse-passwd"
> | (if (executable-find "getent")
> | (with-temp-buffer
> |   (when (zerop (tramp-call-process nil "getent" nil t nil "passwd"))
> | (goto-char (point-min))
> | (cl-loop while (not (eobp)) collect
> |  (tramp-parse-etc-group-group
> |   (tramp-parse-file filename #'tramp-parse-passwd-group
>
> it is not clear to me why the output of "getent passwd" and
> the contents of /etc/passwd are parsed differently.
> Shouldn't the output of "getent passwd" (and its other vari-
> ants) be of the same structure as /etc/passwd?

You are right, it is a nasty cut'n'waste error. Of course,
tramp-parse-passwd-group must be used in both cases.

I've fixed this in the repositories. Will be available on GNU ELPA as
Tramp 2.6.2.1 later this month.

> Tim

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Hi Michael,

Hi Kirill,

>> tramp-sh.el is loaded dynamically. It overrides your patch. So rather call
>>
>> --8<---cut here---start->8---
>> emacs -Q -l tramp -l tramp-sh -l 
>> ~/Downloads/patched-tramp-bundle-read-file-names.el 
>> /ssh:playground.island.local:
>> --8<---cut here---end--->8---
>
> Now it works as expected:
>
>> Tramp: Opening connection nil for playground.island.local using ssh...failed
>> File error: /bin/sh: can't create temporary file /tmp/shKhv3tHXB: Permission 
>> denied
>>
>> signal: /bin/sh: can't create temporary file /tmp/shKhv3tHXB: Permission 
>> denied

Uff, we've got it :-) Thanks a lot for your patient testing!

I'll push the patch to the Tramp and Emacs repositories, master branch.

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Hi Michael,

Hi Kirill,

> When I run:
>
>> emacs -Q -l tramp --eval '(load-file 
>> "~/Downloads/patched-tramp-bundle-read-file-names.el")' 
>> /ssh:playground.island.local:

tramp-sh.el is loaded dynamically. It overrides your patch. So rather call

--8<---cut here---start->8---
emacs -Q -l tramp -l tramp-sh -l 
~/Downloads/patched-tramp-bundle-read-file-names.el 
/ssh:playground.island.local:
--8<---cut here---end--->8---

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

Hi Kirill,
>> I don't understand. Does this happen when remote /tmp is not writable?
>> Tramp shall show you the error message, and cease to work - at least
>> that's the intention of the patch.
>
> After your patch it works without any error in Message buffer.
>
> Works mean that I can open files via tramp.

I'm confused ...

Could you please start as before

--8<---cut here---start->8---
# emacs -l tramp --eval '(setq tramp-verbose 10)' /ssh:playground.island.local:
--8<---cut here---end--->8---

and show the debug buffer? I *want* to expose the error to the user :-)

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Hi Michael,

Hi Kirill,

>> On 8. Jan 2024, at 11:37, Michael Albinus  wrote:
>> 
>> The error "/bin/sh: can't create temporary file /tmp/shZPInbtKR:
>> Permission denied" isn't propagated properly by Tramp. Could you,
>> please, try the appended patch? There shall be a better error message.
>
> Thanks for the patch.
>
> Now I can open any file on that machine. From *Messages* buffer:
>
> Tramp: Opening connection nil for island.catap.net using ssh...done
> Making completion list...
> Tramp: Inserting ‘/ssh:island.catap.net:/home/catap/.forward’...done
> 
> Tramp: Checking ‘vc-registered’ for 
> /ssh:island.catap.net:/home/catap/.forward...done
> Tramp: Inserting ‘/ssh:island.catap.net:/home/catap/.forward’...done

I don't understand. Does this happen when remote /tmp is not writable?
Tramp shall show you the error message, and cease to work - at least
that's the intention of the patch.

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Greetings,

Hi Kirill,

> When I run emacs your way I have:
>
> In *Messages*
>> For information about GNU Emacs and the GNU system, type C-h C-a.
>> Tramp: Opening connection nil for playground.island.local using ssh...failed
>> signal: Wrong type argument: "listp /bin/sh:"
>
> and I've attached tramp.log

Thanks a lot, this has helped. In the debug buffer we see what happens:

--8<---cut here---start->8---
22:48:20.976768 tramp-send-command (6) # tramp_bundle_read_file_names 
<<'36584af1a5124300cf63739980e7d186'
/usr/bin
/bin
/usr/sbin
/sbin
/usr/X11R6/bin
/usr/local/bin
/usr/local/sbin
/local/bin
/local/freeware/bin
/local/gnu/bin
/usr/freeware/bin
/usr/pkg/bin
/usr/contrib/bin
/opt/bin
/opt/sbin
/opt/local/bin
36584af1a5124300cf63739980e7d186
22:48:21.035464 tramp-wait-for-regexp (6) #
/bin/sh: can't create temporary file /tmp/shZPInbtKR: Permission denied
///8056ae9d33677d9550fbd457130ab698#$
22:48:21.049964 tramp-bundle-read-file-names (10) #
  backtrace()
  tramp-signal-hook-function(wrong-type-argument (listp /bin/sh:))
  tramp-bundle-read-file-names((tramp-file-name "ssh" nil nil 
"playground.island.local" nil "" nil) ("/usr/bin" "/bin" "/usr/sbin" "/sbin" 
"/usr/X11R6/bin" "/usr/local/bin" "/usr/local/sbin" "/local/bin" 
"/local/freeware/bin" "/local/gnu/bin" "/usr/freeware/bin" "/usr/pkg/bin" 
"/usr/contrib/bin" "/opt/bin" "/opt/sbin" "/opt/local/bin"))
  ...
--8<---cut here---end--->8---

The error "/bin/sh: can't create temporary file /tmp/shZPInbtKR:
Permission denied" isn't propagated properly by Tramp. Could you,
please, try the appended patch? There shall be a better error message.

> wbr, Kirill

Best regards, Michael.

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 6489f473..8ec9467a 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -3652,20 +3652,20 @@ filled are described in `tramp-bundle-read-file-names'."

 (dolist
 	(elt
-	 (ignore-errors
+	 (with-current-buffer (tramp-get-connection-buffer vec)
 	   ;; We cannot use `tramp-send-command-and-read', because
 	   ;; this does not cooperate well with heredoc documents.
-	   (tramp-send-command
-	vec
-	(format
-	 "tramp_bundle_read_file_names <<'%s'\n%s\n%s\n"
-	 tramp-end-of-heredoc
-	 (mapconcat #'tramp-shell-quote-argument files "\n")
-	 tramp-end-of-heredoc))
-	   (with-current-buffer (tramp-get-connection-buffer vec)
-	 ;; Read the expression.
-	 (goto-char (point-min))
-	 (read (current-buffer)
+	   (unless (tramp-send-command-and-check
+		vec
+		(format
+		 "tramp_bundle_read_file_names <<'%s'\n%s\n%s\n"
+		 tramp-end-of-heredoc
+		 (mapconcat #'tramp-shell-quote-argument files "\n")
+		 tramp-end-of-heredoc))
+	 (tramp-error vec 'file-error "%s" (tramp-get-buffer-string)))
+	   ;; Read the expression.
+	   (goto-char (point-min))
+	   (read (current-buffer

   (tramp-set-file-property vec (car elt) "file-exists-p" (nth 1 elt))
   (tramp-set-file-property vec (car elt) "file-readable-p" (nth 2 elt))

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Hi Michael,

Hi Kirill,

> I've achieved that on OpenBSD where I've switched /tmp to ramdisk.
>
> It uses so-called mfs driver which inherits permissions from the
> mountpoint.

That sounds a little bit too expansive for my test environment.

I've tried to reproduce the problem with setting the
remote-temporary-file-directory to /home/admin/tmp w/o write
permissions. Then I've started

--8<---cut here---start->8---
emacs --eval '(add-to-list (quote tramp-connection-properties) (list nil 
"tmpdir" "/home/admin/tmp"))' /ssh:ubuntu-2204:
--8<---cut here---end--->8---

Which tells Tramp to use "/home/admin/tmp" instead of "/tmp" on the
remote side. Tramp did connect w/o problem, meaning that it didn't try
to write to "/home/admin/tmp". When copying a local file to the remote
host in Emacs, there is the error

--8<---cut here---start->8---
Copying /home/albinus/123 to /ssh:ubuntu-2204:/home/admin/123...failed
File error: Directory /ssh:ubuntu-2204:/home/admin/tmp not accessible
--8<---cut here---end--->8---

This is expected, because Tramp needs to create a temporary file for
this, which doesn't work. And the check inside Tramp works.

That indicates, that the error in your case, writing to an improper /tmp
directory, isn't caused by Tramp directly, but rather by an external
program Tramp runs on the remote side behind the scenes. Could you
please verify this? Remove write permissions from the remote /tmp
directory, and start Emacs like this:

--8<---cut here---start->8---
# emacs -l tramp --eval '(setq tramp-verbose 10)' /ssh:some.host:
--8<---cut here---end--->8---

The error should appear, and there should be a Tramp debug buffer. This
might tell us more. If you fear to uncover sensible information with the
debug buffer, send it directly to me.

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Greetings,

Hi Kirill,

> I can easy reproduce this error by remove write access to /tmp folder on 
> remote host like that:
>
> ssh some.host sudo chmod -w /tmp

Which kind of remote machine do you use? I cannot apply this, neither on
a remote Ubuntu, a remote Fedora, or a remote FreeBSD machine:

--8<---cut here---start->8---
# ssh -t -t ubuntu-2204 sudo chmod -w /tmp
[sudo] password for admin:
chmod: /tmp: new permissions are r-xrwxrwt, not r-xr-xr-t
Connection to ubuntu-2204.fritz.box closed.
--8<---cut here---end--->8---

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

Hi Kirill,

>>> Meanwhile I was able to localise the issue: permission to /tmp on server 
>>> side.
>>> As soon as I fixed it and my user may write into it, everything works fine.
>>
>> Good to know there's no bug. Thanks for letting us know.
>
> I really think that error message should be a bit more different that this 
> one.
> It was quite tricky to ping point the issue.

The error message came from somewhere else, it was a follow-up error. So
it couldn't speak about the real problem.

But I don't understand yet how this could happen. Tramp has the function
tramp-handle-temporary-file-directory, which returns a proper remote
temporary directory, or nil if there isn't. And this includes the
file-writable-p check. Hmm.

I'll dig further. First, I need a simple recipe for error reproduction.

> wbr, Kirill

Best regards, Michael.

Re: Tramp 2.7.0

[Michael Albinus]

Jim Porter  writes:

Hi Jim,

>> The Tramp team is happy to announce the release of Tramp 2.7.0.
>
> I noticed that this version doesn't seem to be on GNU ELPA (the latest
> version I see is Tramp 2.6.2.0). I suppose that's intentional, since
> the announcement for 2.6.2 mentions ELPA, and this message doesn't.

Indeed. The initial intention was to bring Tramp 2.7.0 to GNU ELPA. This
decision was changed then.

However, there is now the Tramp 2.7.0 tarball. People who prefer the
usual ./configure; make; make install triad can install it.

> Out of curiosity, is there a reason for this? (In particular, I'm
> wondering if there's some missing feature in ELPA or package.el that
> would make it easier to distribute both versions via ELPA...)

Tramp 2.7.0 is complete, there's nothing missing. There might be new
features until Emacs 30 is released, but that's another story.

We still have the problem, that Tramp cannot be installed from GNU ELPA
with Emacs 29.1 out-of-the-box. So I decided I'll wait for the Emacs
29.2 release. If this is available (and if this has hit major Linux
distributions, like Ubuntu or Fedora), Tramp 2.7 will be released on GNU
ELPA. Although it could be problematic for the Linux distributions to
get Emacs 29.2 in time (April 2024), so this criterium might be
revisited.

> - Jim

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Greeting,

Hi Kirill,

> Meanwhile I was able to localise the issue: permission to /tmp on server side.
>
> As soon as I fixed it and my user may write into it, everything works fine.

Good to know there's no bug. Thanks for letting us know.

> wbr, Kirill

Best regards, Michael.

Re: tramp (2.7.0-pre nil/nil); Can't connect to server anymore

[Michael Albinus]

"Kirill A. Korinsky"  writes:

> Greetings,

Hi Kirill,

> I'd like to report that I can't connect to the same server that I've
> used for a couple of days.
>
> An attempt to read anything from via C-c f /ssh:tramp-test-server: fails
> as below. I've created unique record at /etc/hosts for this server to
> prevent anything from affect it. Thus, this report was created when I've
> run emacs -q, so it isn't something my settings specified.
>
> I've tried to remove .tramp_history on remote server without of any
> sucess, and the last things: the output of failed comannad:
>
> /home/catap
> tramp_exit_status 0

Could you please try "emacs -Q /ssh:tramp-test-server:" instead? If this
also fails, try "emacs -Q --eval '(setq tramp-verbose 10)' 
/ssh:tramp-test-server:"

There will be a debug buffer *debug tramp/ssh tramp-test-server* which
should tell us more.

> wbr, Kirill

Best regards, Michael.

Tramp 2.7.0

[Michael Albinus]

Hello,

The Tramp team is happy to announce the release of Tramp 2.7.0.

Tramp is a remote file editing package for Emacs.  Tramp uses different
methods like ssh or scp to access files on remote hosts as if they were
local files.  Access to the remote file system for editing files,
version control, directory editing and running processes on the remote
host are transparently enabled.

Tramp 2.7.0 supports Emacs since version 27.1.  This is the version
which will be integrated with the upcomind Emacs 30.

The download files are located at .  Tramp's
homepage is .

Best regards, Michael.

Tramp 2.6.2

[Michael Albinus]

Hello,

The Tramp team is happy to announce the release of Tramp 2.6.2.

Tramp is a remote file editing package for Emacs.  Tramp uses different
methods like ssh or scp to access files on remote hosts as if they were
local files.  Access to the remote file system for editing files,
version control, directory editing and running processes on the remote
host are transparently enabled.

Tramp 2.6.2 supports Emacs since version 26.1.  This is the version
integrated with the upcoming Emacs 29.2.

The download files are located at .

Tramp 2.6.2 is also available via GNU ELPA.

Best regards, Michael.

Re: Is there any way to make direct-async-process support tramp-remote-path?

[Michael Albinus]

Liu Bo  writes:

> Hi,

Hi,

> I found direct-async-process really make tramp faster(especially
> magit), but since it dose not use tramp-remote-path so I can't make
> command that need extra programs work, like eglot or
> consult-ripgrep(says command not found).
>
> So is there any way to make direct-async-process support tramp-remote-path?

I've pushed a change to the Tramp and Emacs master repositories, which
implements this. Until now it is enabled by default, let's see whether
people complain.

Could you try to test it?

Best regards, Michael.

Re: Is there any way to make direct-async-process support tramp-remote-path?

[Michael Albinus]

Liu Bo  writes:

Hi,

> Hi, I found direct-async-process really make tramp faster(especially
> magit), but since it dose not use tramp-remote-path so I can't make
> command that need extra programs work, like eglot or
> consult-ripgrep(says command not found).
>
> So is there any way to make direct-async-process support tramp-remote-path?

Not directly, because it calls the program as part of the ssh
invocation, like (example from magit)

--8<---cut here---start->8---
ssh -q -l albinus -o ControlMaster=auto -o 
ControlPath=/home/albinus/.cache/emacs/tramp.%C -o ControlPersist=no -e none 
gandalf cd /home/albinus/src/emacs/ && ( env 
INSIDE_EMACS\=30.0.50\,magit\,tramp\:2.7.0-pre ENV\=\'\' TMOUT\=0 
LC_CTYPE\=\'\' CDPATH\= HISTORY\= MAIL\= MAILCHECK\= MAILPATH\= PAGER\=cat 
autocorrect\= correct\= git --no-pager --literal-pathspecs -c 
core.preloadindex\=true -c log.showSignature\=false -c color.ui\=false -c 
color.diff\=false apply --reverse --cached -p0 --ignore-space-change - )
--8<---cut here---end--->8---

But perhaps we could add a user option, which instruct Tramp to expand
tramp-remote-path as part of the command, like PATH=/a/b:/c/d. I will
think about.

Bst regards, Michael.

Re: tramp (2.7.0-pre master/463cd87f0590970cfd2918ba7028304e0f712001); tramp-signal-process doesn't accept string arguments

[Michael Albinus]

Philipp Stephani  writes:

Hi Philipp,

> That works in all cases I've tested, thanks!

Thanks for the feedback. I've pushed it to the Tramp repository,
branches tramp-2-6-stable and master. It will appear with Tramp 2.6.2 on
GNU ELPA, later this year.

I've pushed it also to the Emacs master branch.

On the emacs-29 branch, I've pushed an extension for the signal-process
doc, saying that PROCESS can also be a string.

Best regards, Michael.

Re: tramp (2.7.0-pre master/463cd87f0590970cfd2918ba7028304e0f712001); tramp-signal-process doesn't accept string arguments

[Michael Albinus]

Philipp Stephani  writes:

Hi Philipp,

> Almost, but one case is still missing.  Try:
>
> emacs -Q -batch -eval '(progn (require (quote tramp)) (let ((proc 
> (start-process "sleep" nil "sleep" "60"))) (signal-process (number-to-string 
> (process-id proc)) (quote ABRT'
>
> This will fail even with the patch.
> `signal-process' accepts a process ID converted to a string.  This doesn't 
> seem to be documented, but is necessary for interactive use because 
> `signal-process' uses `read-string' to read its first argument interactively.

Yes. What about the modified patch?

Best regards, Michael.

diff --git a/lisp/tramp.el b/lisp/tramp.el
index 8a5c9617..5b1a4f84 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -6729,6 +6729,11 @@ If PROCESS is a process object which contains the property
 PROCESS is interpreted as process on the respective remote host, which
 will be the process to signal.
 SIGCODE may be an integer, or a symbol whose name is a signal name."
+  (when (stringp process)
+(setq process (or (get-process process)
+		  (and (string-match-p (rx bol (+ digit) eol) process)
+			   (string-to-number process))
+		  (signal 'wrong-type-argument (list #'processp process)
   (let (pid vec)
 (cond
  ((processp process)

Re: tramp (2.7.0-pre master/463cd87f0590970cfd2918ba7028304e0f712001); tramp-signal-process doesn't accept string arguments

[Michael Albinus]

Philipp  writes:

Hi Philipp,

> The function `tramp-signal-process' should accept a process name as a
> string (to be converted to a process via `get-process') or a process ID
> converted to a string, as documented for `signal-process'.

Yep. What about the appended patch?

Best regards, Michael.

diff --git a/lisp/tramp.el b/lisp/tramp.el
index 8a5c9617..c515e305 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -6732,9 +6733,9 @@ will be the process to signal.
 SIGCODE may be an integer, or a symbol whose name is a signal name."
   (let (pid vec)
 (cond
- ((processp process)
-  (setq pid (process-get process 'remote-pid)
-vec (process-get process 'tramp-vector)))
+ ((or (processp process) (stringp process))
+  (setq pid (process-get (get-process process) 'remote-pid)
+vec (process-get (get-process process) 'tramp-vector)))
  ((numberp process)
   (setq pid process
 vec (and (stringp remote) (tramp-dissect-file-name remote

Re: tramp (HEAD/b20b7910b413c13352213d37cac8e948ad554dc3); CR-LF woes again — PATCH ATTACHED

[Michael Albinus]

Dominique Quatravaux  writes:

Hi Dominique,

> --8<---cut here---start->8---
> (when (search-forward-regexp (rx (not space))
> (line-end-position) t)
> --8<---cut here---end--->8---
>
> Could you, please, test in your environment?
>
> Thanks Michael, `(not space)` works as well.

Thanks for the feedback. I've pushed the change to the repositories.

> Yours truly,
>
>  Dominique Quatravaux

Best regards, Michael.

Re: tramp (HEAD/b20b7910b413c13352213d37cac8e948ad554dc3); CR-LF woes again — PATCH ATTACHED

[Michael Albinus]

Dominique Quatravaux  writes:

Hi Dominique,

> When accessing some file over the /kubernetes: Tramp connection type,
> browsing goes sideways the instant I am done typing in the “hostname”
> (i.e. in this case, the pod name). It appears that for some reason or
> other, the CR/LF situation is being handled poorly.

Yes. This happens, because Tramp tries do read something early (during
completion), when the CRLF handling hasn't been established yet. I believe.

> diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
> index 186ef127..769138dc 100644
> --- a/lisp/tramp-sh.el
> +++ b/lisp/tramp-sh.el
> @@ -5540,7 +5540,7 @@ raises an error."
>(unless noerror signal-hook-function)))
>   (read (current-buffer)))
>   ;; Error handling.
> - (when (search-forward-regexp (rx (not blank)) (line-end-position) t)
> + (when (search-forward-regexp (rx (not (or "\r" blank))) 
> (line-end-position) t)
> (error nil)))
>   (error (unless noerror
>(tramp-error

Thanks for the patch, it looks fine. However, perhaps it is more simple
to use 'space' instead of 'blank', this includes also characters like
CR, LF, FF, you name it. For all characters with whitespace syntax, see
.

--8<---cut here---start->8---
(when (search-forward-regexp (rx (not space)) (line-end-position) t)
--8<---cut here---end--->8---

Could you, please, test in your environment?

>  Dominique Quatravaux

Best regards, Michael.

Re: Orderless matches all TRAMP methods

[Michael Albinus]

Michael Albinus  writes:

Hi Peter,

>> I experience the following problem when trying to complete local filenames 
>> when TRAMP is loaded.
>>
>> This isn’t correct, since none of these match "/zzz".
>
> Thanks, I can reproduce it. I'll debug next days (I'm not so fluent with
> orderless, so I need some initia first).

I've pushed a fix to the emacs-29 branch, could you pls check?

It will also be available with the next GNU ELPA release, Tramp 2.6.1.5,
later this month.

Best regards, Michael.

Re: Orderless matches all TRAMP methods

[Michael Albinus]

Peter Oliver  writes:

Hi Peter,

> I experience the following problem when trying to complete local filenames 
> when TRAMP is loaded.
>
> In "emacs-29.1.90 -Q", I run:
>
> (package-initialize)
> (require 'orderless)
> (setq completion-styles '(orderless))
>
> Then, I type:
>
> C-x C-f C-a C-k /ssh: C-a C-k /zzz TAB
>
> I see:
>
> Click on a completion to select it.
> In this buffer, type RET to select the completion near point.
>
> 45 possible completions:
> -: adb:   afp:   dav:   davs:
> doas:  docker:fcp:   ftp:   gdrive:
> krlogin:   ksu:   kubernetes:mtp:   nc:
> nextcloud: plink-ksh: plink: plinkx:podman:
> pscp:  psftp: rclone:rcp:   remcp:
> remsh: rsh:   rsync: scp-ksh:   scp:
> scpx:  sftp:  sg:smb:   ssh-ksh:
> ssh:   sshfs: sshx:  su:sudo-csw-ksh:
> sudo-csw:  sudo-ksh:  sudo:  sudoedit:  telnet:
>
> This isn’t correct, since none of these match "/zzz".

Thanks, I can reproduce it. I'll debug next days (I'm not so fluent with
orderless, so I need some initia first).

Perhaps it is a good idea to write it also as Emacs bug report. This
would make it easier to add a possible solution to Emacs 29.2, which is
in its pretest already.

Best regards, Michael.

Re: tramp (2.6.0.29.1) : incorrect parsing of group with "-" in name

[Michael Albinus]

brickviking  writes:

Hi Gaël & brickviking,

> After tinkering with this, I can confirm the initial bug, which
> affects me much like it does Gael. My user is a member of groups, one
> of which has a hyphen in it.
> /bin/id:
> uid=1001(viking)  gid=1001(viking) groups=1001(viking),27(sudo),29
> (audio),44(video),119(pulse),120(pulse-access),1000(zoombuggy)
> Inside the debug buffer for tramp: 
> ...
> groups-integer: (1001 27 29 44 119)
> groups-string: (viking sudo audio video pulse)
> ...
>
> Local system is 64-bit Fedora 29, with emacs-29 (git revision:
> aa253c533d23d0fda1ecc512d94dea24501803ed). tramp version: 2.6.2.29.2
> Remote system is 64-bit Ubuntu 22.04.

Thanks for the report. I've fixed this in tramp.el, and pushed it to the
Emacs and Tramp repositories. It will be available with Emacs 29.2 as
well as with Tramp 2.6.1.4 (to be released in a couple of days on GNU
ELPA).

> Regards, brickviking

Best regards, Michael.

Re: Connection local profiles uses default instead of created one

[Michael Albinus]

Simon Chevolleau  writes:

> Dear TRAMP community,

Hi Simon,

> I'm trying to figure out how to set connection local profiles properly.
> The problem is, when connecting through ssh using TRAMP, it never uses
> the profile I want.

When a connection-local variable is set in different profiles (as you do
with `shell-file-name', it depends on the order the profiles are taken
from connection-local-criteria-alist. There is always a setting for that
variable which is the last one, and it wins.

The Elisp manual doesn't discuss this case, so I don't know the order
such entries do apply.

> - connection-local-criteria-alist is a variable defined in
> (((:application tramp :protocol "ssh" :user "username" :machine "remote")
> remote-bird)
> ((:protocol "ssh")
> remote-bird)
> ((:application tramp)
> tramp-connection-local-default-system-profile 
> tramp-connection-local-default-shell-profile))
>
> I don't understand why the remote-bird profile is not used with that set up.
> Anything I could do to make it work ?

You have two entries which use the remote-bird profile. One entry is
enough, remove the entry with the criteria '(:protocol "ssh").

Try to reorder then entries in `connection-local-criteria-alist'.

> Best regards,
> Simon

Best regards, Michael.

Re: Installing TRAMP with package.el fails on startup

[Michael Albinus]

Martin Kjær Jørgensen  writes:

> Hey,

Hi Martin,

> I'm using 'GNU Emacs 29.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
> cairo version 1.16.0) of 2023-08-30, modified by Debian' with ...
>
> (setq package-install-upgrade-built-in t)
>
> ... set in order to update built-in packages. Updating other build-in pacakges
> seems to work, except tramp (2.6.1.3), which installs without errors, but
> produces errors on emacs startup, like:

This is a known bug of Emacs 29.1, fixed in the upcoming 29.2. A
mitigation recipe for installing Tramp from GNU ELPA is in
~/.emacs.d/elpa/tramp-2.6.1.3/README

> /Martin

Sorry for the inconvenience.

Best regards, Michael.

Re: Strange behavior of newline code

[Michael Albinus]

Takahiro OHKUBO  writes:

Hi Takahiro,

> I have updated Emacs from 28.2 to 29.1 and when editing a file
> with unix newline code using tramp via ssh, the newline code is
> forced to be "^M". This behavior was not seen in Emacs 28.2.

This sounds like bug#64164 / bug#65022. It will be fixed in Emacs 29.2.

For the time being, you could install Tramp 2.6.1.3 from GNU ELPA, which
contains also the fix. Note that there is a bug in Emacs 29.1 which
prevents this installation, read 
how to install it nevertheless.

> Takahiro

Best regards, Michael.

Re: tramp (2.6.0-pre master/b30b33ed9b3cdacecebef73ad1131f03c635de7a); tramp over MFA/totp

[Michael Albinus]

Randy Yates  writes:

> Michael,

Hi Randy,

> Thank you. The patch worked perfectly.

Thanks for the feedback. I've pushed the patch, slightly changed and
added documentation, to the repositories. Will appear with Emacs
29.2. Will also be part of the next GNU ELPA Tramp release, scheduled
later this month.

> –Randy

Best regards, Michael.

Re: tramp (2.6.0-pre master/b30b33ed9b3cdacecebef73ad1131f03c635de7a); tramp over MFA/totp

[Michael Albinus]

Randy Yates  writes:

Hi Randy,

> My organization has recently enabled MFA/totp over ssh on all our linux
> systems. We are using JumpCloud and it was apparently configured by
> JumpCloud automatically when they turned MFA on.
>
> I am able to ssh from one linux box (ubuntu 18.04) to another using the
> command-line ssh (openssh). The transaction looks like this:
>
> @Lubuntu-22259:/$ ssh -YC @
> Password:
> Verification code: 

[...]

> I found that when I use tramp, it prompts me for the password as usual,
> but after entering the correct password, it prompts me again for the
> password instead of the verification code.

[...]

> Finally, note that before these security updates, tramp was working
> fine. I am using emacs version
>
> GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 
> 3.22.30, cairo version 1.15.10) of 2022-06-16.
>
> Is it possible to update tramp so that it operates correctly with this
> configuration?

Well, my first reaction was to ask you to add the "verification code"
prompt to password-word-equivalents. This is what Tramp uses for
password prompts.

However, one-time passwords are different. So I've assembled the
appended patch, could you pls test? It is on top of the emacs-29 branch
in git, I hope it applies to your more-than-a-year-old Emacs 29
version. If not, you could install the recent Tramp 2.6.1.2 from GNU
ELPA, and apply the patch on this. Note that there is a problem in
installing Tramp from GNU ELPA with Emacs 29.1, see
.

> Thanks for your help, and let me know if there is anything else you
> need.
>
> Randy Yates

Best regards, Michael.

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 5a1e73aa..59d5c005 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -535,6 +535,7 @@ shell from reading its init file."
 (defconst tramp-actions-before-shell
   '((tramp-login-prompt-regexp tramp-action-login)
 (tramp-password-prompt-regexp tramp-action-password)
+(tramp-otp-password-prompt-regexp tramp-action-otp-password)
 (tramp-wrong-passwd-regexp tramp-action-permission-denied)
 (shell-prompt-pattern tramp-action-succeed)
 (tramp-shell-prompt-pattern tramp-action-succeed)
@@ -558,6 +559,7 @@ corresponding PATTERN matches, the ACTION function is called.")
 
 (defconst tramp-actions-copy-out-of-band
   '((tramp-password-prompt-regexp tramp-action-password)
+(tramp-otp-password-prompt-regexp tramp-action-otp-password)
 (tramp-wrong-passwd-regexp tramp-action-permission-denied)
 (tramp-copy-failed-regexp tramp-action-permission-denied)
 (tramp-security-key-confirm-regexp tramp-action-show-and-confirm-message)
diff --git a/lisp/tramp.el b/lisp/tramp.el
index d10f93b3..83610193 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -679,6 +679,17 @@ The `sudo' program appears to insert a `^@' character into the prompt."
   :version "29.1"
   :type 'regexp)
 
+(defcustom tramp-otp-password-prompt-regexp
+  (tramp-compat-rx
+   bol (* nonl)
+   ;; JumpCloud.
+   (group (| "Verification code"))
+   (* nonl) (any ":：៖") (* blank))
+  "Regexp matching one-time password prompts.
+The regexp should match at end of buffer."
+  :version "29.2"
+  :type 'regexp)
+
 (defcustom tramp-wrong-passwd-regexp
   (rx bol (* nonl)
   (| "Permission denied"
@@ -5538,6 +5549,25 @@ of."
   (narrow-to-region (point-max) (point-max
   t)
 
+(defun tramp-action-otp-password (proc vec)
+  "Query the user for a one-time password."
+  (with-current-buffer (process-buffer proc)
+(let ((case-fold-search t)
+	  prompt)
+  (goto-char (point-min))
+  (tramp-check-for-regexp proc tramp-process-action-regexp)
+  (setq prompt (concat (match-string 1) " "))
+  (tramp-message vec 3 "Sending %s" (match-string 1))
+  ;; We don't call `tramp-send-string' in order to hide the
+  ;; password from the debug buffer and the traces.
+  (process-send-string
+   proc
+   (concat
+	(tramp-read-passwd-without-cache proc prompt) tramp-local-end-of-line))
+  ;; Hide password prompt.
+  (narrow-to-region (point-max) (point-max
+  t)
+
 (defun tramp-action-succeed (_proc _vec)
   "Signal success in finding shell prompt."
   (throw 'tramp-action 'ok))

Re: Reopening ssh file with sudo

[Michael Albinus]

Joseph Garvin  writes:

Hi Joseph,

> That hangs for me too. I set tramp-verbose to 10, then did a C-g, got
> this trace:

You've tried it with the running Emacs instance. Could you pls start a
new Emacs instance like "emacs -Q -l tramp"? Then eval the code I've
sent, and rerun your tests.

If it still doesn't work, describe exactly what you have done, in order
to give me a chance to reproduce the problem.

Best regards, Michael.

Re: Reopening ssh file with sudo

[Michael Albinus]

Joseph Garvin  writes:

Hi Joseph,

> Yeah, I haven't tried automating tramp much before so I'm not super
> familiar with what best practice is here. I'm on 29.1.

OK, so I've used the opportunity to write a more stable command
`tramp-revert-buffer-with-sudo'. Works for buffers visitng a file, and
for dired buffers. Could you pls check?

--8<---cut here---start->8---
(defcustom tramp-file-name-with-method "sudo"
  "Which method to be used in `tramp-file-name-with-sudo'."
  :group 'tramp
  :type '(choice (const "su")
 (const "sudo")
 (const "doas")
 (const "ksu")))

(defun tramp-file-name-with-sudo (filename)
  "Convert FILENAME into a multi-hop file name with \"sudo\".
An alternative method could be chosen with `tramp-file-name-with-method'."
  (setq filename (expand-file-name filename))
  (if (tramp-tramp-file-p filename)
  (with-parsed-tramp-file-name filename nil
(if (and (tramp-multi-hop-p v)
 (not (string-equal method tramp-file-name-with-method)))
(tramp-make-tramp-file-name
 (make-tramp-file-name
  :method (tramp-find-method tramp-file-name-with-method nil host)
  :user (tramp-find-user tramp-file-name-with-method nil host)
  :host (tramp-find-host tramp-file-name-with-method nil host)
  :localname localname :hop (tramp-make-tramp-hop-name v)))
  (tramp-user-error v "Multi-hop with `%s' not applicable" method)))
(tramp-make-tramp-file-name
 (make-tramp-file-name
  :method tramp-file-name-with-method :localname filename

(defun tramp-revert-buffer-with-sudo ()
  "Revert current buffer to visit with \"sudo\" permissions.
An alternative method could be chosen with `tramp-file-name-with-method'.
If the buffer visits a file, the file is replaced.
If the buffer runs `dired', the buffer is reverted."
  (interactive)
  (cond
   ((buffer-file-name)
(find-alternate-file (tramp-file-name-with-sudo (buffer-name
   ((derived-mode-p 'dired-mode)
(setq default-directory (tramp-file-name-with-sudo default-directory)
  list-buffers-directory
  (tramp-file-name-with-sudo list-buffers-directory))
(if (consp dired-directory)
(setcar
 dired-directory (tramp-file-name-with-sudo (car dired-directory)))
  (setq dired-directory (tramp-file-name-with-sudo dired-directory)))
(revert-buffer
--8<---cut here---end--->8---

Best regards, Michael.

Re: Reopening ssh file with sudo

[Michael Albinus]

Joseph Garvin  writes:

Hi Joseph,

> I originally posted this on the emacs stack exchange
> (https://emacs.stackexchange.com/questions/78768/reopen-with-sudo-command-that-works-through-tramp)
> but they suggested I try here.
>
> I have tried to define this command:
>
> ```
> (defun etc-reopen-with-sudo ()
>   (interactive)
>   (let* ((vec (tramp-dissect-file-name (buffer-file-name
> (current-buffer
>(method (tramp-file-name-method vec))
>(user (tramp-file-name-user vec))
>(host (tramp-file-name-host vec))
>(localname (tramp-file-name-localname vec)))
> (message "localname=%s" localname)
> (find-alternate-file
>  (if (tramp-tramp-file-p (buffer-file-name (current-buffer)))
>  (concat (format "/%s:%s@%s|sudo::%s" method user host
> localname))
>(concat "/sudo:root@localhost:" buffer-file-name)
> ```
>
> It worked once... and now ever since mysteriously hangs whenever I try
> to use it to open a file with sudo on a remote host with ssh. The idea
> for what file string to use for this comes from
> https://stackoverflow.com/questions/2177687/open-file-via-ssh-and-sudo-with-emacs
> which suggests `/ssh:you@remotehost|sudo:remotehost:/path/to/file`
> should work. I'm able to visit files on the host just fine without
> sudo. Any ideas what I'm doing wrong here?

Well, your function has some flaws. For example, it doesn't handle the
case there's no user in the buffer-file-name, like in
"/ssh:remotehost:/path/to/file". Furthermore, you use the "sudo::"
construct in the second hop, which works reliably since Tramp 2.4 (Emacs
27.1) only. Which versions do you use?

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

Hi Daniel,

> Thank you, this confirmed that the tramp version is shipped by the
> emacs system pacakge (/usr/share/emacs/29.1/lisp/net/tramp.elc) and
> not the doom framework (~/.config/emacs) like I originally thought (I
> might have looked at the "other versions" field in the
> "doom/help-packages" buffer by mistake).

I haven't used Doom Emacs ever, so I can't help you with that.

> sign_and_send_pubkey: signing failed for ED25519
> "cardno:000F73AE79E9" from agent: agent refused operation
>
> I've switched temporarily back to pinentry (non-gtk) version and run
> tramp in emacs -Q. Log at level 10 and backtrace attached. It yielded
> the same error (quoted above) as on the terminal.

Hmm, there's nothing Tramp can help you, I fear. According to the debug traces.

> GTK3 version pinentry still works fine though (both on emacs -Q and
> doom).

Good to know.

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

Hi Daniel,

> Thank you, I've upgraded doom's packages to the latest version, which
> bumped tramp up to version 2.6.1.1.
>
> I've tried looking for a way to upgrade it to 2.6.1.2 (package is
> managed by straight.el and doom's documentation doesn't mention that
> you can use release number instead of the hash commit), but this broke
> emacs and I had to revert back (thankfully all is backed by btrfs'
> snapshots).

Well, the difference between 2.6.1.1 and 2.6.1.2 is small. If you have
2.6.1.1 it would be OK.

> After restarting emacs daemon, I've tried to connect again (thinking
> that it's on version 2.6.1.1) and initially this time it worked.
>
> It did however produce the same backtrace errors, which revealed that
> tramp is still on version 2.6.0.29.1.

You can check the Tramp version by 'C-h v tramp-version'. You can also
check where Tramp is started from, by 'M-x locate-library RET tramp'.

>> It is the "prompting PIN" text I'm interested in.
>
> Because, ~/.gnupg/gpg-agent.conf was configured to use
>
> pinentry-program /usr/bin/pinentry-gnome3
>
> gtk3 for pinentry (it's necessary when working with gpg in
> thunderbird) there was not output in the terminal - just the typical
> graphical window appears and login succeeds.

I see. In this case, Tramp is not involved, and I don't need to see
anything :-)

> Interestingly though if you change it to /usr/bin/pinentry (and
> restart gpg-agent) it fails:
>
> sign_and_send_pubkey: signing failed for ED25519 "cardno:000F
> 73AE79E9" from agent: agent refused operation
>
> I'm not sure why though?

Again, show the debug buffer produced with tramp-verbose 10.

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

Hi Daniel,

> Then I've tried connecting with all the emacs plugins loaded (Doom
> Emacs framework, helm), but sadly something is still giving a
> "tramp-error - wrong-type-argument "listp i". Do you know if there are
> other options to isolate emacs before loading all the plugins or any
> other ways to diagnose this problem? Just in case I've attached the
> tramp debug output again from this doom emacs session.

Yes, according to the backtrace it is helm where Tramp errs out. Well,
you use "Emacs: 29.1 Tramp: 2.6.0.29.1". There has been major rework in
file name completion, so it might be interesting to know, whether the
recent Tramp 2.6.1.2 from GNU ELPA behaves better. Caution: There is an
error in Emacs 29.1 which would prevent to install Tramp; read the
instructions at .

>> What is the shell output, if you call ssh -o ControlMaster=auto -o
>> ControlPath=tramp.%C -o ControlPersist=no -e none 128.40.123.56
>
> There is no output, ssh logs in correctly (after prompting PIN) with a
> regular prompt, letting you issue any command and logout normally
> (ctrl+d).

It is the "prompting PIN" text I'm interested in.

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

Hi Daniel,

> Sure, please see the (truncated) debug buffer at log level 10.

Yes, the debug buffer tells us more.

> This time emacs didn't unfreeze immediately after removing the gpg
> card. It prompted for the PIN and unfroze on the second attempt. I
> don't know exactly where these events appeared in the log (it's still
> almost 3 million lines long) and I didn't include them. If you think
> that they may be needed please let me know.

Well, the discussion about the PIN isn't visible, likely in the
suppressed traces. This might result in problems in the future.

What is the shell output, if you call

--8<---cut here---start->8---
# ssh -o ControlMaster=auto -o ControlPath=tramp.%C -o ControlPersist=no -e 
none 128.40.123.56
--8<---cut here---end--->8---

However,

> 14:24:58.019684 tramp-send-command (6) # exec ssh -o ControlMaster=auto -o 
> ControlPath=tramp.%C -o ControlPersist=no -e none 128.40.123.56
> 14:26:58.670118 tramp-process-actions (6) #
> [32m[[m[32muser1[m[31m@[m[32mhost1[m:[36m~[m[32m][m[32m$[m

This tells us the current truth. Your remote prompt contains color
escape sequences, and Tramp dislikes them. Pls consult the Tramp manual
how to suppress them.

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

> Apologies Micheal for sending this message twice - I'm new to mailing
> lists and I forgot to add CC.

Sorry, I've seen this later only. So you're brave and keep the Cc :-)

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

Hi Daniel,

> Thank you, it seems to get stuck on a few tramp functions freezing
> emacs - however it returns after I physically remove the gpg smart
> card. Please see the truncated output from the debug buffer:

Please send next time the debug buffer as attachment, and w/o line
numbers. This makes it more easy to analyze.

And please keep tramp-devel@gnu.org in Cc, for the archives.

The last command send by Tramp is

--8<---cut here---start->8---
13:30:27.791080 tramp-send-command (6) # exec ssh -o ControlMaster=auto -o 
ControlPath=tramp.%C -o ControlPersist=no -e none 128.40.123.56
13:32:28.976193 tramp-process-actions (6) #
user1@128.40.123.56: Permission denied (publickey).
13:32:28.976298 tramp-process-actions (1) # File error: Tramp failed to connect.
 If this happens repeatedly, try
 ‘M-x tramp-cleanup-this-connection’
--8<---cut here---end--->8---

So there's nothing seen when communicating with the smart card. Could
you please repeat the test, but now with tramp-verbose 10?

> Best,
>
> Daniel

Best regards, Michael.

Re: "Tramp failed to connect" to an ssh server (gpg as an ssh agent)

[Michael Albinus]

Daniel Krajnik  writes:

> Hi

Hi Daniel,

> I'm wondering if anyone is able to use Tramp with a gpg smart card or
> a gpg agent that manages the ssh socket (so acts as an ssh agent)?
>
> For me it fails with file-error "Tramp failed to connect. If this
> happens repeatedly, try\n ‘M-x tramp-cleanup-this-connection’"))"
>
> I tried tramp-cleanup-this-connection, but no, that didn't help.
>
> Tramp seems to work locally - I am able to "sudo-this-file" and enter
> pin to unlock sudo (it's managed by the smart card U2F pam module).
>
> Any suggestions how to approach debugging this would be very welcome.

My crystal ball tells me that the gpg smart card communicates with Tramp
with messages Tramp isn't aware of. So we need traces.

Could you please start

--8<---cut here---start->8---
# emacs -Q -l tramp --eval '(setq tramp-verbose 6)'
--8<---cut here---end--->8---

Open the connection. There will be a debug buffer
*debug tramp/ssh 128.40.123.56#48273* which should tell us more.

> Best wishes,
>
> Daniel

Best regards, Michael.

Re: Reopening ssh file with sudo

[Michael Albinus]

Joseph Garvin  writes:

Hi Joseph,

> I originally posted this on the emacs stack exchange
> (https://emacs.stackexchange.com/questions/78768/reopen-with-sudo-command-that-works-through-tramp)
> but they suggested I try here.

"They" is me :-)

> I have tried to define this command:
>
> ```
> (defun etc-reopen-with-sudo ()
>   (interactive)
>   (let* ((vec (tramp-dissect-file-name (buffer-file-name
> (current-buffer
>(method (tramp-file-name-method vec))
>(user (tramp-file-name-user vec))
>(host (tramp-file-name-host vec))
>(localname (tramp-file-name-localname vec)))
> (message "localname=%s" localname)
> (find-alternate-file
>  (if (tramp-tramp-file-p (buffer-file-name (current-buffer)))
>  (concat (format "/%s:%s@%s|sudo::%s" method user host
> localname))
>(concat "/sudo:root@localhost:" buffer-file-name)
> ```
>
> It worked once... and now ever since mysteriously hangs whenever I try
> to use it to open a file with sudo on a remote host with ssh. The idea
> for what file string to use for this comes from
> https://stackoverflow.com/questions/2177687/open-file-via-ssh-and-sudo-with-emacs
> which suggests `/ssh:you@remotehost|sudo:remotehost:/path/to/file`
> should work. I'm able to visit files on the host just fine without
> sudo. Any ideas what I'm doing wrong here?

I'm a little bit overloaded, but I hope to come back to you within a
couple of days.

Best regards, Michael.

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

> Hi Michael,

Hi Gene,

> Here is my proposed patch (against Emacs master) for adding
> documentation for `tramp-completion-multi-hop-methods' to tramp.texi
> and also announcing the update in etc/NEWS.
>
> Please feel free to edit as you see fit or I'm happy to make any additional 
> changes.

Thanks. I've applied it to the repositories. Please remember next time
to add ChangeLog style commit messages.

Best regards, Michael.

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

> Hi Michael,

Hi Gene,

>> > I think this is a great idea. What about calling the variable
>> > `tramp-completion-multi-hop-methods', with a default of '("docker"
>> > "podman")? I am not confident including "kubernetes" here as I have
>> > not tested it but could do so if you are comfortable with that.
>
>> Sounds OK to me.
>
> Great... I've attached a patch that makes this change to current TRAMP master.

Thanks. I've pushed it to the repositories, with some minor changes.

Best regards, Michael.

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

> Hi Michael,

Hi Gene,

>> For the futere I could imagine more different values. t and nil are fine
>> ATM, but if there are more methods which support this feature, it could
>> be a list. '("docker" "podman" "kubernetes") would enable this for the
>> respective methods, and disable it for all other methods in multi-hop
>> completions. WDYT?
>
> I think this is a great idea. What about calling the variable
> `tramp-completion-multi-hop-methods', with a default of '("docker"
> "podman")? I am not confident including "kubernetes" here as I have
> not tested it but could do so if you are comfortable with that.

Sounds OK to me.

>> What you could perhaps contribute is documentation of
>> tramp-completion-remote-containers (with its changed name) in
>> tramp.texi. Maybe in node "Ad-hoc multi-hops"?
>
> I'll get started on this and an addition to etc/NEWS about the new user 
> option.

Thanks. I will study in parallel, how we could pimp up
tramp-kubernetes--completion-function for this feature. Shouldn't be a
problem, but I'm uncertain whether the other tramp-kubernetes--*
functions will need also some love.

Best regards, Michael.

Re: tramp (2.7.0-pre HEAD/dc0839de9b3654837ec8f5e66d187319b9eecd6f); bracktrace when saving modified remote file

[Michael Albinus]

Gregor Zattler  writes:

Hi Gregor,

> When I saved a modified buffer to it*'s associated
> remote file (via ssh on an Univention Corporate Server
> (debian/buster)), the file was saved but I got a
> backtrace, see below.
>
> I tried to reproduce (modify, save) with tramp-verbose
> set to 9 but no problem occured.

Well, this is a problem with Tramp for a long time. Tramp keeps a
connection to the remote host (say an ssh connection). Whenever Tramp
needs to know something, it sends a command over the connection, and
reads the reply.

This works fine when everything is serialized. But there are
asynchronous Emacs events which could trigger a new remote command while
another command is still waiting for the response. This cannot work,
because once a response arrives, Tramp wouldn't know which command is
waiting for what response.

Tramp has tried to fix this. It suppresses timers while waitng for a
response. Timers are the most likely trouble-shooter in this game.

Obviously, there are still cases where it doesn't work as expected. Like here.

> Here is the backtrace and further info (I redacted one
> directory name)

Lets' condense the backtrace to the relevant calls.

--8<---cut here---start->8---
> Debugger entered--Lisp error: (remote-file-error "Forbidden reentrant call of 
> Tramp")
>   signal(remote-file-error ("Forbidden reentrant call of Tramp"))
>   tramp-send-command((tramp-file-name "ssh" "root" nil "fs2" nil 
> "/data/projekte/Projekte//Aufgaben/" nil) "test -r 
> /data/projekte/Projekte//Aufga...")
>   
> tramp-sh-handle-file-readable-p("/ssh:root@fs2:/data/projekte/Projekte/...")
>   dired-buffer-stale-p(t)
>   auto-revert-buffer(#)
>   timer-event-handler([t 25836 43972 364989 5 auto-revert-buffers nil nil 
> 762000 nil])
--8<---cut here---end--->8---

--8<---cut here---start->8---
>   accept-process-output(# 0 nil t)
>   tramp-send-command((tramp-file-name "ssh" "root" nil "fs2" nil 
> "/data/projekte/Projekte//Aufgaben/" nil) "test -e 
> /data/projekte/Projekte//Aufga...")
>   file-exists-p("/ssh:root@fs2:/data/projekte/Projekte/...")
>   insert-directory("/ssh:root@fs2:/data/projekte/Projekte/..." 
> "-altrF --group-directories-first --block-size='1" nil t)
>   dired-revert(ignore-auto dont-ask)
>   revert-buffer(ignore-auto dont-ask preserve-modes)
>   auto-revert-handler()
>   file-notify-handle-event((file-notify (# (delete) 
> ".#MDMunattended--Antwortfile-Win11.txt") file-notify-callback))
>   tramp-sh-inotifywait-process-filter(# 
> "/data/projekte/Projekte//Aufgaben/ MOD...")
--8<---cut here---end--->8---

The lower block shows a file-notify event '(#
(delete) ".#xxx")'. This is a file lock which is deleted, because the
corresponding file was written. This calls up to "test -e
/data/projekte/Projekte//Aufga..."  (checking, whether the
directory still exists, before inserting the changed contents in the
dired buffer. Everything normal.

But while it waits for the response, a timer has started (also in the
auto-revert cosmos) which wants to send "test -r
/data/projekte/Projekte//Aufga..."  (checking, whether a
file is readable. Since the first command didn't finish yet, Tramp has
signalled it. Boom.

Obviously, Tramp didn't try hard enough to suppress timers. I've pushed
the appended patch to the repositories. Could you please observe next
days, whether the situation has been changed? Thanks.

> HTH, Gregor

Best regards, Michael.

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 9d555c5621b..b34d3ff6695 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -5615,7 +5615,8 @@ tramp-accept-process-output
 	 (v (process-get proc 'tramp-vector)))
 (dolist (p (delq proc (process-list)))
   (when (tramp-file-name-equal-p v (process-get p 'tramp-vector))
-	(with-local-quit (accept-process-output p 0 nil t)
+	(with-tramp-suspended-timers
+	  (with-local-quit (accept-process-output p 0 nil t))

   (with-current-buffer (process-buffer proc)
 (let ((inhibit-read-only t)

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

> Hi Michael,

Hi Gene,

> I have also incorporated your other suggestions and added a `defcustom
> tramp-completion-remote-containers' to allow the user to turn this
> feature on. It is off by default, but I can change this if you feel it
> appropriate.

Well, yes. I believe we could use a better name for it, because the
feature is not restricted to container methods. It could be added for
any method; it just needs to adapt the respective completion
function. A name like tramp-completion-use-ad-hoc-proxies? Perhaps you
have a better proposal; I'm notoriously bad in finding good names.

I'm kind of undecided what the default should be. nil is conservative,
no surprise of users. OTOH, this new feature will remain undetected by
the majority of users, if we don't peopagate it accordingly (for
example, enabling it by default :-)

For the futere I could imagine more different values. t and nil are fine
ATM, but if there are more methods which support this feature, it could
be a list. '("docker" "podman" "kubernetes") would enable this for the
respective methods, and disable it for all other methods in multi-hop
completions. WDYT?

> I have attached the patch in its current form. Note that this patch is
> now made against the tramp.git repository... my original patch was
> against the main Emacs repository.

Tramp repo is OK. All my development happens there. Mostly, I keep the
Tramp and Emacs repos in sync, but sometimes, when a new feature is
under development, I delay the sync.

I've pushed your patch to both the Tramp and Emacs repos. I've added
another patch afterwards, most of it are minor changes. The change which
matters is in tramp-container--completion-function. I've changed the
argument to METHOD, by this we avoid the problem in
tramp-set-completion-function checking the argument properly. Furthermore,
non-essential is bound to nil here. During completion, it is usually
bound to t, in order to suppress opening a new connection. But here, we
*want* the new connection, therefore this change. Now you can do something like

C-x C-f /ss TAB :  TAB | TAB doc TAB TAB

without to connect to any host explicitly in advance.

Furthermore, the result of tramp-container--completion-function is
cached now. I've obeserved that the function is called 5 times in a row;
we don't need so many shells to be applied on remote.

What you could perhaps contribute is documentation of
tramp-completion-remote-containers (with its changed name) in
tramp.texi. Maybe in node "Ad-hoc multi-hops"?

And also a short entry for Emacs' etc/NEWS file would be great.

Best regards, Michael.

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

Hi Gene,

>> No. Don't do that. Instead, bind tramp-last-hop-directory to nil in that
>> let-clause:
>
> I'm not sure how best to handle the lifetime of
> tramp--last-hop-directory. It needs to be explicity set to nil in
> tramp-completion-handle-file-name-all-completions UNLESS we're
> completing a multi-hop path, and only in that case should it be set to
> the parsed last hop directory.

As I have said above: Bind it in a let-clause.

> That value is then consumed later by
> tramp-container--completion-function. The value remains set after use
> however, so it is important that it is subsequently reset to nil the
> next time tramp-completion-handle-file-name-all-completions is called,
> otherwise we could end up with a stale value of
> tramp--last-hop-directory for a subsequent single-hop TRAMP
> connection.
>
> If you can suggest a better way to handle this I'd appreciate it.

The let-clause shall be the one in
tramp-completion-handle-file-name-all-completions, see below. When you
leave that clause, the original value (nil) is re-set, everything is fine.

>> Replace this by
>
>>   (setq hop (match-string 1 fullname)
>>  fullname (replace-match "" nil nil fullname 1)
>>  tramp-last-hop-directory
>>  (file-remote-p
>>   (tramp-make-tramp-file-name (tramp-dissect-hop-name hop)
>
> I tried this but ended up with a tramp--last-hop-directory that I wasn't able 
> to use to execute the container executable remotely.
>
> Here is a specific example. For the path:
>
> /ssh:in...@line5.timetiger.com|sudo:line5.timetiger.com|docker:
>
> My original patch produces this value of tramp--last-hop-directory:
>
> #("/sudo:r...@line5.timetiger.com:/" 6 10
>   (tramp-default t))
>
> When I set this as the default-directory, I am able to run the container 
> program on the remote host and obtain the list of completion candidates 
> (running containers).
>
> After incorporating your suggested changes, the value of 
> tramp--last-hop-directory becomes:
>
> "/ssh:in...@line5.timetiger.com|sudo:line5.timetiger.com:"
>
> Setting this as the default-directory, I am /not/ able to run the container 
> program (TRAMP seems to hang, at least on my macOS system).
>
> I think that, at least on my system, the root@ is required, and I think the 
> multi-hop path also doesn't work as a default-directory.

Well, testing everything I was hit by a similar problem. It took me a
while of debugging until I found the culprit.

For good reasons, tramp-completion-handle-file-name-all-completions
binds tramp-default-user and friends to nil. This is in order to get
proper completion results. But then, the code I have proposed above does
not expand to root@.

The solution is, that computing tramp-last-hop-directory must happen in
time, before tramp-default-* variables are let-bound to nil. The
following has worked in my tests:

--8<---cut here---start->8---
(defun tramp-completion-handle-file-name-all-completions (filename directory)
  "Like `file-name-all-completions' for partial Tramp files."
  (let ((fullname
 (tramp-drop-volume-letter (expand-file-name filename directory)))
(directory (tramp-drop-volume-letter directory))
tramp-last-hop-directory hop result result1)

;; Suppress hop from completion.
(when (string-match
   (rx
(regexp tramp-prefix-regexp)
(group (+ (regexp tramp-remote-file-name-spec-regexp)
  (regexp tramp-postfix-hop-regexp
   fullname)
  (setq hop (match-string 1 fullname)
fullname (replace-match "" nil nil fullname 1)
tramp-last-hop-directory
(tramp-make-tramp-file-name (tramp-dissect-hop-name hop

(let (;; When `tramp-syntax' is `simplified', we need a default method.
  (tramp-default-method
   (and (string-empty-p tramp-postfix-method-format)
tramp-default-method))
  (tramp-default-method-alist
   (and (string-empty-p tramp-postfix-method-format)
tramp-default-method-alist))
  tramp-default-user tramp-default-user-alist
  tramp-default-host tramp-default-host-alist)

  ;; Possible completion structures.
  (dolist (elt (tramp-completion-dissect-file-name fullname))
  ...
--8<---cut here---end--->8---

> Thanks for your help so far!

Best regards, Michael.

Re: [PATCH] Provide completion candidates for remote containers over a TRAMP connection

[Michael Albinus]

Gene Goykhman  writes:

Hi Gene,

I've seen your name on file at the FSF. Congratulations!

> Following up on 
> https://lists.gnu.org/archive/html/tramp-devel/2023-08/msg1.html, I have 
> prepared a patch to add remote container completion support to TRAMP.

Thanks for this! As usual, I have some comments.

> A couple of questions/things I'm uncertain of in this patch:
>
> 1. I am removing the check for executable-find for the docker and
> podman programs since they need to be run on the remote system (and
> executable-find will fail as it only checks the local machine while
> multi-hopping). I'm not sure if there is a better way to handle this.

That's a problem, indeed. I'll think about a better solution, ATM we
should do it as proposed by you.

> 2. In tramp-set-completion-function I am short-circuiting the cond
> check for a valid "file or registry key" since again, file-exists-p
> will fail when checking for the docker or podman program on a remote
> system when checking a multi-hop path. I'm open to a better ideas here
> since my change effectively no-ops the whole cond.

Same as above.

> 3. (style question) I've added my required variable tramp-last-hop-directory 
> as an autoload in tramp.el but if there's a better place for it I'd be happy 
> to move it.

No autoload is needed. tramp-container.el requires tramp.el, the
variable is known therefore.

> --- a/lisp/net/tramp-container.el
> +++ b/lisp/net/tramp-container.el
> @@ -157,6 +157,8 @@ If it is nil, the default context will be used."
>  (defconst tramp-flatpak-method "flatpak"
>"Tramp method name to use to connect to Flatpak sandboxes.")
>
> +(defvar tramp-last-hop-directory)

Not needed.

>  This function is used by `tramp-set-completion-function', please
>  see its function help for a description of the format."
> -  (when-let ((default-directory tramp-compat-temporary-file-directory)
> +  (when-let ((default-directory (or tramp-last-hop-directory
> +tramp-compat-temporary-file-directory))

Please give it a verbose comment, that poor souls reading Tramp sources
understand the trick.

> diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
> index a0092a2d706..427c0871310 100644
> --- a/lisp/net/tramp.el
> +++ b/lisp/net/tramp.el
> @@ -82,6 +82,9 @@
>  (defvar tramp-completion-method-regexp)
>  (defvar tramp-completion-file-name-regexp)
>
> +;;;###tramp-autoload
> +(defvar tramp-last-hop-directory nil)
> +

No ;;;###tramp-autoload cookie needed.  A docstring is missing, which
explains the purpose. For example, that it shouldn't be set manually,
but be let-bound.

> @@ -2133,8 +2136,8 @@ Example:
>   ;; DNS-SD service type.
>   ((string-match-p
> tramp-dns-sd-service-regexp (nth 1 (car v
> - ;; Configuration file or empty string.
> - (t (file-exists-p (nth 1 (car v))
> +;; Allow arbitrary executable name (for remote systems) 
> too.
> +(t t)))

Perhaps you mark it with a FIXME: comment.

> +(defun container-host-directory (orig)
> +  "Strips off the `tramp-docker-program' or `tramp-podman-program' suffix 
> from ORIG."
> +  "Returned path can be used to start programs on the container host."
> +  (let ((regexp (format "\\(.*\\)|\\(\\(%s\\)\\|\\(%s\\)\\)" 
> tramp-docker-program tramp-podman-program)))
> +(if (string-match regexp orig)
> +(concat (match-string 1 orig) ":/")
> +  orig)))

This function has several problems:

- It hard-codes "docker" and "podman". We want to use the new
  functionality also for other methods.

- It just replaces "docker" or "podman" whereever it is in the file
  name. What if you have "/docker:host|ssh:"?

- It expects the default file name syntax, where the method is followed
  by a colon, like in "/ssh:host|docker:". We have also other Tramp
  syntaxes, like the separate syntax. The same file name to be completed
  would be there "/[ssh/host|docker/".

- It doesn't allow docker host parts of the file name. What if a user
  wants to complete "/ssh:host|docker:ab", when she knows the target
  host starts with "ab"?

- We don't need it at all :-) See below.

>  (defun tramp-completion-handle-file-name-all-completions (filename directory)
>"Like `file-name-all-completions' for partial Tramp files."
> +  (let ((use-container-host-directory (container-host-directory directory)))
> +(when (tramp-tramp-file-p use-container-host-directory)
> +  (let ((minibuffer-completing-file-name nil))
> +(setq tramp-last-hop-directory (tramp-make-tramp-file-name
> + (tramp-dissect-file-name 
> use-container-host-directory))

No. Don't do that. Instead, bind tramp-last-hop-directory to nil in that
let-clause:

>(let ((fullname ...

Furthermore, the hop is already evall'ed in the function. We have

--8<---cut here---start->8---
  (setq hop 

Re: emacs dired + tramp seeing invalid characters

[Michael Albinus]

Michael Albinus  writes:

Hi Kiran,

>> I am using TRAMP with emacs, when use dired to browse files on remote
>> machine i see lot of invalid characters
>> for example
>> for docs i see  [01;34mdocs [0m/
>> similarly for home i see [01;34mhome [0m/
>
> This is colrized ls output ("[01;34m" is blue). Do you see it only in
> the Tramp debug buffer, or also in a dired buffer?
>
> Tramp tries to handle this, and a more recent Tramp should have no
> problem. Which Emacs / Tramp version are you using?

I've just read your debug buffer again, and ls is called like this:

--8<---cut here---start->8---
12:31:01.469621 tramp-send-command (6) # /usr/bin/ls --color=never --dired 
-hltFGH --time-style\=long-iso --group-directories-first --color\=auto 
--sort\=version --quoting-style\=literal --show-control-chars 
/xxx/x//. 2>/dev/null
--8<---cut here---end--->8---

"--color=never" comes from Tramp. But there are other arguments, like
"--color\=auto", which are not from Tramp, and which trigger ls output
colorization. Could you, please, check your customization?

Best regards, Michael.

Re: emacs dired + tramp seeing invalid characters

[Michael Albinus]

kiran dasoju  writes:

> Hi

Hi Kiran,

> I am using TRAMP with emacs, when use dired to browse files on remote
> machine i see lot of invalid characters
> for example
> for docs i see  [01;34mdocs [0m/
> similarly for home i see [01;34mhome [0m/

This is colrized ls output ("[01;34m" is blue). Do you see it only in
the Tramp debug buffer, or also in a dired buffer?

Tramp tries to handle this, and a more recent Tramp should have no
problem. Which Emacs / Tramp version are you using?

Best regards, Michael.

Re: Tramp over eshell local vs. remote directories

[Michael Albinus]

Reza Housseini  writes:

Hi Reza,

>> Yes, this is intended. "cd" is a built-in command of eshell, and it
>> interprets the directory name like a Lisp file name. "~" is your local
>> home directory. If you do "C-x C-f ~", you will always open your local
>> home directory, whereever you call it from in Emacs.
>
> I was just wondering because calling cp /some/file ~, copies the file
> to the remote home directory, therefore the tilde is interpreted
> differently in this case.

Hmm, this looks inconsistent, indeed. Perhaps you report the case as
Emacs bug?

(I'm not sure whether this is a bug or an inconsistency, but it might be
good to be clarified)

> Best regards,

Best regards, Michael.

Re: Slow initial connection to remote

[Michael Albinus]

adam bobrow  writes:

> Hey Michael,

Hi Adam,

> Can't thank you enough, I didn't expect that level of support! I hope
> you didn't spend too much time going over that huge log :)

No problem. These days I'm working on Tramp optimizations, so your
request was a nice opportunity to check the state-of-the-art.

The upcoming Tramp 2.7 will contain some of the optimizations. It is
integrated in Emacs 30, likely it will be availble via GNU ELPA as well.

> As you said, when running emacs normally (no -q) everything feels much
> faster. I also set tramp-verbose to 1 and that also might have helped
> I suppose.

See also the Tramp manual at (info "(tramp) Frequently Asked Questions")
There is an entry "How to speed up TRAMP",. which could give you further ideas.

> Thanks again for the help, very much appreciated,
>
> Adam

Best regards, Michael.

Re: Tramp over eshell local vs. remote directories

[Michael Albinus]

Reza Housseini  writes:

> Hi Michael

Hi Reza,

> I open a virgin emacs with emacs -Q and then connect to a remote
> server via dired: C-x d /ssh:user@remote.server:/some/directory and
> from there open a eshell M-x eshell.
> Now when I change directory in the eshell with cd ~ I go back to my
> local user directory. I this intended behavior (kind of unexpected)
> and is there a way to stay on the remote side?

Yes, this is intended. "cd" is a built-in command of eshell, and it
interprets the directory name like a Lisp file name. "~" is your local
home directory. If you do "C-x C-f ~", you will always open your local
home directory, whereever you call it from in Emacs.

Since Emacs 29, there is the extension module eshell-elecslash. Add the
following lines to your .emacs:

--8<---cut here---start->8---
(require eshell
(add-to-list 'eshell-modules-list 'eshell-elecslash)
--8<---cut here---end--->8---

Now, when you are in eshell on a remote directory, and you type a
command argument beginning with "/" or "~/", this will be replaced by
the remote file identification. That is, in your example you type
"cd ~/" and that's it. Read the Eshell manual.

> Thanks for any hints!
>
> Cheers,

Best regards, Michael.

Re: Slow initial connection to remote

[Michael Albinus]

adam bobrow  writes:

> HI,

Hi Adam,

> While connecting a remote host using Tramp, I encounter an
> indeterminate 20 seconds delay after a successful initial connection.

Yes. You seem to have a slow connection to that host (no surpise going
to AWS), but Tramp must support this as well.

> From my understanding, when inspecting the debug log it seems that
> Tramp identifies the remote prompt after about 3 seconds, which I
> assume is just the network latency but then it takes another ~20
> seconds to finally connect.

Yes.

> I'm unsure what's Tramp's actions during these seconds. Your input
> would be appreciated.

I'm inserting all messages with level 6 from your log (these are the
sent commands, and the returned output) plus a comment.

--8<---cut here---start->8---
10:10:42.724900 tramp-call-process (6) # ‘locale -a’ nil  *temp*
10:10:42.748631 tramp-call-process (6) # 0
en_NZ
nl_NL.UTF-8
...
10:10:42.748966 tramp-call-process (6) # ‘ssh -G -o ControlMaster=auto 0.0.0.1’ 
nil nil
10:10:42.760573 tramp-call-process (6) # 0
Pseudo-terminal will not be allocated because stdin is not a terminal.
user bob
hostname 0.0.0.1
...
10:10:42.760697 tramp-call-process (6) # ‘ssh -G -o ControlPath=tramp.%C 
0.0.0.1’ nil nil
10:10:42.766693 tramp-call-process (6) # 0
Pseudo-terminal will not be allocated because stdin is not a terminal.
user bob
hostname 0.0.0.1
...
10:10:42.766811 tramp-call-process (6) # ‘ssh -G -o ControlPersist=no 0.0.0.1’ 
nil nil
10:10:42.772112 tramp-call-process (6) # 0
Pseudo-terminal will not be allocated because stdin is not a terminal.
user bob
hostname 0.0.0.1
port 22
...
--8<---cut here---end--->8---

These are some local commands, where Tramp determines existing locale
encodings and ssh client properties. They are not performance relevant

--8<---cut here---start->8---
10:10:42.783066 tramp-maybe-open-connection (6) # /bin/sh -i
10:10:42.788943 tramp-wait-for-regexp (6) #
#$
10:10:42.789916 tramp-send-command (6) # exec ssh -o ControlMaster=auto -o 
ControlPath=tramp.%C -o ControlPersist=no -e none bobrows
10:10:45.360996 tramp-process-actions (6) #
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1029-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support:https://ubuntu.com/advantage

  System information as of Tue Aug  8 07:10:44 UTC 2023

  System load:  0.0Processes: 108
  Usage of /:   26.2% of 15.32GB   Users logged in:   1
  Memory usage: 71%IPv4 address for eth0: 172.31.83.132
  Swap usage:   0%

 * Ubuntu Pro delivers the most comprehensive open source security and
   compliance features.

   https://ubuntu.com/aws/pro

Expanded Security Maintenance for Applications is not enabled.

7 updates can be applied immediately.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


Last login: Tue Aug  8 06:55:11 2023 from 176.230.32.30
$
--8<---cut here---end--->8---

Tramp established the remote shell. 3 seconds, as you have said.

--8<---cut here---start->8---
10:10:45.361199 tramp-send-command (6) # exec env TERM='dumb' 
INSIDE_EMACS='29.1,tramp:2.6.0.29.1' ENV='' HISTFILE=~/.tramp_history 
PROMPT_COMMAND='' PS1=///b2d958bbca0546596c9001895132444f\#\$ PS2='' PS3='' 
/bin/sh  -i
10:10:45.507803 tramp-wait-for-regexp (6) #
///b2d958bbca0546596c9001895132444f#$
--8<---cut here---end--->8---

Tramp opens an interactive shell /bin/sh. This is necessary, because
Tramp cannot know, whether your login shell is POSIX compliant. It sets
also the shell prompt to the random string
"///b2d958bbca0546596c9001895132444f#$" in order to detect it reliably.

Now it checks some shell properties.

--8<---cut here---start->8---
10:10:45.508291 tramp-send-command (6) # (cd ~/) 2>/dev/null; echo 
tramp_exit_status $?
10:10:45.652286 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///b2d958bbca0546596c9001895132444f#$
--8<---cut here---end--->8---

Check, whether the remote shell supports tilde expansion.

--8<---cut here---start->8---
10:10:45.653454 tramp-send-command (6) # set +o vi +o emacs
10:10:45.796193 tramp-wait-for-regexp (6) #
///b2d958bbca0546596c9001895132444f#$
--8<---cut here---end--->8---

Remove some shell options for editing.

--8<---cut here---start->8---
10:10:45.796312 tramp-send-command (6) # stty -inlcr -onlcr -echo kill '^U' 
erase '^H'
10:10:45.940105 tramp-wait-for-regexp (6) #
///b2d958bbca0546596c9001895132444f#$
--8<---cut 

Re: tramp (2.6.1 HEAD/3ff676c2f98cb6c47fecb37f31a589a910dd3876); docker-container ssh multi-hop support

[Michael Albinus]

Gene Goykhman  writes:

> Hi Michael,

Hi Gene,

>> Well, you disable this for all completions. I don't know whether we want
>> this. It is useful for the very first hop, and not only for the docker
>> method, but for all methods.
>
> I agree, and it doesn't seem that TRAMP currently allows that level of 
> granularity for disabling caching. It does seem out of scope for this 
> particular enhancement.

Indeed. Perhaps we can leave it out for now. I also don't believe that
we need to disable completion from the cache at all. If we would be able
to determine, which cache information is valid where, we could use this
information. For example, "host1" is reachable via ssh only from the
local host, and "host2" is reachable via ssh only from "host1". "host2"
would be a completion candidate only after a first hop to "host1".

But we don't have this information yet. Let's do this later.

>> > (add-to-list 'tramp-completion-function-alist
>> >  `("docker"
>> >(tramp-container--completion-function ,tramp-docker-program)
>> >(my-tramp-container-completion-function 
>> > ,tramp-docker-program)))
>
>> So you have two entries. Do you still need 
>> tramp-container--completion-function?
>
> I wasn't sure whether it was important to keep the original function in the 
> list.

It depends. If the new implementation in
my-tramp-container-completion-function covers also the case of
tramp-container--completion-function, then we need only one function. I
would propose to do so, and add all changes into the latter function.

Btw, it shouldn't work only for docker, but also for podman.

>> > (defun my-tramp-docker-host-path (orig)
>> >   ;; Strips off "|docker" suffix from ORIG allowing
>> >   ;; it to be converted to a tramp-file using TRAMP's path functions.
>> >   (let ((regexp "\\(.*\\)|docker"))
>> > (if (string-match regexp orig)
>> > (concat (match-string 1 orig) ":/")
>> >   orig)))
>
>> Well, this is docker related. The proper Tramp functions would be
>
>> (tramp-make-tramp-file-name (tramp-dissect-hop-name
>> (tramp-file-name-hop (tramp-dissect-file-name FILENAME
>
>> Hmm, perhaps this is worth an own function. Or perhaps we have it
>> already, and I don't remember. We have a similar logic in computing the
>> previous hop in function tramp-compute-multi-hops, but not in the same
>> sense as we need it here.
>
>> And this also does not handle the case of more than two hops, like in
>> "/ssh:host1|ssh:host2|docker:container::"
>
> Yes I have to defer to your judgement here. It was a very simple
> attempt that worked for my particular use-case.

We could start with just one hop. But we shouldn't hard-code "docker"
and alike, in order to be open for other methods.

>> As said, I propose to integrate it in 
>> tramp-completion-handle-file-name-all-completions.
>> And I don't understand, why you call (tramp-make-tramp-file-name 
>> (tramp-dissect-file-name ...))
>> my-host-path is already a proper file name, isn't it?
>
> They are slightly different. For example, testing in my sutation I have this 
> my-host-path:
>
>   /ssh:in...@line5.timetiger.com|sudo:line5.timetiger.com:/
>
> But after the tramp-make-tramp-file-name transformation it is:
>
>   /ssh:in...@line5.timetiger.com|sudo:r...@line5.timetiger.com:/
>
> I think the additional username after sudo: is needed in order to execute 
> docker on the remote host. I'm not sure if this is the best way to accomplish 
> this.

/sudo:line5.timetiger.com:/ and /sudo:r...@line5.timetiger.com:/ are
identical in terms of remote file names. The user name is just expanded
due to the respective entry in tramp-default-user-alist (the second entry):

--8<---cut here---start->8---
tramp-default-user-alist is a variable defined in ‘tramp.el’.

Its value is
(("\\`\\(?:fcp\\|krlogin\\|nc\\|r\\(?:cp\\|emcp\\|sh\\)\\|telnet\\)\\'"
  nil "albinus")
 ("\\`\\(?:doas\\|ksu\\|su\\(?:do\\)?\\)\\'" nil #1="root")
 ("\\`smb\\'" nil nil) ("\\`sudoedit\\'" nil #1#))
Original value was nil
--8<---cut here---end--->8---

I propose you try to write patches towards the existing Tramp. We can
discuss them, and we can add them once your FSF legal papers are fine,
this shouldn't last longer than two weeks I hope.

If you want to check your changes regularly, you might consider
tramp-tests.el There are already two test cases for completion, but
nothing for multi-hop completion yet. I find it always useful to write
tests in parallel to new features, in order to be sure nothing
breaks. And also in order to demonstrate how it is intended to work.

Best regards, Michael.

Re: tramp (2.6.1 HEAD/3ff676c2f98cb6c47fecb37f31a589a910dd3876); docker-container ssh multi-hop support

[Michael Albinus]

Michael Albinus  writes:

Hi,

>> I've made some progress and have published a gist with my current
>> approach to multi-hop Docker container completion in TRAMP. I'd
>> appreciate comments and suggestions, and whether this is a reasonable
>> direction to take or if I'm missing something important.
>
> As said already, I like the idea. It has potential to be used for other
> Tramp methods as well, so it might be added to Tramp proper.

Thinking more about, it isn't only parsing multi-hop file names. Proxies
could be defined already in tramp-default-proxies-alist, which must be
taken into account as well.

Best regards, Michael.

Re: tramp (2.6.1 HEAD/3ff676c2f98cb6c47fecb37f31a589a910dd3876); docker-container ssh multi-hop support

[Michael Albinus]

Gene Goykhman  writes:

Hi Gene & Kristofer,

Sorry for the delay, but bug fixing has priority ...

> I've made some progress and have published a gist with my current
> approach to multi-hop Docker container completion in TRAMP. I'd
> appreciate comments and suggestions, and whether this is a reasonable
> direction to take or if I'm missing something important.

As said already, I like the idea. It has potential to be used for other
Tramp methods as well, so it might be added to Tramp proper.

However, this would require to assign the copyright of the code to the
FSF, as all non-trivial contributions to Emacs require. Would you be
willing to sign this?

>> A very simple change I tried when previously trying to get multihop
>> completion working
>> with another backend was that if I changed the variable
>> `tramp-compat-temporary-file-directory'
>> for the function `temporary-file-directory' in the completion
>> functions (like `tramp-container--completion-function'), then at least
>
> I'm advising :before tramp-completion-handle-file-name-all-completions
> and parsing out the required remote directory from which we need to
> call the docker program to get remote completions. Then, I'm adding a
> custom completion function my-tramp-container-completion-function that
> takes that directory (if it has been parsed) and sets it as the
> default-directory.

I guess we don't need to advice 
tramp-completion-handle-file-name-all-completions.
The code should be added directly there.

>> Another issue that I found, which maybe is unavoidable, is that some
>> of the completion functions in `tramp-container' requires that the
>> container
>> program is installed on the local host, but sometimes I only have
>
> I /think/ the approach I've taken only requires that the container program be 
> available on the remote host on which we are trying to get the completion 
> list.

Yep. On every hop of a multi-hop file name only the program for that hop
must be available. That is, for "/ssh:host|docker:container:" we need
ssh on the local host, and docker on the host "host".

> ;; Disable TRAMP caching so we don't see unavailable containers
> (setq tramp-completion-use-cache nil)

Well, you disable this for all completions. I don't know whether we want
this. It is useful for the very first hop, and not only for the docker
method, but for all methods.

However, we might need a marker in the cache, whether the entry belongs
to the very first hop, or not. This we can implement later.

>   (when-let ((default-directory (if tramp-last-hop-directory
> tramp-last-hop-directory
>   tramp-compat-temporary-file-directory))

I would write (or tramp-last-hop-directory 
tramp-compat-temporary-file-directory)

> (add-to-list 'tramp-completion-function-alist
>  `("docker"
>(tramp-container--completion-function ,tramp-docker-program)
>(my-tramp-container-completion-function 
> ,tramp-docker-program)))

So you have two entries. Do you still need tramp-container--completion-function?

> (defun my-tramp-docker-host-path (orig)
>   ;; Strips off "|docker" suffix from ORIG allowing
>   ;; it to be converted to a tramp-file using TRAMP's path functions.
>   (let ((regexp "\\(.*\\)|docker"))
> (if (string-match regexp orig)
> (concat (match-string 1 orig) ":/")
>   orig)))

Well, this is docker related. The proper Tramp functions would be

(tramp-make-tramp-file-name (tramp-dissect-hop-name (tramp-file-name-hop 
(tramp-dissect-file-name FILENAME

Hmm, perhaps this is worth an own function. Or perhaps we have it
already, and I don't remember. We have a similar logic in computing the
previous hop in function tramp-compute-multi-hops, but not in the same
sense as we need it here.

And this also does not handle the case of more than two hops, like in 
"/ssh:host1|ssh:host2|docker:container::"

> ;; When TRAMP handles completions, first set tramp-last-hop-directory
> ;; so that my completion function can set it before running the Docker 
> executable
>
> (defun my-tramp-set-docker-default-directory (filename directory)
>   (setq tramp-last-hop-directory nil)
>   (let ((my-host-path (my-tramp-docker-host-path directory)))
> (when (tramp-tramp-file-p my-host-path)
>   (setq tramp-last-hop-directory (tramp-make-tramp-file-name
>   (tramp-dissect-file-name 
> my-host-path))
>
> (advice-add #'tramp-completion-handle-file-name-all-completions :before 
> #'my-tramp-set-docker-default-directory)

As said, I propose to integrate it in 
tramp-completion-handle-file-name-all-completions.
And I don't understand, why you call (tramp-make-tramp-file-name 
(tramp-dissect-file-name ...))
my-host-path is already a proper file name, isn't it?

Best regards, Michael.

Re: Errors when installing Tramp from ELPA on Emacs 29.1

[Michael Albinus]

Michael Albinus  writes:

Hi Jim,

>> When installing Tramp from GNU ELPA on Emacs 29.1, I get a bunch of
>> errors (see the attached "tramp-compile.log"). In addition, when I
>> start Emacs up the next time, I get this error in the *Messages*
>> buffer:
>>
>> I'm guessing this is due to Emacs 29.1 having a new method for
>> generating autoloads (loaddefs-gen.el). It looks like the Makefile for
>> Tramp from ELPA still tries to use the old way of generating
>> autoloads, but I guess package.el tries to use the new one, causing
>> some kind of conflict. Looking at the generated tramp-autoloads.el, it
>> seems to be a lot different from how it was generated in Emacs 28.2,
>> and in particular, has some calls to 'tramp--with-startup' (I've
>> attached this as well). I don't quite understand what's going on
>> beyond that, though...
>
> Yep, it seems to be an autoload problem. An I also don't understand
> what's up :-(
>
> I'll try next days to fix it. Ach ...

I've debugged it for a while, but I must confess that the whole
machinery is mystery to me. Therefore, I've written bug#65023.

Best regards, Michael.

Re: tramp (2.6.1 HEAD/3ff676c2f98cb6c47fecb37f31a589a910dd3876); docker-container ssh multi-hop support

[Michael Albinus]

Gene Goykhman  writes:

Hi Gene & Kristofer,

> I've made some progress and have published a gist with my current
> approach to multi-hop Docker container completion in TRAMP. I'd
> appreciate comments and suggestions, and whether this is a reasonable
> direction to take or if I'm missing something important.
>
>   https://gist.github.com/genegoykhman/09868acd369b3fae866be6446997de9f
>
> I've taken an approach similar to the one Kristofer describes ...
>
>> A very simple change I tried when previously trying to get multihop
>> completion working
>> with another backend was that if I changed the variable
>> `tramp-compat-temporary-file-directory'
>> for the function `temporary-file-directory' in the completion
>> functions (like `tramp-container--completion-function'), then at least
>
> I'm advising :before tramp-completion-handle-file-name-all-completions
> and parsing out the required remote directory from which we need to
> call the docker program to get remote completions. Then, I'm adding a
> custom completion function my-tramp-container-completion-function that
> takes that directory (if it has been parsed) and sets it as the
> default-directory.
>
>> Another issue that I found, which maybe is unavoidable, is that some
>> of the completion functions in `tramp-container' requires that the
>> container
>> program is installed on the local host, but sometimes I only have
>
> I /think/ the approach I've taken only requires that the container program be 
> available on the remote host on which we are trying to get the completion 
> list.
>
> Feedback appreciated!

Thanks for all your work! The idea is nice. I'll check next days, and
send you feedback.

Unfortunately, I have first to fix the Tramp ELPA package. People have
reported, that it doesn't install any longer with the just-released
Emacs 29.1 ...

Best regards, Michael.

Re: Errors when installing Tramp from ELPA on Emacs 29.1

[Michael Albinus]

Jim Porter  writes:

Hi Jim,

> When installing Tramp from GNU ELPA on Emacs 29.1, I get a bunch of
> errors (see the attached "tramp-compile.log"). In addition, when I
> start Emacs up the next time, I get this error in the *Messages*
> buffer:
>
>   Error loading autoloads: (void-function tramp--with-startup)
>
> If I eval "(require 'tramp)", I get this error:
>
>   Debugger entered--Lisp error: (void-function
>   tramp-register-crypt-file-name-handler)
> tramp-register-crypt-file-name-handler()
> tramp-register-file-name-handlers()
> #f(compiled-function () #)()
> run-hooks(tramp--startup-hook)
>
> byte-code("\301\302\303\304#\210\305\306!\210\307\300!\210\310\211\20\207"
> [tramp--startup-hook put tramp-unload-tramp tramp-autoload t provide
> tramp run-hooks nil] 5)
> require(tramp)
> eval-expression((require 'tramp) nil nil 127)
> funcall-interactively(eval-expression (require 'tramp) nil nil 127)
> command-execute(eval-expression)
>
> I'm guessing this is due to Emacs 29.1 having a new method for
> generating autoloads (loaddefs-gen.el). It looks like the Makefile for
> Tramp from ELPA still tries to use the old way of generating
> autoloads, but I guess package.el tries to use the new one, causing
> some kind of conflict. Looking at the generated tramp-autoloads.el, it
> seems to be a lot different from how it was generated in Emacs 28.2,
> and in particular, has some calls to 'tramp--with-startup' (I've
> attached this as well). I don't quite understand what's going on
> beyond that, though...

Yep, it seems to be an autoload problem. An I also don't understand
what's up :-(

I'll try next days to fix it. Ach ...

Re: tramp (2.7.0-pre master/27944247d161f0ed65bbb34959c4493f7984987a); Error in remote grep-find from Windows host

[Michael Albinus]

Richard Copley  writes:

Hi Richard,

>> Thanks for the report and the analysis! Could you pls check, whether the
>> appended patch fixes this?
>
> Thank you. Yes, that works.

Thanks for the feedback. I've pushed the fix to the repositories.

Best regards, Michael.

Re: tramp (2.7.0-pre master/27944247d161f0ed65bbb34959c4493f7984987a); Error in remote grep-find from Windows host

[Michael Albinus]

Richard Copley  writes:

Hi Richard,

> Recipe starting from 'emacs -Q', on Windows assuming a PuTTy session
> called "session" is configured:
>
> C-x C-f /plink:session:/tmp RET
> M-x grep-find RET
> x RET
>
> This occurs because 'null-device' is let-bound to nil in 'grep-find',
> and this is not handled by the regex constructor in
> tramp-sh-handle-expand-file-name (only used on Windows). Evaluating
> the following form gives the same error:
>
> (let ((null-device nil))
>   (rx bol (| (: alpha ":") (: (literal null-device) eol

Thanks for the report and the analysis! Could you pls check, whether the
appended patch fixes this?

Best regards, Michael.

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 3ab58c27..0cb953e2 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -2715,7 +2715,8 @@ the result will be a local, non-Tramp, file name."
   ;; there could be the false positive "/:".
   (if (or (and (eq system-type 'windows-nt)
 	   (string-match-p
-		(rx bol (| (: alpha ":") (: (literal null-device) eol))) name))
+		(rx bol (| (: alpha ":") (: (literal (or null-device "")) eol)))
+		name))
 	  (and (not (tramp-tramp-file-p name))
 	   (not (tramp-tramp-file-p dir
   (tramp-run-real-handler #'expand-file-name (list name dir))

Re: tramp (2.6.1 HEAD/3ff676c2f98cb6c47fecb37f31a589a910dd3876); docker-container ssh multi-hop support

[Michael Albinus]

Gene Goykhman  writes:

> Hello and thank you for making TRAMP amazing!

Hi Gene,

> I don't know whether this is a bug or currently unsupported
> functionality, but I am trying to get docker-container support through an
> ssh connection and it is not working. docker-container works fine for
> docker running on the local machine.
>
> However:
>
>   M-x find-file RET /ssh:root@remotehost|docker:
>
> Also suggests the SAME list of docker containers running on the local
> host, rather than the (expected) list of containers on remotehost.

Yes, this is a general weakness of Tramp, for all methods. It doesn't
complete user and host names for multi-hop names but the first
hop. Perhaps I shall add something which doesn't offer completion in
this case.

> I tried ignoring the (incorrect) completion candidates and manually typing 
> the name of the
> running docker container on the remote host as follows:
>
>   M-x find-file RET /ssh:root@remotehost|docker:remotecontainername
>
> But that opened a new, empty buffer in Fundamental mode and emitted the
> error: "File not found and directory write-protected"

This misses the trailing colon, you need

M-x find-file RET /ssh:root@remotehost|docker:remotecontainername:

> Is using TRAMP to connect to docker containers running on a remote host
> via ssh (i.e. multihop) supported? If so, could you point me to some 
> information or
> instructions?

Yes, it is supported. I test it regularly with the regression tests. If
it still doesn't work for you, pls set tramp-verbose to 6 and rerun the
test. Show the resulting Tramp debug buffer.

> Thank you!

Best regards, Michael.

Re: [EXTERNAL] Re: tramp (2.5.2.28.1 nil/nil); stdin is not a tty

[Michael Albinus]

"Ali, Md *"  writes:

Hi,

pls keep tramp-devel in CC, for the archives.

> I was trying to connect to docker container.
> /docker:containerid:/home
>
> And I got error
> docker exec -it container_id sh && exit || exit
> stdin is no a tty
> exit
>
> I am running emacs 28.1 on windows 10.

You are using Tramp 2.5.2.28.1. This Tramp version doesn't support
docker. Likely, you are using docker integration from somewhere else.

The recent Tramp 2.6.1 supports docker integration. You might install it
from GNU ELPA.

Another problem might be that you're on Windows 10. I haven't tried to
access a docker container from that OS.

> Thanks
>
> Md Yousuf Ali

Best regards, Michael.

Re: tramp (2.5.2.28.1 nil/nil); stdin is not a tty

[Michael Albinus]

"Ali, Md *"  writes:

Hi,

could you pls be more verbose? What have you done, and when did you see
the error message?

Best regards, Michael.

Tramp 2.6.1

[Michael Albinus]

Hello,

The Tramp team is happy to announce the release of Tramp 2.6.1.

Tramp is a remote file editing package for Emacs.  Tramp uses different
methods like ssh or scp to access files on remote hosts as if they were
local files.  Access to the remote file system for editing files,
version control, directory editing and running processes on the remote
host are transparently enabled.

Tramp 2.6.1 supports Emacs since version 26.1.  This is the version
integrated with Emacs 29.

The download files are located at .  Tramp's
homepage is .

Tramp 2.6.1 is also available via GNU ELPA.

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Provides option to disable shortened multi-hop tramp file name

[Michael Albinus]

Warren Lynn  writes:

> Hi Michael,

Hi Warren,

> I verified your latest patch works. Thanks a lot for your quick
> action.

Thanks for the feedback. I've pushed the patch to the Tramp and Emacs
repositories. It will appear on GNU ELPA with Tramp 2.6.1, scheduled for
next week. And also, months away from now, it will be in Emacs 29.2.

> Best regards.
>
> Warren

Best regards, Michael.

Re: tramp (2.6.0.29.1 nil/nil); Provides option to disable shortened multi-hop tramp file name

[Michael Albinus]

Warren Lynn  writes:

> Hi Michael,

Hi Warren,

> I actually put in a similar patch as yours myself, and found that is
> not enough. The tricky part is when tramp-make-tramp-file-name is
> called by tramp-make-tramp-hop-name, the hop name must be omitted (set
> to nil). Otherwise, we will get into some infinite loop. For other
> cases, keep the hop. With that additional consideration, it works for
> me, although I do not know how the whole system works and the code is
> really hacky. Maybe you can give some thoughts on that?

I've seen the same problem in my regression tests. See a reworked patch
appended. Note also, that I have renamed the new user option to
`tramp-show-ad-hoc-proxies', in order to be consistent with other
variable names.

> Thanks.
>
> Warren

Best regards, Michael.

diff --git a/lisp/tramp.el b/lisp/tramp.el
index 90fd53b0..8e5b7527 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -523,6 +523,11 @@ interpreted as a regular expression which always matches."
   :version "24.3"
   :type 'boolean)

+(defcustom tramp-show-ad-hoc-proxies nil
+  "Whether to show ad-hoc proxies in file names."
+  :version "29.2"
+  :type 'boolean)
+
 ;; For some obscure technical reasons, `system-name' on w32 returns
 ;; either lower case or upper case letters.  See
 ;; .
@@ -1825,8 +1830,8 @@ the form (METHOD USER DOMAIN HOST PORT LOCALNAME  HOP)."
   (when (cadr args)
 	(setq localname (and (stringp (cadr args)) (cadr args
   (when hop
-	;; Keep hop in file name for completion.
-	(unless minibuffer-completing-file-name
+	;; Keep hop in file name for completion or when indicated.
+	(unless (or minibuffer-completing-file-name tramp-show-ad-hoc-proxies)
 	  (setq hop nil))
 	;; Assure that the hops are in `tramp-default-proxies-alist'.
 	;; In tramp-archive.el, the slot `hop' is used for the archive
@@ -1877,7 +1882,7 @@ the form (METHOD USER DOMAIN HOST PORT LOCALNAME  HOP)."
  (tramp-compat-rx
   (regexp tramp-postfix-host-regexp) eos)
  tramp-postfix-hop-format
- (tramp-make-tramp-file-name vec 'noloc)
+ (tramp-make-tramp-file-name (tramp-file-name-unify vec))

 (defun tramp-completion-make-tramp-file-name (method user host localname)
   "Construct a Tramp file name from METHOD, USER, HOST and LOCALNAME.

Re: tramp (2.6.0.29.1 nil/nil); Provides option to disable shortened multi-hop tramp file name

[Michael Albinus]

Michael Albinus  writes:

Hi Warren,

>> Again, I was upgrading from Emacs 26.3 to 29.0.91 (Linux), and found
>> shortened multi-hop tramp file names are mandatory now. I am sure many
>> people like this feature, but my own preference is to have every
>> multi-hop tramp file name to be self-contained, with all the hops
>> included in the name.
>
> I will work on this. However, it won't be a short shot, it might take
> some days.

It seems easier than expected. Could you pls check the appended patch?
It's on top of Tramp 2.6.0.5 (from ELPA), but it should also apply on
Tramp integrated into the Emacs 29 pretest (I hope).

There's a new user option `tramp-show-ad-hoc-multi-hops'. When you set
it to t, the full multi-hop file names shall be visible.

I haven't finished regression tests, but it looks promising.

>> Warren

Best regards, Michael.

diff --git a/lisp/tramp.el b/lisp/tramp.el
index 90fd53b0..745913f8 100644
--- a/lisp/tramp.el
+++ b/lisp/tramp.el
@@ -1800,6 +1800,11 @@ See `tramp-dissect-file-name' for details."

 (put #'tramp-buffer-name 'tramp-suppress-trace t)

+(defcustom tramp-show-ad-hoc-multi-hops nil
+  "Whether to show ad-hoc multi-hops in file names."
+  :version "29.2"
+  :type 'boolean)
+
 ;;;###tramp-autoload
 (defun tramp-make-tramp-file-name ( args)
   "Construct a Tramp file name from ARGS.
@@ -1825,8 +1830,8 @@ the form (METHOD USER DOMAIN HOST PORT LOCALNAME  HOP)."
   (when (cadr args)
 	(setq localname (and (stringp (cadr args)) (cadr args
   (when hop
-	;; Keep hop in file name for completion.
-	(unless minibuffer-completing-file-name
+	;; Keep hop in file name for completion or when indicated.
+	(unless (or minibuffer-completing-file-name tramp-show-ad-hoc-multi-hops)
 	  (setq hop nil))
 	;; Assure that the hops are in `tramp-default-proxies-alist'.
 	;; In tramp-archive.el, the slot `hop' is used for the archive