Control: user ubuntu-de...@lists.ubuntu.com
Control: usertag -1 origin-ubuntu ubuntu-patch
Hi Ben,
as I mentioned to you, I have implemented sftp support for dput
again. I was merging the latest dput 1.0.1 (tag missing, BTW) and
had to rewrite the method because bzrlib was not available on Python 3.
I attached a patch introducing the new method and manual page updates
for it. I tried to make sure to follow the coding style of the other
code as far as possible :) (I also attached a small bugfix for dput.cf's
ubuntu entries).
This implements sftp support by using paramiko's sftp support on top
of an openssh connection (invoking the ssh binary). It's a stripped down
version of what bzrlib was doing, and there is reason for doing it that
way:
I looked at both paramiko and asyncssh, as well as bindings to libssh2,
and they all suffer from an important problem: Interactively establishing
a connection. For example, paramiko just raises an exception if you have
an encrypted private key file; and it also only provides 3 strategies for
dealing with unknown hosts: Add automatically, Warn and add, or Reject
- no interactive stuff. asyncssh is similarly limited, perhaps even more,
and libssh2 bindings only provide explicit authentication methods.
Hence I decided to follow bzr and use ssh(1) to establish the connection
in a subprocess and then use paramiko's sftp implementation on top of it.
This ensures that we get all the normal connection setup and authentication
handling as we do if we just ssh somewhere.
A possible improvement would be to upload the file atomically - bzr does
that. The other methods don't, but it seems reasonably easy to implement
- I'd be happy to do so.
I also have a few other commits in my git branch you might find interesting:
https://git.launchpad.net/~juliank/+git/dput for-debian
https://git.launchpad.net/~juliank/+git/dput?h=for-debian (web view)
Let me know what you think!
Thanks,
Julian
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
>From aa58ec7ea124d3f2cb51ebbd4371dc68b1798460 Mon Sep 17 00:00:00 2001
From: Julian Andres Klode <julian.kl...@canonical.com>
Date: Thu, 11 Jan 2018 09:06:10 +0100
Subject: [PATCH 1/6] Implement SFTP support
Implement SFTP by using paramiko's SFTP support on top of an
SSH process which acts as the SFTP channel.
The reason for doing it this way, rather than using paramiko
directly is that paramiko is not really designed for interactive
use. Integration with agents does not seem to work, and a form
of interactive unknown host adding does not exist either. By reusing
openssh for the connection part, we solve this problem.
Other python3 libraries are not better either. asyncssh suffers from
the same connection setup problem, and bindings for libssh2 are even
worse - they require you manually handling which authentication types
are tried.
Closes: #505173
---
debian/control | 2 +-
doc/man/dput.cf.5 | 4 ++
dput/methods/sftp.py | 159 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 164 insertions(+), 1 deletion(-)
create mode 100644 dput/methods/sftp.py
diff --git a/debian/control b/debian/control
index 2162572..7244944 100644
--- a/debian/control
+++ b/debian/control
@@ -25,7 +25,7 @@ Depends:
python3-pkg-resources,
${python3:Depends},
${misc:Depends}
-Suggests: openssh-client, lintian, mini-dinstall, rsync
+Suggests: openssh-client, python3-paramiko, lintian, mini-dinstall, rsync
Breaks:
devscripts (<< 2.17.0)
Replaces:
diff --git a/doc/man/dput.cf.5 b/doc/man/dput.cf.5
index 5e78a57..14753ff 100644
--- a/doc/man/dput.cf.5
+++ b/doc/man/dput.cf.5
@@ -120,6 +120,10 @@ This transfers files using a secure SSH tunnel, and needs
authentication credentials on the remote machine.
.
.TP
+.B sftp
+the package will be uploaded using ssh's sftp. This transfers files using a
+secure ssh tunnel, and needs sftp access on the upload machine.
+.TP
.B rsync
The package will be uploaded using
.B rsync
diff --git a/dput/methods/sftp.py b/dput/methods/sftp.py
new file mode 100644
index 0000000..bd4ca6f
--- /dev/null
+++ b/dput/methods/sftp.py
@@ -0,0 +1,159 @@
+# dput/methods/sftp.py
+# Part of ‘dput’, a Debian package upload toolkit.
+#
+# This is free software, and you are welcome to redistribute it under
+# certain conditions; see the end of this file for copyright
+# information, grant of license, and disclaimer of warranty.
+
+
+""" Implementation for SFTP upload method. """
+
+import errno
+import os
+import os.path
+import sys
+import socket
+import subprocess
+
+from ..helper import dputhelper
+
+
+class ProcessAsChannelAdapter(object):
+ """ Wrap a process as a paramiko Channel. """
+
+ def __init__(self, argv):
+ # Need to use a socket for some unknown reason
+ self.__socket, subproc_sock = socket.socketpair()
+ self.__proc = subprocess.Popen(argv, stdin=subproc_sock,
+ stdout=subproc_sock)
+
+ def get_name(self):
+ """ Return a simple name for the adapter. """
+ return "ProcessAsChannelAdapter"
+
+ def send(self, data):
+ """ Send a number of bytes to the subprocess. """
+ return self.__socket.send(data)
+
+ def recv(self, num_bytes):
+ """ Receive a number of bytes from the subprocess. """
+ try:
+ return self.__socket.recv(num_bytes)
+ except socket.error as e:
+ if e.args[0] in (errno.EPIPE, errno.ECONNRESET, errno.ECONNABORTED,
+ errno.EBADF):
+ # Connection has closed. Paramiko expects an empty string in
+ # this case, not an exception.
+ return ''
+ raise
+
+ def close(self):
+ """ Close and wait for process to finish. """
+ self.__socket.close()
+ self.__proc.terminate()
+ self.__proc.wait()
+
+
+def get_ssh_command_line(login, fqdn):
+ """ Gather a command line for connection to a server. """
+ return ["ssh",
+ "-oForwardX11 no",
+ "-oForwardAgent no",
+ "-oPermitLocalCommand no",
+ "-oClearAllForwardings yes",
+ "-oProtocol 2",
+ "-oNoHostAuthenticationForLocalhost yes",
+ "-l", login,
+ "-s", "--", fqdn, "sftp"]
+
+
+def copy_file(sftp_client, local_path, remote_path, debug, progress):
+ """ Upload a single file. """
+ with open(local_path, 'rb') as fileobj:
+ if progress:
+ try:
+ size = os.stat(local_path).st_size
+ except Exception:
+ size = -1
+ if debug:
+ sys.stdout.write(
+ "D: Determining size of file '%s' failed\n"
+ % local_path)
+
+ fileobj = dputhelper.FileWithProgress(fileobj, ptype=progress,
+ progressf=sys.stdout,
+ size=size)
+
+ # TODO: Do atomic?
+ with sftp_client.file(remote_path, "w") as remote_fileobj:
+ while True:
+ data = fileobj.read(4096)
+ if not data:
+ break
+ remote_fileobj.write(data)
+
+
+def upload(fqdn, login, incoming, files, debug, compress, progress=0):
+ """ Upload the files via SFTP.
+
+ Requires paramiko for SFTP protocol, but uses the ssh binary
+ for setting up the connection so we get proper prompts for
+ authentication, unknown hosts and other stuff.
+
+ """
+ try:
+ import paramiko.sftp_client
+ except Exception as e:
+ sys.stdout.write(
+ "E: paramiko must be installed to use sftp transport.\n")
+ sys.exit(1)
+
+ if not login or login == '*':
+ login = os.getenv("USER")
+
+ if not incoming.endswith("/"):
+ incoming = "%s/" % incoming
+
+ try:
+ channel = ProcessAsChannelAdapter(get_ssh_command_line(login,
+ fqdn))
+ sftp_client = paramiko.sftp_client.SFTPClient(channel)
+ except Exception as e:
+ sys.stdout.write("%s\nE: Error connecting to remote host.\n" % e)
+ sys.exit(1)
+
+ try:
+ for local_path in files:
+ path_to_package, base_filename = os.path.split(local_path)
+ remote_path = os.path.join(incoming, base_filename)
+ sys.stdout.write(" Uploading %s: " % base_filename)
+ sys.stdout.flush()
+ try:
+ copy_file(sftp_client, local_path, remote_path,
+ debug, progress)
+ except Exception as e:
+ sys.stdout.write("\n%s\nE: Error uploading file.\n" % e)
+ sys.exit(1)
+ sys.stdout.write("done.\n")
+ finally:
+ channel.close()
+
+
+# Copyright © 2006-2018 Canonical Ltd.
+# Copyright © 2006 Robey Pointer <ro...@lag.net>
+# (parts of ProcessAsChannelAdapter)
+#
+# Authors: Cody A.W. Somerville <cody.somervi...@canonical.com>
+# Julian Andres Klode <julian.kl...@canonical.com>
+#
+# This is free software: you may copy, modify, and/or distribute this work
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; version 3 of that license or any later version.
+# No warranty expressed or implied. See the file ‘LICENSE.GPL-3’ for details.
+
+
+# Local variables:
+# coding: utf-8
+# mode: python
+# End:
+# vim: fileencoding=utf-8 filetype=python :
--
2.15.1
>From c289098b259d4063fb3bf809a48c69ab9aa8bc25 Mon Sep 17 00:00:00 2001
From: Julian Andres Klode <julian.kl...@canonical.com>
Date: Thu, 11 Jan 2018 09:11:05 +0100
Subject: [PATCH 2/6] dput.cf: Adjust upload paths for Ubuntu and ppas
The default upload paths changed a while ago, let's bring that
up to date.
LP: #1340130
---
dput.cf | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dput.cf b/dput.cf
index 764025b..d328686 100644
--- a/dput.cf
+++ b/dput.cf
@@ -78,14 +78,14 @@ pre_upload_command = /usr/share/dput/helper/security-warning
[ubuntu]
fqdn = upload.ubuntu.com
method = ftp
-incoming = /
+incoming = /ubuntu
login = anonymous
[ppa]
fqdn = ppa.launchpad.net
method = ftp
# replace <launchpad-id> with your Launchpad ID
-incoming = ~<launchpad-id>/ubuntu
+incoming = ~<launchpad-id>
login = anonymous
[mentors]
--
2.15.1
