Well then, let's try setting the PROT setting, even though the data will
not be secure. Try this version.
On Fri, May 29, 2020 at 6:15 AM Richard Cropper <richardcropper...@gmail.com>
wrote:
> I'm on v 4.0.0 - it can't have been more than a week since I installed it
>
> Yes I did try setting secure_data to false. With the previous version of
> ftpupload.py I then get
>
> May 29 14:10:56 raspberrypi weewx[4780] ERROR weeutil.ftpupload: Attempt
> #1. Failed uploading /public_html/weewx/weekwind.png to
> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be
> opened with this PROT setting.
>
> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: Attempt
> #2. Failed uploading /public_html/weewx/weekwind.png to
> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be
> opened with this PROT setting.
>
> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: Attempt
> #3. Failed uploading /public_html/weewx/weekwind.png to
> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be
> opened with this PROT setting.
>
> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: Failed to
> upload file /public_html/weewx/weekwind.png
>
>
> Regards
>
> Richard
> On Friday, May 29, 2020 at 12:40:44 PM UTC+1, Tom Keffer wrote:
>>
>> The file I posted was to fix a problem introduced in V4.1, which I had
>> assumed you were using. Apparently not. Disregard the file. Sorry for any
>> confusion.
>>
>> Did you try setting secure_data to false?
>>
>> -tk
>>
>> On Fri, May 29, 2020 at 4:22 AM Richard Cropper <richardc...@gmail.com>
>> wrote:
>>
>>> Incidentally, I've been nowhere near reportengine.py
>>>
>>> On Friday, May 29, 2020 at 12:19:38 PM UTC+1, Richard Cropper wrote:
>>>>
>>>> Hi Tom
>>>>
>>>> This what I get with the new ftpupload.py
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] INFO weewx.reportengine: Copied
>>>> 5 files to /var/www/html/weewx
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> Caught unrecoverable exception in generator
>>>> 'weewx.reportengine.FtpGenerator'
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** __init__() got an unexpected keyword argument 'max_tries'
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** Traceback (most recent call last):
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 197, in run
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** obj.start()
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 280, in start
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** self.run()
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 320, in run
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** secure_data=to_bool(self.skin_dict.get('secure_data', True)))
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** TypeError: __init__() got an unexpected keyword argument
>>>> 'max_tries'
>>>>
>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine:
>>>> **** Generator terminated
>>>>
>>>> I get the same message whether secure_data is set to True or False.
>>>>
>>>> Regards
>>>>
>>>> Richard
>>>>
>>>>
>>>>
>>>>
>>>> On Thursday, May 28, 2020 at 9:28:49 PM UTC+1, Tom Keffer wrote:
>>>>>
>>>>> I just noticed that in refactoring the ftp uploader, the function
>>>>> _make_remote_dir() never gets called! Try this version of
>>>>> weeutil/ftpupload.py.
>>>>>
>>>>> -tk
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 28, 2020 at 12:50 PM Tom Keffer <tke...@gmail.com> wrote:
>>>>>
>>>>>> Hard to say, but I think you're getting hit by issue #284
>>>>>> <https://github.com/weewx/weewx/issues/284>. Try adding an option
>>>>>> secure_data and setting it to false:
>>>>>>
>>>>>> [[FTP]]
>>>>>> ...
>>>>>> secure_ftp = True
>>>>>> secure_data = False
>>>>>>
>>>>>> -tk
>>>>>>
>>>>>> On Thu, May 28, 2020 at 9:36 AM Richard Cropper <
>>>>>> richardc...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Tom
>>>>>>>
>>>>>>> I had in fact tried those steps but did so again, without success.
>>>>>>>
>>>>>>> Suspecting that the directory might be the cause even though its
>>>>>>> permissions looked good, I deleted the weewx directory and then created
>>>>>>> it
>>>>>>> again.
>>>>>>>
>>>>>>> This had made a difference but not solved the problem.
>>>>>>>
>>>>>>> Here's an extract from the log:
>>>>>>>
>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload:
>>>>>>> Attempt #1. Failed uploading /public_html/weewx/daytempfeel.png to
>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>>>>>>
>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload:
>>>>>>> Attempt #2. Failed uploading /public_html/weewx/daytempfeel.png to
>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>>>>>>
>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload:
>>>>>>> Attempt #3. Failed uploading /public_html/weewx/daytempfeel.png to
>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>>>>>>
>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload:
>>>>>>> Failed to upload file /public_html/weewx/daytempfeel.png
>>>>>>>
>>>>>>> I have tried all possible file paths again, as you suggest but I
>>>>>>> always get this broken pipe message. If I extend the file path any
>>>>>>> higher,
>>>>>>> I get the 'cannot open this file' message.
>>>>>>>
>>>>>>> I have been able to upload the index file from /var/www/html/weewx
>>>>>>> to the weewx directory on my server using curl.
>>>>>>>
>>>>>>> Richard
>>>>>>>
>>>>>>> On Wednesday, May 27, 2020 at 7:56:28 PM UTC+1, Tom Keffer wrote:
>>>>>>>>
>>>>>>>> Two things to try:
>>>>>>>>
>>>>>>>> 1. Make sure the FTP username has write permissions on the server.
>>>>>>>>
>>>>>>>> 2. Try different variations on the FTP path. For example,
>>>>>>>>
>>>>>>>> path = public_html/weewx/ # What you have
>>>>>>>> path = public_html/weewx
>>>>>>>> path = /public_html/weewx/
>>>>>>>> path = /public_html/weewx
>>>>>>>> path = weewx
>>>>>>>> path = /weewx
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 27, 2020 at 10:02 AM Richard Cropper <
>>>>>>>> richardc...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Tom
>>>>>>>>>
>>>>>>>>> Yes all the files are in /var/www/html/weewx
>>>>>>>>>
>>>>>>>>> If I understand you correctly, this has more to do with the file
>>>>>>>>> path of the remote host server.
>>>>>>>>>
>>>>>>>>> I am at a loss to understand what is going on at that end.
>>>>>>>>>
>>>>>>>>> I can at least connect but I either get a message as you have seen
>>>>>>>>> or a 'broken pipe' message if I play around with the file path.
>>>>>>>>>
>>>>>>>>> I guess this is more to do with the remote server than weewx but
>>>>>>>>> thanks for your advice.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> Richard
>>>>>>>>> On Wednesday, May 27, 2020 at 3:17:35 PM UTC+1, Tom Keffer wrote:
>>>>>>>>>>
>>>>>>>>>> Hello, Richard
>>>>>>>>>>
>>>>>>>>>> Unfortunately, the error message is misleading (and should be
>>>>>>>>>> changed). When it says
>>>>>>>>>>
>>>>>>>>>> Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>> file or directory
>>>>>>>>>>
>>>>>>>>>> what it means is it could not upload to the *target*
>>>>>>>>>> public_html/weewx/monthtempfeel.png*.* The unfortunate part is
>>>>>>>>>> that the error message doesn't give the source.
>>>>>>>>>>
>>>>>>>>>> Take a look in /var/www/html/weewx and make sure the files you
>>>>>>>>>> expect to be in there are actually in there.
>>>>>>>>>>
>>>>>>>>>> -tk
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 27, 2020 at 5:50 AM Richard Cropper <
>>>>>>>>>> richardc...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>> Having secured a connection, I still don't seem to be able to
>>>>>>>>>>> upload files
>>>>>>>>>>>
>>>>>>>>>>> I get the following error message on the log
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #1. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #2. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #3. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Failed to upload file public_html/weewx/monthtempfeel.png
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #1. Failed uploading public_html/weewx/yearwind.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #2. Failed uploading public_html/weewx/yearwind.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #3. Failed uploading public_html/weewx/yearwind.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Failed to upload file public_html/weewx/yearwind.png
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #1. Failed uploading public_html/weewx/statistics.html to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #2. Failed uploading public_html/weewx/statistics.html to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #3. Failed uploading public_html/weewx/statistics.html to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Failed to upload file public_html/weewx/statistics.html
>>>>>>>>>>>
>>>>>>>>>>> May 27 13:30:41 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>>>>>>> Attempt #1. Failed uploading public_html/weewx/weekbarometer.png to
>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Here's the relevant bit of my weewx.conf:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [[FTP]]
>>>>>>>>>>>
>>>>>>>>>>> # FTP'ing the results to a webserver is treated as just
>>>>>>>>>>> another report,
>>>>>>>>>>>
>>>>>>>>>>> # albeit one with an unusual report generator!
>>>>>>>>>>>
>>>>>>>>>>> skin = Ftp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # If you wish to use FTP, set "enable" to "true", then
>>>>>>>>>>>
>>>>>>>>>>> # fill out the next four lines.
>>>>>>>>>>>
>>>>>>>>>>> # Use quotes around passwords to guard against parsing
>>>>>>>>>>> errors.
>>>>>>>>>>>
>>>>>>>>>>> enable = true
>>>>>>>>>>>
>>>>>>>>>>> user =[user name]
>>>>>>>>>>>
>>>>>>>>>>> password = [password]
>>>>>>>>>>>
>>>>>>>>>>> server = [server name] # The ftp server name, e.g,
>>>>>>>>>>> www.myserver.org
>>>>>>>>>>>
>>>>>>>>>>> path = public_html/weewx/ # The destination
>>>>>>>>>>> directory, e.g., /weather
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Set to True for an FTP over TLS (FTPS) connection. Not
>>>>>>>>>>> all servers
>>>>>>>>>>>
>>>>>>>>>>> # support this.
>>>>>>>>>>>
>>>>>>>>>>> secure_ftp = True
>>>>>>>>>>>
>>>>>>>>>>> secure_data = True
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # To upload files from something other than what
>>>>>>>>>>> HTML_ROOT is set
>>>>>>>>>>>
>>>>>>>>>>> # to above, specify a different HTML_ROOT here.
>>>>>>>>>>>
>>>>>>>>>>> HTML_ROOT = /var/www/html/weewx/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Most FTP servers use port 21
>>>>>>>>>>>
>>>>>>>>>>> port = 21
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Set to 1 to use passive mode, zero for active mode
>>>>>>>>>>>
>>>>>>>>>>> passive = 1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The connection is FTP over TLS.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> My weather data is from an Aercus Weather Sleuth using
>>>>>>>>>>> interceptor on a Raspberry pi.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It's as if FTP is looking for the wrong path on my pi but maybe
>>>>>>>>>>> I misunderstand the error message. That's why I tried removing the
>>>>>>>>>>> hash
>>>>>>>>>>> before HTML_ROOT but I got the same result with the hash.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Otherwise very pleased with weewx which I can view on my pi
>>>>>>>>>>> using nginx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Any help gratefully received.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Richard
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "weewx-user" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to weewx...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com
>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "weewx-user" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to weewx...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "weewx-user" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to weewx...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com
>>>>>>> <https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/c53bbd3e-00ee-4291-872d-6de13557e08c%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-user/c53bbd3e-00ee-4291-872d-6de13557e08c%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/67c65342-a58e-440b-a92a-8ed9dd6b3395%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-user/67c65342-a58e-440b-a92a-8ed9dd6b3395%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/CAPq0zEBQEky58SX8V4CaE8ot7ecu25rGHkYFSts%2BF-bk7Bdy_g%40mail.gmail.com.
#
# Copyright (c) 2009-2020 Tom Keffer <tkef...@gmail.com>
#
# See the file LICENSE.txt for your full rights.
#
"""For uploading files to a remove server via FTP"""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import with_statement
import ftplib
import logging
import os
import sys
import time
from six.moves import cPickle
log = logging.getLogger(__name__)
class FtpUpload(object):
"""Uploads a directory and all its descendants to a remote server.
Keeps track of when a file was last uploaded, so it is uploaded only
if its modification time is newer."""
def __init__(self, server,
user, password,
local_root, remote_root,
port=21,
name="FTP",
passive=True,
secure=False,
debug=0,
secure_data=True,
reuse_ssl=False):
"""Initialize an instance of FtpUpload.
After initializing, call method run() to perform the upload.
server: The remote server to which the files are to be uploaded.
user,
password : The user name and password that are to be used.
name: A unique name to be given for this FTP session. This allows more
than one session to be uploading from the same local directory. [Optional.
Default is 'FTP'.]
passive: True to use passive mode; False to use active mode. [Optional.
Default is True (passive mode)]
secure: Set to True to attempt an FTP over TLS (FTPS) session.
debug: Set to 1 for extra debug information, 0 otherwise.
secure_data: If a secure session is requested (option secure=True),
should we attempt a secure data connection as well? This option is useful
due to a bug in the Python FTP client library. See Issue #284.
[Optional. Default is True]
reuse_ssl: Work around a bug in the Python library that closes ssl sockets that should
be reused. See https://bit.ly/3dKq4JY [Optional. Default is False]
"""
self.server = server
self.user = user
self.password = password
self.local_root = os.path.normpath(local_root)
self.remote_root = os.path.normpath(remote_root)
self.port = port
self.name = name
self.passive = passive
self.secure = secure
self.debug = debug
self.secure_data = secure_data
self.reuse_ssl = reuse_ssl
if self.reuse_ssl and sys.version < '3.6':
raise ValueError("Reusing an SSL connection requires Python version 3.6 or greater")
def run(self):
"""Perform the actual upload.
returns: the number of files uploaded."""
# Get the timestamp and members of the last upload:
timestamp, fileset = self.get_last_upload()
n_uploaded = 0
try:
if self.secure:
log.debug("Attempting secure connection to %s" % self.server)
if self.reuse_ssl:
# Activate the workaround for the Python ftplib library.
from ssl import SSLSocket
class ReusedSslSocket(SSLSocket):
def unwrap(self):
pass
class WeeFTPTLS(ftplib.FTP_TLS):
"""Explicit FTPS, with shared TLS session"""
def ntransfercmd(self, cmd, rest=None):
conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
if self._prot_p:
conn = self.context.wrap_socket(conn,
server_hostname=self.host,
session=self.sock.session)
conn.__class__ = ReusedSslSocket
return conn, size
log.debug("Reusing SSL connections.")
ftp_server = WeeFTPTLS()
else:
ftp_server = ftplib.FTP_TLS()
else:
log.debug("Attempting connection to %s" % self.server)
ftp_server = ftplib.FTP()
if self.debug:
ftp_server.set_debuglevel(self.debug)
ftp_server.set_pasv(self.passive)
ftp_server.connect(self.server, self.port)
ftp_server.login(self.user, self.password)
if self.secure:
ftp_server.prot_p()
log.debug("Secure data connection to %s" % self.server)
else:
log.debug("Connected to %s" % self.server)
# Walk the local directory structure
for (dirpath, unused_dirnames, filenames) in os.walk(self.local_root):
# Strip out the common local root directory. What is left
# will be the relative directory both locally and remotely.
local_rel_dir_path = dirpath.replace(self.local_root, '.')
if self._skip_this_dir(local_rel_dir_path):
continue
# This is the absolute path to the remote directory:
remote_dir_path = os.path.normpath(os.path.join(self.remote_root,
local_rel_dir_path))
# Make the remote directory if necessary:
_make_remote_dir(ftp_server, remote_dir_path)
# Now iterate over all members of the local directory:
for filename in filenames:
full_local_path = os.path.join(dirpath, filename)
# See if this file can be skipped:
if self._skip_this_file(timestamp, fileset, full_local_path):
continue
full_remote_path = os.path.join(remote_dir_path, filename)
stor_cmd = "STOR %s" % full_remote_path
with open(full_local_path, 'rb') as fd:
try:
ftp_server.storbinary(stor_cmd, fd)
except ftplib.all_errors as e:
# Unsuccessful. Log it, then reraise the exception
log.error("Failed uploading %s to server %s. Reason: '%s'"
% (full_local_path, self.server, e))
raise
# Success.
n_uploaded += 1
fileset.add(full_local_path)
log.debug("Uploaded file %s to %s" % (full_local_path, full_remote_path))
finally:
try:
ftp_server.quit()
except Exception:
pass
timestamp = time.time()
self.save_last_upload(timestamp, fileset)
return n_uploaded
def get_last_upload(self):
"""Reads the time and members of the last upload from the local root"""
timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name)
# If the file does not exist, an IOError exception will be raised.
# If the file exists, but is truncated, an EOFError will be raised.
# Either way, be prepared to catch it.
try:
with open(timestamp_file_path, "rb") as f:
timestamp = cPickle.load(f)
fileset = cPickle.load(f)
except (IOError, EOFError, cPickle.PickleError, AttributeError):
timestamp = 0
fileset = set()
# Either the file does not exist, or it is garbled.
# Either way, it's safe to remove it.
try:
os.remove(timestamp_file_path)
except OSError:
pass
return timestamp, fileset
def save_last_upload(self, timestamp, fileset):
"""Saves the time and members of the last upload in the local root."""
timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name)
with open(timestamp_file_path, "wb") as f:
cPickle.dump(timestamp, f)
cPickle.dump(fileset, f)
def _skip_this_dir(self, local_dir):
"""Determine whether to skip a directory."""
return os.path.basename(local_dir) in ('.svn', 'CVS')
def _skip_this_file(self, timestamp, fileset, full_local_path):
"""Determine whether to skip a specific file."""
filename = os.path.basename(full_local_path)
if filename[-1] == '~' or filename[0] == '#':
return True
if full_local_path not in fileset:
return False
if os.stat(full_local_path).st_mtime > timestamp:
return False
# Filename is in the set, and is up to date.
return True
def _make_remote_dir(ftp_server, remote_dir_path):
"""Make a remote directory if necessary."""
try:
ftp_server.mkd(remote_dir_path)
except ftplib.all_errors as e:
# Got an exception. It might be because the remote directory already exists:
if sys.exc_info()[0] is ftplib.error_perm:
msg = str(e).strip()
# If a directory already exists, some servers respond with a '550' ("Requested
# action not taken") code, others with a '521' ("Access denied" or "Pathname
# already exists") code.
if msg.startswith('550') or msg.startswith('521'):
# Directory already exists
return
# It's a real error. Re-raise the exception.
raise
log.debug("Made directory %s" % remote_dir_path)
return
