Re: get todays files
Tim Chase wrote:
This looks very good and I have tested successfully, but is there a
way I can set the today to automatically become todays date in that
format?
Yes...see the python datetime module[1]...particularly the strftime()
call on date/datetime objects.
-tkc
[1]
http://docs.python.org/library/datetime.html
I know this will sound like I am being very cheeky, but is there a way
you can make this for where the ftp server is actually windows server?
The script runs and it shows the date the file was modified, but still
downloads them, I have attached the script I am now using with your code in.
import os
from ftplib import FTP
import subprocess
from datetime import datetime
from time import gmtime, strftime
curr_date = strftime(%d %B %Y, gmtime())
# config
ftp_uname = ''
ftp_pwd = ''
ftp_server = ''
# end of config
fileroot = 'DBBackup_'
os.chdir('c:/')
files = []
logfile = open('c:/dbbackup/getmdbackups.log','a+')
def makelist(line):
if (line.startswith(fileroot)):
fs = [line]
files.append(fs)
def log(line):
ll = str(datetime.now()) + ' : ' + str(line)
print ll
logfile.write(ll + '\n')
def fileexists(ff, size):
if (os.path.exists(ff)):
stat = os.stat(ff)
if (stat.st_size == size):
return True
return False
try:
# first connect using ftp to get a list of valid backup failes available
log('Connecting to ftp server')
ftp = FTP(ftp_server)
ftp.set_pasv(False)
#ftp.set_debuglevel(2)
resp = ftp.login(ftp_uname,ftp_pwd)
log(resp)
ftp.retrlines('NLST',makelist)
log(str(files))
ftp.quit()
# fetch files in a loop using wget.
for ff in files:
ftp = FTP(ftp_server)
ftp.set_pasv(False)
resp = ftp.login(ftp_uname,ftp_pwd)
log(resp)
size = ftp.size(ff[0])
log('Size of server file = ' + str(size))
#ftp.quit()
try:
if (not fileexists(ff[0],size)):
results = ftp.sendcmd(MDTM %s % ff[0])
code, stamp = results.split(None, 1)
assert code == 213, Unexpected result
print %s was modified on %s % (ff[0], stamp)
today = curr_date
if stamp[:8] == today:
log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1
else:
log('File ' + ff[0] + ' already exists locally, not
transferring')
ff[1] = 1
except Exception, e:
log(str(e))
log('Transfer complete')
# delete the server files that have been transferred or are already
here with the right filesize.
for ff in files:
if (ff[1] == 1):
log('delete ' + ff[0])
except Exception,e:
log(str(e))
# clean up temp files
log('Finished.')
logfile.close()
--
http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
I know this will sound like I am being very cheeky, but is there a way
you can make this for where the ftp server is actually windows server?
For Windows Server, I don't have a Windows FTP server to test
with -- I've got the company Linux server, and the previous
testing site I used (I think I used ftp.mozilla.org) which also
likely runs some flavor of Linux. Neither supports the NLST from
my testing.
curr_date = strftime(%d %B %Y, gmtime())
The first thing I noticed was that your strftime formating needs
to match the format of the date that comes back from the FTP
site. In my test, that was MMDD. As such, your %d %B %Y
would likely need to be %Y%m%d.
ftp.retrlines('NLST',makelist)
The servers I tried didn't support the NLST command so I can't
exactly follow along here. However assuming that it correctly
populates the list of files here
for ff in files:
correctly, that's immaterial to me.
ftp = FTP(ftp_server)
ftp.set_pasv(False)
resp = ftp.login(ftp_uname,ftp_pwd)
Just curious why you're logging into the server each pass through
the loop -- I'd just connect once at the beginning of the loop,
pull the files, and then disconnect at the end of the loop.
assert code == 213, Unexpected result
Does this assert fail at any point?
if stamp[:8] == today:
Given the above date-formatting, this should fail *every* time
unless your FTP server is returning the date in some format other
than MMDDhhmmss
It's hard to pinpoint actual problems as this block of code has
been modified, or doesn't run...there's some bogus indentation in
your post:
log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1
else:
log('File ' + ff[0] + ' already exists locally, not
transferring')
because this else is hanging oddly. Additionally, the FTP
object has methods for downloading the content of a file, so I'd
not bother shelling out to wget as an additional dependency.
-tkc
--
http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
Tim Chase wrote:
I know this will sound like I am being very cheeky, but is there a
way you can make this for where the ftp server is actually windows
server?
For Windows Server, I don't have a Windows FTP server to test with --
I've got the company Linux server, and the previous testing site I
used (I think I used ftp.mozilla.org) which also likely runs some
flavor of Linux. Neither supports the NLST from my testing.
curr_date = strftime(%d %B %Y, gmtime())
The first thing I noticed was that your strftime formating needs to
match the format of the date that comes back from the FTP site. In my
test, that was MMDD. As such, your %d %B %Y would likely need
to be %Y%m%d.
ftp.retrlines('NLST',makelist)
The servers I tried didn't support the NLST command so I can't exactly
follow along here. However assuming that it correctly populates the
list of files here
for ff in files:
correctly, that's immaterial to me.
ftp = FTP(ftp_server)
ftp.set_pasv(False)
resp = ftp.login(ftp_uname,ftp_pwd)
Just curious why you're logging into the server each pass through the
loop -- I'd just connect once at the beginning of the loop, pull the
files, and then disconnect at the end of the loop.
I support it would be somewhat better to download in a 'bulk' download
rather that a file at a time, this script was not written by me, I am
just the developer who has to make a new or modify the old one.
assert code == 213, Unexpected result
Does this assert fail at any point?
Nope, nothing shows up in my logs or on screen.
if stamp[:8] == today:
Given the above date-formatting, this should fail *every* time unless
your FTP server is returning the date in some format other than
MMDDhhmmss
This line appears to just get missed in the code, I think it might be
one of the problems when it downloads all the files.
It's hard to pinpoint actual problems as this block of code has been
modified, or doesn't run...there's some bogus indentation in your post:
log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1
else:
log('File ' + ff[0] + ' already exists locally, not
transferring')
because this else is hanging oddly. Additionally, the FTP object
has methods for downloading the content of a file, so I'd not bother
shelling out to wget as an additional dependency
I am running kubuntu 8.04 and have edited the code in kate, It seemed to
indent on every line, so I just 'pulled' it back a little.
.
-tkc
Thanks for you comments, I think I will try and start from scratch and
see what I get.
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
Andrew D wrote: I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. I am really not sure about how to do this, but it is quite important to me, so all help is highly appreciated! IIRC there's an ftpmirror script in the Tools directory (if you're on Windows - Linux/Unix users have to download the source). I adapted it to several different purposes. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
On Dec 10, 5:55 pm, Steve Holden [EMAIL PROTECTED] wrote: Andrew D wrote: I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. I am really not sure about how to do this, but it is quite important to me, so all help is highly appreciated! IIRC there's an ftpmirror script in the Tools directory (if you're on Windows - Linux/Unix users have to download the source). I adapted it to several different purposes. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Thanks Steve, I don't want to sound rude here, but I really want some code to add to my script so that it will only download todays files. Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
Andrew D wrote: On Dec 10, 5:55 pm, Steve Holden [EMAIL PROTECTED] wrote: Andrew D wrote: I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. I am really not sure about how to do this, but it is quite important to me, so all help is highly appreciated! IIRC there's an ftpmirror script in the Tools directory (if you're on Windows - Linux/Unix users have to download the source). I adapted it to several different purposes. Thanks Steve, I don't want to sound rude here, but I really want some code to add to my script so that it will only download todays files. That's OK. Maybe somebody's already got that code ready to go. I don't ... regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
On Dec 10, 6:13 pm, Steve Holden [EMAIL PROTECTED] wrote: Andrew D wrote: On Dec 10, 5:55 pm, Steve Holden [EMAIL PROTECTED] wrote: Andrew D wrote: I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. I am really not sure about how to do this, but it is quite important to me, so all help is highly appreciated! IIRC there's an ftpmirror script in the Tools directory (if you're on Windows - Linux/Unix users have to download the source). I adapted it to several different purposes. Thanks Steve, I don't want to sound rude here, but I really want some code to add to my script so that it will only download todays files. That's OK. Maybe somebody's already got that code ready to go. I don't ... regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Thank you anyway Steve! Lets hope someone has the code ;) Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. Use Python's ftp module and send the MDTM command to get back the timestamp of the filename. Insecurely, this would look something like from ftplib import FTP hostname = ftp.mozilla.org conn = FTP(hostname) user = anonymous password = [EMAIL PROTECTED] conn.login(user, password) filename = pub/README results = conn.sendcmd(MDTM %s % filename) code, stamp = results.split(None, 1) assert code == 213, Unexpected result print %s was modified on %s % (filename, stamp) today = '20081210' if stamp[:8] == today: process(filename) else: print ignoring, filename The MDTM command is not part of the core RFC-959, but rather the RFC-3659[1] so you might run across some servers that don't support it. You can read more about the Python ftplib module at [2] which would be where you want to read up on pulling back a listing of the directory of file-names to check. There is a NLST command (I don't have a server handy that supports this command). The LIST command returns pretty/readable information that's not quite so machine-parsing friendly (at least in a cross-FTP-server sort of way). However, that part, I leave as an exercise for the reader along with the complications of the today bit. Oh, SteveH, I checked your FTP-cloning source in my Python dir, and it doesn't look like it does anything regarding file-times in it, so that may have been a red-herring. Unless you've added something since the ver. I've got here. -tkc [1] http://en.wikipedia.org/wiki/List_of_FTP_commands [2] http://www.python.org/doc/2.5.2/lib/ftp-objects.html -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
On Dec 10, 6:55 pm, Tim Chase [EMAIL PROTECTED] wrote: I have a script that will login to my ftp server and download all the backup files, but I want it to only download the files that were created today, e.g. if I ran the script today I want it to only fetch files created today. Use Python's ftp module and send the MDTM command to get back the timestamp of the filename. Insecurely, this would look something like from ftplib import FTP hostname = ftp.mozilla.org conn = FTP(hostname) user = anonymous password = [EMAIL PROTECTED] conn.login(user, password) filename = pub/README results = conn.sendcmd(MDTM %s % filename) code, stamp = results.split(None, 1) assert code == 213, Unexpected result print %s was modified on %s % (filename, stamp) today = '20081210' if stamp[:8] == today: process(filename) else: print ignoring, filename The MDTM command is not part of the core RFC-959, but rather the RFC-3659[1] so you might run across some servers that don't support it. You can read more about the Python ftplib module at [2] which would be where you want to read up on pulling back a listing of the directory of file-names to check. There is a NLST command (I don't have a server handy that supports this command). The LIST command returns pretty/readable information that's not quite so machine-parsing friendly (at least in a cross-FTP-server sort of way). However, that part, I leave as an exercise for the reader along with the complications of the today bit. Oh, SteveH, I checked your FTP-cloning source in my Python dir, and it doesn't look like it does anything regarding file-times in it, so that may have been a red-herring. Unless you've added something since the ver. I've got here. -tkc [1]http://en.wikipedia.org/wiki/List_of_FTP_commands [2]http://www.python.org/doc/2.5.2/lib/ftp-objects.html This looks very good and I have tested successfully, but is there a way I can set the today to automatically become todays date in that format? Thanks though, this is what I was looking for! Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
This looks very good and I have tested successfully, but is there a way I can set the today to automatically become todays date in that format? Yes...see the python datetime module[1]...particularly the strftime() call on date/datetime objects. -tkc [1] http://docs.python.org/library/datetime.html -- http://mail.python.org/mailman/listinfo/python-list
Re: get todays files
Tim Chase wrote: This looks very good and I have tested successfully, but is there a way I can set the today to automatically become todays date in that format? Yes...see the python datetime module[1]...particularly the strftime() call on date/datetime objects. -tkc [1] http://docs.python.org/library/datetime.html Thanks Tim, I got it in there myself. Thanks for all your help with this and the links proved very useful and interesting reads. Andrew -- http://mail.python.org/mailman/listinfo/python-list
