Hello,
We are a SaaS company, migrating our colocation environment to AWS.
As part of that, we are changing how customers will provide us with files to be
processed by our application.
Currently they SSH to a server in our colo, and place their files in one of
their subdirectories.
When we migrate to AWS, our customers will instead STFP upload their files to a
hosted FTP provider.
These files will be synchronized with two AWS-based servers, ProdN01IL01 and
ProdN02IL01, each of which will be running an LFTP script.
Description -
On each of the two AWS servers, each customer's directories will be under
/opt/ftpusers/home/
Each of these will have 6 subdirectories - as described below.
Some of these directories are to receive files from the customer, some are for
our application's use, and some are for file sharing between us and our
customers.
At the hosted FTP provider we are going to make the following directory
structure...
/Servers
/ProdN01IL01
/[Customer1]
/error
/other
/reports
/saved
/synch
/[Customer2]
/error
/other
/reports
/saved
/synch....(and so on)
/ProdN02IL01
/[Customer51]
/error
/other
/reports
/saved
/synch
/[Customer52]
/error
/other
/reports
/saved
/synch....(and so on)
Note: We don't need /inbox subdirectories on HostedFTP, because they are only
used by our application.
The script --
Below is the script we are thinking of using.
We invite comments and suggestions.
Thank you!
#!/bin/bash
FTPPROTOCOL="sftp"
FTPHOST="ourcompany.hostedftp.com"
FTPUSER="hftp.itaccou...@ourcompany.com"
FTPPASS="ourcompanyftppassword"
FTPURL="$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTPHOST"
FTPREMOTEFOLDER="/Servers/ProdN01IL01" (or ProdN02IL01, depending on the
server)
FTPLOCALFOLDER="/opt/ftpusers/home"
FTPLOGFILE="/var/log/lftpsyncs.log"
# The "Synch" subdirectories -- where customers put files for processing by our
application
# Move all files from HostedFTP "Synch" directories to local server Synch
directories, the delete the source files from HostedFTP Synch directories
# "Regular" mirror, with --Remove-source-files
EXCLUDES="--exclude-glob .*"
EXCLUDES+=" --exclude-glob .*/"
INCLUDES="--include-glob /opt/ftpusers/home/*/Synch/*"
lftp -e "
open '$FTPURL ';
lcd $FTPLOCALFOLDER;
cd $FTPREMOTEFOLDER;
mirror --continue --only-newer --Remove-source-files --verbose=3 --no-perms
--parallel=8 $EXCLUDES $INCLUDES"
# The "Saved" subdirectories - where our application puts files from Inbox,
after successfully processing them
# The "Error" subdirectories -- where our application puts files from Inbox, if
it fails to process them
# The "Reports" subdirectories - where our application puts reports
# Sync-up files from local "Saved", "Error" and "Reports" directories to
HostedFTP, deleting HostedFTP files if they are not present on the local server
# "Reverse" mirror, with deleting of remote files not present locally
EXCLUDES="--exclude-glob .*"
EXCLUDES+=" --exclude-glob .*/"
INCLUDES="--include-glob /opt/ftpusers/home/*/Saved/*"
INCLUDES+=" --include-glob /opt/ftpusers/home/*/Error/*"
INCLUDES+=" --include-glob /opt/ftpusers/home/*/Reports/*"
lftp -e "
open '$FTPURL ';
lcd $FTPLOCALFOLDER;
cd $FTPREMOTEFOLDER;
mirror --reverse --continue --only-newer --delete --verbose=3 --no-perms
--parallel=8 $EXCLUDES $INCLUDES"
# The "Other" subdirectories -- - where customers or staff put various files
# Sync-down files in HostedFTP "Other" directories to the local server
# "Regular" mirror
EXCLUDES="--exclude-glob .*"
EXCLUDES+=" --exclude-glob .*/"
INCLUDES="--include-glob /opt/ftpusers/home/*/Other/*"
lftp -e "
open '$ FTPURL ';
lcd $ FTPLOCALFOLDER;
cd $ FTPREMOTEFOLDER;
mirror --continue --only-newer --verbose=3 --no-perms --parallel=8 $EXCLUDES
$INCLUDES"
AGS
_______________________________________________
lftp mailing list
lftp@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
