Thank you Alexander, Do you suggest then that we use the following? INCLUDES="—include=RX \/.*\/Synch\/.*"
And INCLUDES="—include=RX \/.*\/Saved\/.*" INCLUDES+="—include=RX \/.*\/Error\/.*" INCLUDES+="—include=RX \/.*\/Reports\/.*" And INCLUDES="—include=RX \/.*\/Other\/.*" AGS From: Alexander Lukyanov [mailto:lavv...@gmail.com] Sent: Friday, June 02, 2017 12:11 AM To: Alan Steel <a...@visagemobile.com>; lftp@uniyar.ac.ru Subject: Re: [lftp] Review of script for syncing HostedFTP and local server The --include-glob options look incorrect. They should not be given an absolute path, but rather a relative one. Even more, the glob pattern is matched against just a few last path components (separated by slashes), depending on the number of slashes in the pattern. The regex patterns are different, they are matched against the complete relative path, starting with the base mirrored directory. пт, 2 июня 2017, 5:04 Alan Steel <a...@visagemobile.com<mailto:a...@visagemobile.com>>: Hello, We are a SaaS company, migrating our colocation environment to AWS. As part of that, we are changing how customers will provide us with files to be processed by our application. Currently they SSH to a server in our colo, and place their files in one of their subdirectories. When we migrate to AWS, our customers will instead STFP upload their files to a hosted FTP provider. These files will be synchronized with two AWS-based servers, ProdN01IL01 and ProdN02IL01, each of which will be running an LFTP script. Description – On each of the two AWS servers, each customer’s directories will be under /opt/ftpusers/home/ Each of these will have 6 subdirectories – as described below. Some of these directories are to receive files from the customer, some are for our application’s use, and some are for file sharing between us and our customers. At the hosted FTP provider we are going to make the following directory structure… /Servers /ProdN01IL01 /[Customer1] /error /other /reports /saved /synch /[Customer2] /error /other /reports /saved /synch….(and so on) /ProdN02IL01 /[Customer51] /error /other /reports /saved /synch /[Customer52] /error /other /reports /saved /synch….(and so on) Note: We don’t need /inbox subdirectories on HostedFTP, because they are only used by our application. The script -- Below is the script we are thinking of using. We invite comments and suggestions. Thank you! #!/bin/bash FTPPROTOCOL=”sftp” FTPHOST=”ourcompany.hostedftp.com<http://ourcompany.hostedftp.com>” FTPUSER=”hftp.itaccou...@ourcompany.com<mailto:hftp.itaccou...@ourcompany.com>” FTPPASS=”ourcompanyftppassword” FTPURL="$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTPHOST" FTPREMOTEFOLDER=”/Servers/ProdN01IL01” (or ProdN02IL01, depending on the server) FTPLOCALFOLDER=”/opt/ftpusers/home” FTPLOGFILE=”/var/log/lftpsyncs.log” # The “Synch” subdirectories -- where customers put files for processing by our application # Move all files from HostedFTP “Synch” directories to local server Synch directories, the delete the source files from HostedFTP Synch directories # “Regular” mirror, with --Remove-source-files EXCLUDES="--exclude-glob .*" EXCLUDES+=" --exclude-glob .*/" INCLUDES="--include-glob /opt/ftpusers/home/*/Synch/*" lftp -e " open '$FTPURL '; lcd $FTPLOCALFOLDER; cd $FTPREMOTEFOLDER; mirror --continue --only-newer --Remove-source-files --verbose=3 --no-perms --parallel=8 $EXCLUDES $INCLUDES” # The “Saved” subdirectories – where our application puts files from Inbox, after successfully processing them # The “Error” subdirectories -- where our application puts files from Inbox, if it fails to process them # The “Reports” subdirectories – where our application puts reports # Sync-up files from local “Saved”, “Error” and “Reports” directories to HostedFTP, deleting HostedFTP files if they are not present on the local server # “Reverse” mirror, with deleting of remote files not present locally EXCLUDES="--exclude-glob .*" EXCLUDES+=" --exclude-glob .*/" INCLUDES="--include-glob /opt/ftpusers/home/*/Saved/*" INCLUDES+=" --include-glob /opt/ftpusers/home/*/Error/*" INCLUDES+=" --include-glob /opt/ftpusers/home/*/Reports/*" lftp -e " open '$FTPURL '; lcd $FTPLOCALFOLDER; cd $FTPREMOTEFOLDER; mirror --reverse --continue --only-newer --delete --verbose=3 --no-perms --parallel=8 $EXCLUDES $INCLUDES” # The “Other” subdirectories -- – where customers or staff put various files # Sync-down files in HostedFTP “Other” directories to the local server # “Regular” mirror EXCLUDES="--exclude-glob .*" EXCLUDES+=" --exclude-glob .*/" INCLUDES="--include-glob /opt/ftpusers/home/*/Other/*" lftp -e " open '$ FTPURL '; lcd $ FTPLOCALFOLDER; cd $ FTPREMOTEFOLDER; mirror --continue --only-newer --verbose=3 --no-perms --parallel=8 $EXCLUDES $INCLUDES” AGS _______________________________________________ lftp mailing list lftp@uniyar.ac.ru<mailto:lftp@uniyar.ac.ru> http://univ.uniyar.ac.ru/mailman/listinfo/lftp
_______________________________________________ lftp mailing list lftp@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp