The --include-glob options look incorrect. They should not be given an
absolute path, but rather a relative one. Even more, the glob pattern is
matched against just a few last path components (separated by slashes),
depending on the number of slashes in the pattern. The regex patterns are
different, they are matched against the complete relative path, starting
with the base mirrored directory.
пт, 2 июня 2017, 5:04 Alan Steel :
> Hello,
>
>
>
> We are a SaaS company, migrating our colocation environment to AWS.
>
> As part of that, we are changing how customers will provide us with files
> to be processed by our application.
>
> Currently they SSH to a server in our colo, and place their files in one
> of their subdirectories.
>
> When we migrate to AWS, our customers will instead STFP upload their files
> to a hosted FTP provider.
>
> These files will be synchronized with two AWS-based servers, ProdN01IL01
> and ProdN02IL01, each of which will be running an LFTP script.
>
>
>
> Description –
>
>
>
> On each of the two AWS servers, each customer’s directories will be under
> /opt/ftpusers/home/
>
> Each of these will have 6 subdirectories – as described below.
>
> Some of these directories are to receive files from the customer, some are
> for our application’s use, and some are for file sharing between us and our
> customers.
>
>
>
> At the hosted FTP provider we are going to make the following directory
> structure…
>
> /Servers
>
> /ProdN01IL01
>
> /[Customer1]
>
> /error
>
> /other
>
> /reports
>
> /saved
>
> /synch
>
> /[Customer2]
>
> /error
>
> /other
>
> /reports
>
> /saved
>
> /synch….(and so on)
>
> /ProdN02IL01
>
> /[Customer51]
>
> /error
>
> /other
>
> /reports
>
> /saved
>
> /synch
>
> /[Customer52]
>
> /error
>
> /other
>
> /reports
>
> /saved
>
> /synch….(and so on)
>
>
>
> Note: We don’t need /inbox subdirectories on HostedFTP, because they are
> only used by our application.
>
>
>
> The script --
>
>
>
> Below is the script we are thinking of using.
>
> We invite comments and suggestions.
>
> Thank you!
>
>
>
> #!/bin/bash
>
>
>
> FTPPROTOCOL=”sftp”
>
> FTPHOST=”ourcompany.hostedftp.com”
>
> FTPUSER=”hftp.itaccou...@ourcompany.com”
>
> FTPPASS=”ourcompanyftppassword”
>
> FTPURL="$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTPHOST"
>
> FTPREMOTEFOLDER=”/Servers/ProdN01IL01” (or ProdN02IL01, depending on the
> server)
>
> FTPLOCALFOLDER=”/opt/ftpusers/home”
>
> FTPLOGFILE=”/var/log/lftpsyncs.log”
>
>
>
> # The “Synch” subdirectories -- where customers put files for processing
> by our application
>
> # Move all files from HostedFTP “Synch” directories to local server Synch
> directories, the delete the source files from HostedFTP Synch directories
>
> # “Regular” mirror, with --Remove-source-files
>
>
>
> EXCLUDES="--exclude-glob .*"
>
> EXCLUDES+=" --exclude-glob .*/"
>
>
>
> INCLUDES="--include-glob /opt/ftpusers/home/*/Synch/*"
>
>
>
> lftp -e "
>
> open '$FTPURL ';
>
> lcd $FTPLOCALFOLDER;
>
> cd $FTPREMOTEFOLDER;
>
> mirror --continue --only-newer --Remove-source-files --verbose=3
> --no-perms --parallel=8 $EXCLUDES $INCLUDES”
>
>
>
> # The “Saved” subdirectories – where our application puts files from
> Inbox, after successfully processing them
>
> # The “Error” subdirectories -- where our application puts files from
> Inbox, if it fails to process them
>
> # The “Reports” subdirectories – where our application puts reports
>
> # Sync-*up* files from local “Saved”, “Error” and “Reports” directories
> to HostedFTP, deleting HostedFTP files if they are not present on the local
> server
>
> # “Reverse” mirror, with deleting of remote files not present locally
>
>
>
> EXCLUDES="--exclude-glob .*"
>
> EXCLUDES+=" --exclude-glob .*/"
>
>
>
> INCLUDES="--include-glob /opt/ftpusers/home/*/Saved/*"
>
> INCLUDES+=" --include-glob /opt/ftpusers/home/*/Error/*"
>
> INCLUDES+=" --include-glob /opt/ftpusers/home/*/Reports/*"
>
>
>
> lftp -e "
>
> open '$FTPURL ';
>
> lcd $FTPLOCALFOLDER;
>
> cd $FTPREMOTEFOLDER;
>
> mirror --reverse --continue --only-newer --delete --verbose=3 --no-perms
> --parallel=8 $EXCLU
