Hello again,
Just thought I'd post what worked for me in uploading files from our
Sun box to an offsite mainframe. What I am posting is the korn shell
wrapper script I use to do some pre-processing and call lftp via -f or
-e and I am posting the lftp command file that does the actual work.
The command file was a great help - if I could pass variables into it
I probably wouldn't have needed the wrapper script, but that is ok.
Notes specific to my script/setup. I am only posting one version of
the script - the one that does everything automatically. I also have
a commandfile that sets all the necessary settings then calls "open"
but without the password so the user can then enter it manually and
remain at the lftp> command prompt. The script below calls the
command file with the -f cmdfile option which means it will run all
your commands and then exit. But if you want the script to do all
the "set..." and other prep stuff for you and then leave you at the
prompt to do your own thing, call it with the -e option and use
"source" and then it won't exit:
lftp -e "source ${dir}/manualcmdfile.lftp" 2>&1 | tee -a ${log}
Our setup will probably be different from yours. I saw a lot of lftp
options/solutions that were posted and worked for the requesters that
didn't for me because our setups are all different. For example, the
mainframe allowed PASV connections for FTP but not for FTP over SSL,
so we had to set it to Active. Also we run private/non-routing IP's
internally so we had to use the ftp:port-ipv4 option to send the
server the correct NAT IP. In fact, most of the problems I had were
not lftp specific but due to the return path of the packets due to the
NAT and routing through the firewall. I include extra echo's for the
logfile because I dialed down the debug level to just warnings and
errors (level 3) so the password wouldn't show and so the output
wouldn't be so verbose that the endusers wouldn't check the email they
get to make sure the correct files were uploaded).
I am including my wrapper script even though I am not much of a
scripter just because I include some functionality for calling the
script both interactively (from the command line/shell) and
non-interactively (via cron) as well as some code to on-the-fly change
the password every month (on whatever arbitrary date you set). Not
really lftp specific but it was handy to be able to do that. I also
wanted to see the output if I ran the script from the command line and
have a log file whether run from the command line or cron. In either
case, I also have the results of that session emailed to me (and the
actual users of the mainframe) so that instead of running the script
themselves as they used to, the cron job does it for them and all they
have to do is view the output to make sure the local files got
uploaded remotely (by comparing the local !ls and the remote ls in the
results email). I believe "mailx" is Sun specific - use "mail" on
'nux systems...
Some of the "set" commands don't accept inline comments - for sure
ftps:initial-prot doesn't so even though I added some comments for
this post, you should remove them if they cause a problem. When you
do first run your scripts, you will probably set debug to 9 or 12 and
then you will see if the option ends up appearing as a
printout/warning because of your inline comments - most were fine, but
initial-prot wasn't so heads up. But I do like the fact many of the
options accept inline comments, it helps me remember why I used an
option, especially when trying dozens of different things and mailing
list suggestions - it's easy to lose track of which you tried alone,
together, etc - so the comments help...
So anyhow, I hope that the scripts prove useful to someone in the
future as your many posts proved useful to me in getting my own
project working.
thanks again to Alexander for a great tool and to those folks who
posted helpful stuff in the past (thank heavens for those archives)
and/or emailed me directly.
Here is the lftp command file mainframe_autoup.lftp:
::::::::::::::
# lftp script to automatically upload the Fiscal files...
set cache:enable no
set cmd:prompt "[EMAIL PROTECTED]:> " # Simplify the prompt
set ftp:passive-mode off # This mainframe requires
Active mode i.e. "no"
set ftp:port-ipv4 1.2.3.4 # mainframe needs real/NAT
IP, so force it
set ftp:port-range 4444-4444 # connections only allowed on this 1 port
set ftp:use-size no # Size not supp by mainframe for SSL
set ftp:ssl-protect-list yes # list and data must be protected
set ftp:ssl-protect-data yes # ditto
set ftps:initial-prot "P"
set net:connection-limit 1 # just use 1 conn as mainframe may be dumb
set ftp:timezone "" # empty "" menas take local timezone
# keep a record of the session, perms are 700 so safe to leave the file
echo "debug 3"
debug 3
# open a connection and feed it loginID and password, port and hostname
echo "open -u ftpID,password -p 4444 mainframe.soandso.com"
open -u ftpID,password -p 4444 mainframe.soandso.com
# mainframe specific settings, passed verbatim via "quote" command
# I've garbled a few since they may be specific to us, get them from your
#mainframe folks - often a job will seem to "work" but be garbage text
because you
# didn't set these variables first....
echo "quote site recfm=aa lrecl=1234 blksize=12345 retpd=5 cyl pri=111"
quote site recfm=aa lrecl=1234 blksize=12345 retpd=5 cyl pri=111"
# backslash require in front of single-quotes
echo "cd PD.AAFTPDIR.FTP"
cd \'PD.AAFTPDIR.FTP\'
# list out which files are currently available
echo " ls /home/me/*US.SOANDSO*"
!ls /home/me/*US.SOANDSO*
# upload all files that match SOANDSO pattern (-a = ascii)
echo "mput -a /home/me/*US.SOANDSO*"
mput -a /home/me/*US.SOANDSO*
# check the US.SOANDSO files made it....
# mainframe didnt support leading star, so multiple single % for each character
echo "ls %%%%%%%%.US.SOANDSO*"
ls %%%%%%%%.US.SOANDSO*
echo ""
echo "________________________________________"
echo ""
# etcetera (I have to do same steps as above for multiple directories)
# [ snipped for brevity ]
# Move the files to a backup directory
echo " mv /home/me/*US.SOANDSO* /home/me/BAK/."
!mv /home/me/*US.SOANDSO* /home/me/BAK/.
echo " mv /home/me/*MORE.SOANDSO* /home/me/BAK/."
!mv /home/me/*MORE.SOANDSO.* /home/me/BAK/.
# [ etcetera, snipped for brevity ]
# just to have a copy of the settings in log, for troubleshooting
set -a
# exit
echo "exit"
exit
Here is the wrapper script mainframe_autoup.ksh (apologies for any
poor coding, multiple use of date and touch commands, but it is
working fine these past 2 weeks and speed/efficiency is not the main
issue, clarity for whoever supports it is). I am using the 93
version of ksh, not Sun's default 88 version. If you are using a
linux system, chances are you are already using the newer one or can
easily convert the below to bash...
#!/usr/local/bin/ksh
PATH=$PATH:/usr/local/bin;export PATH
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/ssl/lib;export LD_LIBRARY_PATH
umask 077
# since the log, etc. take the user name, when I am testing I
# use a cmd line option, otherwise the script runs using the main/real user
# name for all lognames...
if [ $1 ]
then
logname=$1
else
logname="defaultuser"
fi
rcpt="[EMAIL PROTECTED] [EMAIL PROTECTED]"
cc="[EMAIL PROTECTED]"
sndr="[EMAIL PROTECTED] "
log="/tmp/mainframe-$logname-lftp.log"
dir="/home/$logname/mainframe"
# should use date '+%b %y %e' | read month year today
# below instead of 3 calls to date, plus 4th call to date for mydate.... Sorry!
month=`date +%b` # 3 letter alpha month Jan, Feb, Mar, etc.
year=`date +%y` # 2 digit year like 07 08 etc.
typeset -i today=`date +%e` # numeric date, no leading 0
typeset -i mytoday=1 # set this to the 1st of the month
mydate=$(date +%y%m%d-%H:%M:%S)
oldpass=$(egrep open ${dir}/mainframe_autoup.lftp | egrep -v
'usage|echo' | sed -e 's/^.*,//' -e 's/ -p.*$//')
newpass=$(egrep -i ${month} ${dir}/mainframe_passlist.${year})
# just in case there is no log file this "touch" keeps mv from complaining
touch ${log}
mv ${log} ${log}.${mydate}
touch ${log}
# if Interactive from cron I want to see it both on my screen and keep
a log copy
# if NonInteractive (i.e. run from cron), no need to tee off instead
redirect all to ${log}
function runlftpjob
{
# interactive shell = use tee to log but also keep output on screen
if [ $(tty > /dev/null; echo $?) != 1 ]
then # = Im an interactive shell
echo "Interactive lftp -f ${cmdfile} 2>&1 | tee -a ${log}"
lftp -f ${cmdfile} 2>&1 | tee -a ${log}
else
echo "Not-interactive lftp -f ${cmdfile} >> ${log} 2>&1" > ${log}
lftp -f ${cmdfile} >> ${log} 2>&1
fi
}
# Every X days we change the password (via a separate cronjob)
# mainframe has no prompting, you just feed it the current/old
password followed by the new password (twice)
# so this function sets which lftp cmd file to use: newpass if
"today:" or just the regular lftp file
function isittoday
{
if (( $today == $mytoday )); then
cp ${dir}/mainframe_autoup.lftp
${dir}/.attic/mainframe_autoup.lftp.${mydate}
# using previous lftp command file w/ passwod in it, create a new lftp
cmd file w/ old and new pass
sed "s/${oldpass}/${oldpass}\/${newpass}\/${newpass}/g"
${dir}/mainframe_autoup.lftp > ${dir}/mainframe_autoup.newpass
sed "s/${oldpass}/${newpass}/g" ${dir}/mainframe_autoup.lftp
${dir}/mainframe_autoup.new
mv -f ${dir}/mainframe_autoup.new ${dir}/mainframe_autoup.lftp
cmdfile=${dir}/mainframe_autoup.newpass
# if not today, just do a normal lftp upload, using the
else
cmdfile=${dir}/mainframe_autoup.lftp
fi
}
function domail
{
if egrep -i -- "failed|no.*found|no.*such" ${log} > /dev/null 2>&1
then
mailx -r ${sndr} -c "${cc}" -s "Mainframe auto upload: Possible
ERROR - `hostname` `date +%y%m%d-%H:%M:%S`...." ${rcpt} < ${log}
else
mailx -r ${sndr} -c "${cc}" -s "Mainframe auto upload: Log results
for - `hostname` `date +%y%m%d-%H:%M:%S`...." ${rcpt} < ${log}
fi
}
# this is like main() in that it calls each of the functions defined above
isittoday
runlftpjob
domail
exit
