The correct set of -src-pipe and -dst-pipe commands will work when run
locally like this:
src-pipe | dst-pipe
The job of the src-pipe command is to produce output to stdout, and the
job of the dst-pipe command is to consume input from stdin.
Your comment that the src-pipe command is producing a compressed file
sounds as though it is not writing to stdout.
Also, the path part of the urls don't matter in this case, so the only
reason you would set those is for server logging purposes.
Mike
On 6/22/2012 11:25 AM, Metin Depaolis wrote:
Hello everyone,
I'm not sure if this is the correct mailing list for my situation, so
please correct me if it isn't.
I have been tweaking the globus-dir-copy command to run a different
command other than tar. Specifically, I am using nanozip with the
command `nz` to compress the data prior to transfer, then decompress
it after transfer. Below I listed the commands I ran to produce the
error and the error itself.
===============================================================================
starting SERVER (nz is program that compresses and uncompresses data)
./globus-gridftp-server -fs-whitelist popen,file,ordering
-popen-whitelist tar:/bin/tar,nz:/usr/bin/nz -p 52111 -aa
===============================================================================
running CLIENT
./test ftp://127.0.0.1:52111/home/metin/small ftp://127.0.0.1:52111/tmp/test1
# nz <command> [-<opt_1>...-<opt_n>] <archive_file> <files...>
# Commands|| a: add to archive | x: extract from archive
# Options || -m: memory use | -co: a compression option | -o is output path
# the command that gets run in third party transfers (SERVER_NANO is /bin/nz)
./globus-url-copy -dbg -cd ${3} ${4} ${5} -src-pipe "${SERVER_NANO} a -m.5g -co
${SRC_PATH}/C0mpress ${SRC_PATH}/${SRC_TARGET}" -dst-pipe "${SERVER_NANO} x
-o${DEST_PATH} ${SRC_PATH}/C0mpress.nz" ${SRC_URL}/TARSTREAM ${DEST_URL}/TARSTREAM
./globus-url-copy -dbg -cd \
-src-pipe 'nz a -m.5g -co /home/metin/C0mpress /home/metin/small' \
-dst-pipe 'nz x -o/tmp/test1 /home/metin/C0mpress.nz' \
ftp://127.0.0.1:52111/home/metin/small/TARSTREAM
ftp://127.0.0.1:52111/tmp/test1/TARSTREAM
===============================================================================
OUTPUT from `set -x`
./test ftp://127.0.0.1:52111/home/metin/small ftp://127.0.0.1:52111/tmp/test1
+ LOCAL_NANO=nz
+ SERVER_NANO=nz
+ '[' Xftp://127.0.0.1:52111/home/metin/small = X -o
Xftp://127.0.0.1:52111/tmp/test1 = X ']'
+ SRC_URL=ftp://127.0.0.1:52111/home/metin/small
+ SRC_PATH=home/metin/small
+ '[' home/metin/small '!=' ftp://127.0.0.1:52111/home/metin/small ']'
+ SRC_PATH=/home/metin/small
+ SRC_TARGET=small
+ SRC_PATH=/home/metin
+ '[' /home/metinX = X ']'
+ DEST_URL=ftp://127.0.0.1:52111/tmp/test1
+ DEST_PATH=tmp/test1
+ '[' tmp/test1 '!=' ftp://127.0.0.1:52111/tmp/test1 ']'
+ DEST_PATH=/tmp/test1
+ '[' /tmp/test1X = X ']'
+ case "$SRC_URL" in
+ case "$DEST_URL" in
+ MODE=tp
+ case "$MODE" in
+ ./globus-url-copy -dbg -cd -src-pipe 'nz a -m.5g -co
/home/metin/C0mpress /home/metin/small' -dst-pipe 'nz x -o/tmp/test1
/home/metin/C0mpress.nz'
ftp://127.0.0.1:52111/home/metin/small/NANOSTREAM
ftp://127.0.0.1:52111/tmp/test1/NANOSTREAM
ERROR
error: globus_ftp_client: the server responded with an error
500 500-Command failed. : callback failed.
500-globus_xio: System error in write: Broken pipe
500-globus_xio: A system call failed: Broken pipe
500 End.
===============================================================================
The src-pipe command gets run because it correctly compresses a file
in the correct directory, which makes me think that it might be the
dst-pipe command causing the error.
Thank you,
Metin
