Bug#882381: [ftpsync] Error using socks and old sed
Hi Leo On Wed, Nov 22, 2017 at 10:15:10PM +0100, Leo wrote: > Hi Bastian > Thank you for your prompt reply. I would like to talk about it below. > > The -E option is common to all current operation systems as it is > > specified by POSIX. As this script not only runs on Linux, using GNU > > specific options is not the way to go. I also found no supported Linux > > distribution without support for it. > I tell SuSE support. Their sed under SLES11 or SLES12 knows no option -E. Somehow I doubt that. SLES12 comes with sed 4.2.2, which is years after the option was introduced. SLES11 may be a different beast. > >I would be more concerned about a stage2 that runs for over an hour. > The problem is that e.g. my existing proxy has a timeout of 120 and rsync > 3600 seconds. The program rsync calculates its keep alive transmissions > according to the following calculation: timeout / 2. This results in a > timeout of 3600 seconds keep alive packets every 1800 seconds. However, the > proxy ends the connection after 120 seconds of inactivity. So while ftpsync > checks in Stage2 what files it can delete (no traffic),the proxy's timeout > fails and terminates the socks connection. If the rsync wants to access the > remote station again, it first notices that it is offline and ends with an > error. But if I synchronize the timeouts of the proxy and rsync, the keep > alive packets will be sent on time and the connection will remain > open.Another note, the proxy is not under my administration and I can not > change that. I'm reluctant to add options just so there are options. If a change in this option does not have side-effects, we can change it for everyone. But this I have to asses further. Bastian -- All your people must learn before you can reach for the stars. -- Kirk, "The Gamesters of Triskelion", stardate 3259.2
Bug#882381: [ftpsync] Error using socks and old sed
Hi Bastian Thank you for your prompt reply. I would like to talk about it below. The -E option is common to all current operation systems as it is specified by POSIX. As this script not only runs on Linux, using GNU specific options is not the way to go. I also found no supported Linux distribution without support for it. I tell SuSE support. Their sed under SLES11 or SLES12 knows no option -E. ;-) >I would be more concerned about a stage2 that runs for over an hour. The problem is that e.g. my existing proxy has a timeout of 120 and rsync 3600 seconds. The program rsync calculates its keep alive transmissions according to the following calculation: timeout / 2. This results in a timeout of 3600 seconds keep alive packets every 1800 seconds. However, the proxy ends the connection after 120 seconds of inactivity. So while ftpsync checks in Stage2 what files it can delete (no traffic),the proxy's timeout fails and terminates the socks connection. If the rsync wants to access the remote station again, it first notices that it is offline and ends with an error. But if I synchronize the timeouts of the proxy and rsync, the keep alive packets will be sent on time and the connection will remain open.Another note, the proxy is not under my administration and I can not change that. Original Text: Das Problem ist, dass z.B. mein vorhandener Proxy einen Timeout von 120 und rsync 3600 Sekunden hat. Das Programm rsync berechnet seine keep alive Sendungen nach folgender Rechnung: timeout/2. Das ergibt bei einem Timeout von 3600 Sekunden keep alive Pakete alle 1800 Sekunden. Der Proxy beendet aber schon nach 120 Sekunden inaktivität die Verbindung. Während also ftpsync in Stage2 überprüft, welche Dateien es löschen kann (kein Datenverkehr), schlägt der timeout des Proxy zu und beendet die Socks Verbindung. Wenn das rsync dann wieder auf die gegenstelle zugreifen möchte merkt es erst, dass es offline ist und beendet mit einem Fehler. Wenn ich aber die timeouts vom Proxy und rsync synchronisiere werden die keep alive Pakete rechtzeitig gesendet und die Verbindung bleibt offen. Noch ein Hinweis, der Proxy steht nicht unter meiner Administration und ich kanns deswegen nicht ändern. Regards, Björn -- mfg Björn ICQ: 22207642, MSN: de...@gmx.de
Bug#882381: [ftpsync] Error using socks and old sed
Hi On Wed, Nov 22, 2017 at 12:07:33AM +0100, Leo wrote: > The program call in line 590 contains too many "which lead to a script error > if the variable RSYNC contains a space. For example, RSYNC = > "/usr/bin/socksify /usr/bin/rsync" Strictly speaking, filenames can contain spaces. But I see the point and may change that back. > Compatibility problem: > Not every Linux OS supports the -E parameter with sed. It is better to use > the GNU compatible option -r, which gives the same result. The -E option is common to all current operation systems as it is specified by POSIX. As this script not only runs on Linux, using GNU specific options is not the way to go. I also found no supported Linux distribution without support for it. > Improvement: > Running the program through a (Socks) proxy whose timeout is less than 3600 > seconds may cause Stage2 to crash. The timeout should be customizable with > its own variable in the configuration. I would be more concerned about a stage2 that runs for over an hour. Regards, Bastian -- I'm frequently appalled by the low regard you Earthmen have for life. -- Spock, "The Galileo Seven", stardate 2822.3
Bug#882381: [ftpsync] Error using socks and old sed
Package: ftpsync
Version: 20171018
Severity: important
--- Please enter the report below this line. ---
There is an error, a compatibility issue, and a fix for me for the
"ftpsync" script attached.
Error:
The program call in line 590 contains too many "which lead to a script
error if the variable RSYNC contains a space. For example, RSYNC =
"/usr/bin/socksify /usr/bin/rsync"
Compatibility problem:
Not every Linux OS supports the -E parameter with sed. It is better to
use the GNU compatible option -r, which gives the same result.
Improvement:
Running the program through a (Socks) proxy whose timeout is less than
3600 seconds may cause Stage2 to crash. The timeout should be
customizable with its own variable in the configuration.
Original Text:
Es sind hier ein Fehler, ein Kompatibilitätsproblem und eine
Verbesserung von mir für das Skript "ftpsync" im Anhang enthalten.
Fehler:
Der Programmaufruf in Zeile 590 enthält zuviele " die zu einem
Skriptfehler führen, wenn die Variable RSYNC ein Leerzeichen enthält.
Zum Beispiel RSYNC="/usr/bin/socksify /usr/bin/rsync"
Kompatibilitätsproblem:
Nicht jedes Linux OS unterstützt bei sed den Parameter -E. Besser ist es
hier die GNU kompatible Option -r zu verwenden, die das selbe ergibt.
Verbesserung:
Wird das Programm über einen (Socks) Proxy betrieben, dessen Timeout
kleiner 3600 Sekunden ist, kann es zu einem abbruch bei der Stage2
führen. Der Timeout sollte mit einer eigenen Variable in der
Konfiguration anpassbar sein.
--- System information. ---
Architecture:
Kernel: Linux 4.13.0-1-amd64
Debian Release: buster/sid
--- Package information. ---
Package's Depends field is empty.
Package's Recommends field is empty.
Package's Suggests field is empty.
ftpsync-bw.gz
Description: application/gzip
36c36
< VERSION="20171018"
---
> VERSION="20171018-Bw"
590c590
< "${_RSYNC[@]}" ${RSYNC_OPTIONS} "$@" \
---
> ${_RSYNC[@]} ${RSYNC_OPTIONS} $@ \
664c664
< sed -Ene 's#.*/binary-([^/]+)$#\1#p; s#.*/(source)$#\1#p' | sort -u | tr '\n' ' ')
---
> sed -rne 's#.*/binary-([^/]+)$#\1#p; s#.*/(source)$#\1#p' | sort -u | tr '\n' ' ')
680c680
< for bytes in $(sed -Ene 's/^sent ([0-9]+) bytes received ([0-9]+) bytes.*/\1/p' "${LOGDIR}/rsync-${NAME}.log"); do
---
> for bytes in $(sed -rne 's/^sent ([0-9]+) bytes received ([0-9]+) bytes.*/\1/p' "${LOGDIR}/rsync-${NAME}.log"); do
870a871,874
>
> # Connection Timeout (Keepalive Pakets Timeout/2)
> RSYNC_TIMEOUT=${RSYNC_TIMEOUT:-3600}
>
872c876
< RSYNC_OPTIONS=${RSYNC_OPTIONS:-"-prltvHSB8192 --safe-links --chmod=D755,F644 --timeout 3600 --stats --no-human-readable"}
---
> RSYNC_OPTIONS=${RSYNC_OPTIONS:-"-prltvHSB8192 --safe-links --chmod=D755,F644 --timeout ${RSYNC_TIMEOUT} --stats --no-human-readable"}
874c878
< RSYNC_OPTIONS=${RSYNC_OPTIONS:-"-prltvHSB8192 --safe-links --timeout 3600 --stats --no-human-readable"}
---
> RSYNC_OPTIONS=${RSYNC_OPTIONS:-"-prltvHSB8192 --safe-links --timeout ${RSYNC_TIMEOUT} --stats --no-human-readable"}
