Re: REST - error for files bigger than 4GB
Steven M. Schweda ha scritto: Are you certain that the FTP _server_ can handle file offsets greater than 4GB in the REST command? i agree with steven here. it's very likely to be a server-side problem. -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Fwd: Re: REST - error for files bigger than 4GB
Seems this wasn't sent to the list ;-) - Weitergeleitete Nachricht von Petr Kras <[EMAIL PROTECTED]> - Datum: Wed, 6 Sep 2006 14:57:32 +0200 Von: Petr Kras <[EMAIL PROTECTED]> Antwort an: Petr Kras <[EMAIL PROTECTED]> Betreff: Re: REST - error for files bigger than 4GB An: Jochen Roderburg <[EMAIL PROTECTED]> Thanks for answer, it helped me to find out the reason. In Wget it looks that this server supports REST command, see samples below. For files under 4G it works fine, over 4G it does'nt. The server itself doesn't accept value over 4GB (2^32). -- ftp> REST ?Invalid command. ftp> quote REST 4294967295 350 Restarting at 4294967295. ftp> quote REST 4294967296 501 Reply marker is invalid. ftp> -- I'm sorry but this server is located in our intranet. Regards, Petr Kras Here is listing of Wget behaviour under and behind 4GB limit. I think it is correct. CORRECT - just under 4G ==> PORT ... done.==> REST 4099397567 ... done. ==> RETR TVDC_SS_01103.ts ... done. Length: 5,632,104,188 (5.2G), 1,532,706,621 (1.4G) remaining [ skipping 400K ] 400K ,, ,, .. .. .. 72% 427.40 KB/s 401K .. .. .. .. .. 73% 411.05 KB/s another file CORRECT - restoration under 4G ==> PORT ... done.==> REST 2818997399 ... done. ==> RETR TVDC_SS_01100.ts ... done. Length: 5,632,104,188 (5.2G), 2,813,106,789 (2.6G) remaining [ skipping 275K ] 275K ,, .. .. .. .. 50% 407.44 KB/s 276K .. .. .. .. .. 50% 318.09 KB/s INCORRECT - restoration behind 4G ==> PORT ... done.==> REST 4998699942 ... REST failed, starting from scratch. ==> RETR TVDC_SS_01100.ts ... done. Length: 5,632,104,188 (5.2G), 633,404,246 (604M) remaining 0K .. .. .. .. .. 0% 378.26 KB/s 1K .. .. .. .. .. 0% 312.72 KB/s Jochen Roderburg <[EMAIL PROTECTED]> wrote on 06.09.2006 09:21:35: > Petr Kras schrieb: > > When transfer is broken and restoration is required > > it doesnt work for files greater than 4GB (not checked for 2GB) > > and brake is behind 4GB (2GB) limit. > > > --13:58:54-- ftp://streamlib.pan.eu/Streams/TVDC_SS_01100.ts > >=> `/opt/streams/Stream1/TVDC_SS_01100.ts' > > ==> CWD not required. > > ==> PORT ... done.==> REST 4998699942 ... > > REST failed, starting from scratch. > > > > ==> RETR TVDC_SS_01100.ts ... done. > > Length: 5,632,104,188 (5.2G), 633,404,246 (604M) remaining > > > > Looks more like your FTP server does not support the REST command which > is needed for partial transfers. > Maybe a run with wget debug option (-d) shows more. > > I tried a visit to the mentioned site, but got only "unknown host". > Is streamlib.pan.eu the real hostname? > > Best regards, > > Jochen Roderburg > ZAIK/RRZK > University of Cologne > Robert-Koch-Str. 10 Tel.: +49-221/478-7024 > D-50931 Koeln E-Mail: [EMAIL PROTECTED] > Germany > .. Confidentiality Notice The information contained in this Email, and any attachments, is intended for the named recipients only. It may contain confidential and/or legally privileged information. If you are not the intended recipient, you must not copy, store, distribute or take any action in reliance on it. Any views expressed do not necessarily reflect the views of the company. If you receive this Email by mistake, please advise the sender by using the reply facility in your Email software and then delete it. . - Ende der weitergeleiteten Nachricht -
Re: REST - error for files bigger than 4GB
Jochen Roderburg <[EMAIL PROTECTED]> writes: > Petr Kras schrieb: >> When transfer is broken and restoration is required >> it doesnt work for files greater than 4GB (not checked for 2GB) >> and brake is behind 4GB (2GB) limit. > >> --13:58:54-- ftp://streamlib.pan.eu/Streams/TVDC_SS_01100.ts >>=> `/opt/streams/Stream1/TVDC_SS_01100.ts' >> ==> CWD not required. >> ==> PORT ... done.==> REST 4998699942 ... REST failed, starting >> from scratch. >> ==> RETR TVDC_SS_01100.ts ... done. >> Length: 5,632,104,188 (5.2G), 633,404,246 (604M) remaining > > Looks more like your FTP server does not support the REST command > which is needed for partial transfers. Or, even more likely, that the server isn't fully 64-bit aware, and its REST chokes on sizes >=2^31. In that case, Wget does what it can: downloads the whole file, ignoring the section it has already seen, and starting appending from the last size onward. That way the file never gets truncated.
Re: REST - error for files bigger than 4GB
Petr Kras schrieb: When transfer is broken and restoration is required it doesnt work for files greater than 4GB (not checked for 2GB) and brake is behind 4GB (2GB) limit. --13:58:54-- ftp://streamlib.pan.eu/Streams/TVDC_SS_01100.ts => `/opt/streams/Stream1/TVDC_SS_01100.ts' ==> CWD not required. ==> PORT ... done.==> REST 4998699942 ... REST failed, starting from scratch. ==> RETR TVDC_SS_01100.ts ... done. Length: 5,632,104,188 (5.2G), 633,404,246 (604M) remaining Looks more like your FTP server does not support the REST command which is needed for partial transfers. Maybe a run with wget debug option (-d) shows more. I tried a visit to the mentioned site, but got only "unknown host". Is streamlib.pan.eu the real hostname? Best regards, Jochen Roderburg ZAIK/RRZK University of Cologne Robert-Koch-Str. 10 Tel.: +49-221/478-7024 D-50931 Koeln E-Mail: [EMAIL PROTECTED] Germany
Re: REST - error for files bigger than 4GB
From: Petr Kras [...] > ==> PORT ... done.==> REST 4998699942 ... > REST failed, starting from scratch. [...] Something more might be learned from adding "-d" to your Wget command line. I don't use this continuation feature, but a quick look at the code suggests that Wget 1.10.2 is using 64-bit integers for file sizes/offsets if it's built with large-file capability, so it's not obviously defective. (If it can _say_ "4998699942", then it should be ok.) Are you certain that the FTP _server_ can handle file offsets greater than 4GB in the REST command? Steven M. Schweda [EMAIL PROTECTED] 382 South Warwick Street(+1) 651-699-9818 Saint Paul MN 55105-2547
REST - error for files bigger than 4GB
When transfer is broken and restoration is required it doesnt work for files greater than 4GB (not checked for 2GB) and brake is behind 4GB (2GB) limit. Downloading will start from zero. version: GNU Wget 1.10.2 Petr Kras - 13:58:33 (85.36 KB/s) - `/opt/streams/Stream1/.listing' saved [61759] Removed `/opt/streams/Stream1/.listing'. Remote file no newer than local file `/opt/streams/Stream1/TVDC_SS_01101.mpg' -- not retrieving. --13:58:43-- ftp://streamlib.pan.eu/Streams/TVDC_SS_01100.ts => `/opt/streams/Stream1/.listing' Connecting to streamlib.pan.eu|10.96.19.16|:21... connected. Logging in as DVBStreamuser ... Logged in! ==> SYST ... done.==> PWD ... done. ==> TYPE I ... done. ==> CWD /Streams ... done. ==> PORT ... done.==> LIST ... done. 0K 108.44 KB/s 13:58:44 (108.44 KB/s) - `/opt/streams/Stream1/.listing' saved [61759] Removed `/opt/streams/Stream1/.listing'. The sizes do not match (local 4998699942) -- retrieving. --13:58:54-- ftp://streamlib.pan.eu/Streams/TVDC_SS_01100.ts => `/opt/streams/Stream1/TVDC_SS_01100.ts' ==> CWD not required. ==> PORT ... done.==> REST 4998699942 ... REST failed, starting from scratch. ==> RETR TVDC_SS_01100.ts ... done. Length: 5,632,104,188 (5.2G), 633,404,246 (604M) remaining 0K .. .. .. .. .. 0% 223.72 KB/s 1K .. .. .. .. .. 0% 225.55 KB/s - Panasonic ideas for live Petr Kras Development Engineer Panasonic Europe Software Development Laboratory s.r.o. Teslova 1, 301 00 Plzen, Czech Republic Tel: +420 377 669 123 Fax: +420 377 669 120 E-mail: [EMAIL PROTECTED] .. Confidentiality Notice The information contained in this Email, and any attachments, is intended for the named recipients only. It may contain confidential and/or legally privileged information. If you are not the intended recipient, you must not copy, store, distribute or take any action in reliance on it. Any views expressed do not necessarily reflect the views of the company. If you receive this Email by mistake, please advise the sender by using the reply facility in your Email software and then delete it. .
