Re: SSH/SFTP question
On Sat, Apr 14, 2007 at 05:32:38PM -0400, Frank Bax wrote: Based on what your vendor says; it looks like the file originally contains only LF and not CRLF; so enabling ASCII transfer should convert LF to CRLF. If your transfer software doesn't have this option find another that does. or just convert the files yourself after you get them -- jared
Re: SSH/SFTP question
On Fri, Apr 13, 2007 at 09:37:14AM -0400, stuart van Zee wrote: I was under the impression that when using SFTP to transfer files they were automatically treated as Binary files. i might totally be wrong, but i had the impression that sftp doesn't incorporate the 'legacy ftp' concept of 'binary' vs. 'ascii', but rather just transferred files without any regard to what they have in them. CRLF to terminate lines, the downloaded file would have CRLF terminating it's lines. So I have a vendor that has replaced his FTP with SSH/SFTP. my code is written to expect CRLF because that is the way the files were when using the old FTP system to download. have you tried scp instead of sftp? (if that is an option). sftp seems to be a bit of the bastard child of the openssh software family. Now, when I use SFTP the files just have the LF. that seems strange. i haven't used sftp nearly as much as scp, but i don't recall either ever having modified a file on me. i tried the following test: - made a test file (5 lines) using windows notepad, saved it to the openbsd box via a samba share. - opened up in vi and saw that it had '^M's at the end of it, quit out. - scp'd that file to another host on the network here - went to the other host, verified MD5s same - back on original host, sftp'd the file off of the remote host down to a new filename on the local host, verified MD5s the same. so in my case, sftp'ing a file from a remote host down locally didn't incur any CRLF-LF translation. maybe you are seeing a behaviour from something else and it just seems to be because of sftp(1) ? do you know for certain that the files, as they exist on the vendor's end, do indeed still have the CRLFs in them? What I really need is an explanation or a pointer to where I can get an explanation so that I really know what I am talking about when I talk to this vendor (and KNOW that I know what I am talking about). might not need to go that far. if you know/trust that they have CRLFs on their end, but when you get the files from them, they have only LFs now, just test stuff on your end with your own remote sftp/ssh server like i did above. if you have a file that you know has CRLFs, and on this one problem host you can pull the file down over sftp from your remote test host, and it still has the CRLFs, that would seem to imply that your local system is not responsible for the translation. -- jared
Re: SSH/SFTP question
At 09:37 AM 4/13/07, stuart van Zee wrote: Sorry if this belongs elsewhere but I was sure someone here would know. I was under the impression that when using SFTP to transfer files they were automatically treated as Binary files. So if the remote file uses CRLF to terminate lines, the downloaded file would have CRLF terminating it's lines. So I have a vendor that has replaced his FTP with SSH/SFTP. my code is written to expect CRLF because that is the way the files were when using the old FTP system to download. Now, when I use SFTP the files just have the LF. The vendors answer is that we need to use ASCII mode to transfer the files to get the CRLF. I didn't know that there WAS an ASCII mode in SFTP let alone that using ASCII as opposed to Binary would change the line terminators. The files in question are technically ASCII text files but shouldn't I be getting an EXACT copy of the file when I use Binary mode (assuming that I am right and that is indeed the default with SFTP)? What I really need is an explanation or a pointer to where I can get an explanation so that I really know what I am talking about when I talk to this vendor (and KNOW that I know what I am talking about). FTP and SFTP clients will often have an option to send files as ASCII/BINARY/AUTO. This option controls how CRLF is handled when encountered in the source file. This option has nothing at all to do with the FTP/SFTP protocol itself; but is an option often included in client software. When transfer option is ASCII (or sometimes TEXT); a translation occurs. When copying from Windows to *BSD, CRLF changes to LF. When copying from *BSD to Windows, LF changes to CRLF. When transfer option is BINARY; no translation ever occurs. When transfer is AUTO; behaviour is either ASCII or BINARY depending on file extension. TXT, HTM, PHP, CGI, PL might all be considered text files; client software often allows this list to be configured. PSCP and WinSCP are two examples of windows software that support SFTP. PSCP does not convert CRLF to LF; WinSCP has a user option controlling this translation. Based on what your vendor says; it looks like the file originally contains only LF and not CRLF; so enabling ASCII transfer should convert LF to CRLF. If your transfer software doesn't have this option find another that does.
SSH/SFTP question
Sorry if this belongs elsewhere but I was sure someone here would know. I was under the impression that when using SFTP to transfer files they were automatically treated as Binary files. So if the remote file uses CRLF to terminate lines, the downloaded file would have CRLF terminating it's lines. So I have a vendor that has replaced his FTP with SSH/SFTP. my code is written to expect CRLF because that is the way the files were when using the old FTP system to download. Now, when I use SFTP the files just have the LF. The vendors answer is that we need to use ASCII mode to transfer the files to get the CRLF. I didn't know that there WAS an ASCII mode in SFTP let alone that using ASCII as opposed to Binary would change the line terminators. The files in question are technically ASCII text files but shouldn't I be getting an EXACT copy of the file when I use Binary mode (assuming that I am right and that is indeed the default with SFTP)? What I really need is an explanation or a pointer to where I can get an explanation so that I really know what I am talking about when I talk to this vendor (and KNOW that I know what I am talking about). Stuart van Zee [EMAIL PROTECTED]
Re: SSH/SFTP question
On Fri, Apr 13, 2007 at 09:37:14AM -0400, stuart van Zee wrote: Sorry if this belongs elsewhere but I was sure someone here would know. I was under the impression that when using SFTP to transfer files they were automatically treated as Binary files. So if the remote file uses CRLF to terminate lines, the downloaded file would have CRLF terminating it's lines. So I have a vendor that has replaced his FTP with SSH/SFTP. my code is written to expect CRLF because that is the way the files were when using the old FTP system to download. Now, when I use SFTP the files just have the LF. The vendors answer is that we need to use ASCII mode to transfer the files to get the CRLF. I didn't know that there WAS an ASCII mode in SFTP let alone that using ASCII as opposed to Binary would change the line terminators. The files in question are technically ASCII text files but shouldn't I be getting an EXACT copy of the file when I use Binary mode (assuming that I am right and that is indeed the default with SFTP)? What I really need is an explanation or a pointer to where I can get an explanation so that I really know what I am talking about when I talk to this vendor (and KNOW that I know what I am talking about). The sftp(1) man page is the canonical source, which doesn't contain ASCII mode, but given that they aren't exactly clued, why not just put a filter in there? I'm sure there are at least a thousand utilities or easily-written scripts that can do this. (On another note, ftps - ftp over SSL - does have an ASCII mode. It's also incompatible with pretty much anything and best replaced by something vaguely modern - say, sftp.) Joachim -- TFMotD: netintro (4) - introduction to networking facilities
Re: SSH/SFTP question
On 4/13/07, Joachim Schipper [EMAIL PROTECTED] wrote: (On another note, ftps - ftp over SSL - does have an ASCII mode. It's also incompatible with pretty much anything and best replaced by something vaguely modern - say, sftp.) What's the problem with ftp/ssl? Beyond the fact that it uses the FTP protocol, that is (modern firewalls should be able to handle ftp by now, so that shouldn't be an issue).
Re: SSH/SFTP question
On Fri, Apr 13, 2007 at 04:34:40PM -0400, bofh wrote: On 4/13/07, Joachim Schipper [EMAIL PROTECTED] wrote: (On another note, ftps - ftp over SSL - does have an ASCII mode. It's also incompatible with pretty much anything and best replaced by something vaguely modern - say, sftp.) What's the problem with ftp/ssl? Beyond the fact that it uses the FTP protocol, that is (modern firewalls should be able to handle ftp by now, so that shouldn't be an issue). Modern firewalls do handle FTP pretty well, *if* they can take a look at the control stream. However, encrypting the control stream is critical, as most of the point of ftps is to make sure you don't send passwords in cleartext. And that's even avoiding compatibility issues; it's not easy to find both an FTP server and an FTP client that will encrypt both command and data stream (quite a few cannot encrypt the data stream; I saw that as essential for the application I used ftps for, a couple of years ago). Joachim -- TFMotD: maxtmp (4) - Maxim MAX6642/MAX6690 temperature sensor
Re: SSH/SFTP question
On Fri, 13 Apr 2007 09:37:14 -0400, stuart van Zee wrote: Sorry if this belongs elsewhere but I was sure someone here would know. I was under the impression that when using SFTP to transfer files they were automatically treated as Binary files. So if the remote file uses CRLF to terminate lines, the downloaded file would have CRLF terminating it's lines. So I have a vendor that has replaced his FTP with SSH/SFTP. my code is written to expect CRLF because that is the way the files were when using the old FTP system to download. Now, when I use SFTP the files just have the LF. The vendors answer is that we need to use ASCII mode to transfer the files to get the CRLF. I didn't know that there WAS an ASCII mode in SFTP let alone that using ASCII as opposed to Binary would change the line terminators. The files in question are technically ASCII text files but shouldn't I be getting an EXACT copy of the file when I use Binary mode (assuming that I am right and that is indeed the default with SFTP)? What I really need is an explanation or a pointer to where I can get an explanation so that I really know what I am talking about when I talk to this vendor (and KNOW that I know what I am talking about). Stuart van Zee I cannot duplicate your findings. maybe we need to know a bit more about what is running at each end. I did: Make short file with CRLF at end of each line except last. That used a windows text editor (UE). Used winscp to send it to an OpenBSD box using sftp (you can choose that or scp). Note: Winscp does offer Text mode, Binary mode and Automatic. I chose binary. Note that the conversion happens in winscp if you let it do Text or Auto. Then I used sftp on another OpenBSD box to get the file from the other one. All of the CRLF pairs were intact. Conclusion: My theory is that the conversion happens at the other end. Insufficient data to speculate further but the CR stripping does NOT happen in OpenBSD's sftp. HTH, HAND. More testing if you'd like to spec it. Rod/ A consultant is someone who's called in when someone has painted himself into a corner. He's expected to levitate his client out of that corner. -The Sayings of Chairman Morrow. 1984.
