On Apr 13, 2011, at 08:57, Hughes, Jim wrote: > The RFC for ftp says text data is placed on the wire in ascii with a > carriage return/line feed at the end of the line. However not all > systems put CR/LF at the end of the records. > > Without getting into a religious battle, here is a pipeline we use. We > get files from our ftp server in binary mode and then process it with a > pipeline similar to this one. > > "PIPE < consumer binary a |", > " deblock linend 0a |", > " strip trailing 0d 1 |", > " xlate from 819 to 1047 |", > " > consumer text a" > Hmmm. Classic Macintosh use[sd] as a line separator, but I don't know that these were ever placed on the wire.
But I believe HTTP allows any of CR/LF, LF, or CR as a line separator. A challenge to deblock these accepting the constraints that: o The CR and LF may be split between packets. o The entire file transmitted may be too big to fit in memory and/or it's desirable to stream it in real time. -- gil
