If the computing world only ever consisted of "files" each consisting of one (and only one) stream of sequential data, with no other attributes or metadata, then many utilities, including FTP, would be a lot easier to operate. We don't live in such a world.
Yes, non-sequential z/OS datasets are "interesting." Dataset concepts really don't exist on many other systems, so *of course* we should expect problems when we don't instruct the transfer tools correctly (or at all). If I try to send a 30 kilogram package through a postal service that only supports packages weighing a maximum of 10 kilograms, I shouldn't be *too* surprised if my package either doesn't arrive at the other end or arrives sliced up into 3 or more pieces. For better or worse, most other platforms don't know what datasets are and don't know how to handle them well. That's a shortcoming of those other platforms. Such problems are not unique to z/OS datasets. As another example, extended attributes (and resource forks) typically won't survive FTP transfers unless, again, you know what you're doing. Extended file attributes and/or resource forks are found in AIX (JFS2 v2), FreeBSD (UFS1 and UFS2), Linux (not commonly used but available), Mac OS X (10.4 and higher), OS/2 (Version 1.2 and higher), Solaris, and Windows NT (and successors; not commonly used but available). Filenames and directory names can get mangled across platforms. Some systems are case retentive but not case sensitive, and others are case sensitive. Date and time stamp formats vary, and it's complicated to preserve timestamps since various systems store (or don't store) various timestamps. FTP doesn't automatically address any of these permutations, on any platform -- and sometimes FTP doesn't even offer any viable "manual" SITE options either. I'm only scratching the surface, and these issues are likely to get more numerous and more complex in the future as file/storage systems evolve. I have some suggestions: 0. Should you be transferring datasets in the first place? Many file transfers simply shouldn't be happening, for security and other reasons. There are many, many alternative data access methods, most with better security attributes. You also have more options than ever to operate directly on z/OS-hosted data without copying it. That's the zeroth step, to make sure you ought to be transferring the data. 1. Assuming file transfer is the right option, avoid needless platform intermediaries. If you're transferring from z/OS to z/OS, then transfer from z/OS to z/OS. (Yes, agreed, MVSPUT and MVSGET in z/OS 2.1 and higher are great.) There are many options to do that. If somebody tries to argue "security," I really wonder if that person has considered whether inserting a Windows PC in the middle is more secure. 2. Use a packager/archiver appropriate for your platform as part of the file transfer process, preferably with in-memory processing. As examples: z/OS's AMATERSE and the IBM Encryption Facility for z/OS. (Does anybody have a nice in-memory example to share? Optionally, submit a RFE to IBM to add AMATERSE and/or Encryption Facility processing into the z/OS FTP "loop," as a subcommand option of some kind.) 3. Work with the IETF to see if there's room to enhance RFCs 959 and 4217, the two primary standards documents for FTP. 4. Choose an alternative file transfer protocol. z/OS includes IND$FILE, NFS (client and server), and CIFS/SMB (z/OS as server), as examples. SFTP is also available, although you'll probably want to add Co:Z SFTP. IBM Sterling Connect:Direct and MQ File Transfer Edition are examples of commercial file transfer products. As a "weird" example, Columbia University's Kermit is traditionally famous for moving data between disparate platforms, and it's still around and available from The Kermit Project. There are several other examples. -------------------------------------------------------------------------------------------------------- Timothy Sipples IT Architect Executive, Industry Solutions, IBM z Systems, AP/GCG/MEA -------------------------------------------------------------------------------------------------------- E-Mail: [email protected] ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
