Re: idea for a universal disk interface
I'm using Zynq SOMs (System on a module) that will plug into a "base board" (with hilrose connectors). It is the base board that will have the "personality" of the emulator. The baseboard will be fairly simple (level shifters, a small bit of logic and the drive interface transceivers). So the base board is fairly simple (I think I have an early version in KiCAD...but it needs updating). I'm trying to use as much as I can from the free libraries so I'm trying to keep stuff as simple as possible from a logic design perspective. Since I already have everything (in multiples) except for the base board, the cost to me is time at this point (which I don't have a lot of at the moment). I also didn't want to get into doing any design with BGAs (at least where I need to worry about it) hence the decision to go with SOMs. With those, the SOM has the Zynq FPGA, flash, DRAM, etc (including the critical VRs and clocks). All I need to provide is 3.3v. ;-) I should be able to dig up the docs. Many are already on bitsavers. Let me know what you can't find on Bitsavers. TTFN - Guy On 4/20/22 11:22, shad via cctech wrote: Guy, I agree that accessing data in blockram (synchronous with fixed latency) is really easier than accessing it from RAM (asynchronous with variable latency). Anyway I'm weighting the "cost" of additional complexity, which in change would allow to spare on Zynq cost. In any case memory access is never sequential, but a sequence of bursts with shorter length (16 beats or less). Considering this, the fact of starting or ending a sequential transfer is just a matter of generating addresses multiple of burst length. For this however you have to forget about Xilinx's free IP cores, and work directly with AXI3 bus of HP ports. As I would have to invest a large amount of time and of money, it would be nice to have somebody interested in buying a working and assembled kit with moderate price gain, in way to repay part of the investment. This however drives to bottom end FPGAs, with very limited amount of internal memory... whence the memory-spare design. About documentation: you mentioned several documents about SMD/ESDI standards and related details. Would you mind sharing this collection? Many thanks. Andrea -- TTFN - Guy
Re: idea for a universal disk interface
Guy, I agree that accessing data in blockram (synchronous with fixed latency) is really easier than accessing it from RAM (asynchronous with variable latency). Anyway I'm weighting the "cost" of additional complexity, which in change would allow to spare on Zynq cost. In any case memory access is never sequential, but a sequence of bursts with shorter length (16 beats or less). Considering this, the fact of starting or ending a sequential transfer is just a matter of generating addresses multiple of burst length. For this however you have to forget about Xilinx's free IP cores, and work directly with AXI3 bus of HP ports. As I would have to invest a large amount of time and of money, it would be nice to have somebody interested in buying a working and assembled kit with moderate price gain, in way to repay part of the investment. This however drives to bottom end FPGAs, with very limited amount of internal memory... whence the memory-spare design. About documentation: you mentioned several documents about SMD/ESDI standards and related details. Would you mind sharing this collection? Many thanks. Andrea
Re: idea for a universal disk interface
Good data, thanks! I kind of like the ESDI disks, they're more solid and reliable than the MFM ones, and to be honest the RQDX3 was not a super fast disk controller. I wonder if it could do block mode DMA. I'll keep an eye out for the Sigma, the only thing I wish I had on the MTI MQD13 would be some disk cache to speed things up. Granted 11/M+ does have disk caching in the OS, I should check to see how much quicker that makes things. Good used ESDI disks come up on Ebay from time to time. A nice 660mb CDC is enough for most general pdp usage... C On 4/19/2022 1:03 PM, Glen Slick via cctech wrote: I also have multiple ESDI controllers, more than one these flavors: Dilog DQ686 http://www.bitsavers.org/pdf/dilog/2120-0137-1_DQ686_Nov89.pdf Emulex QD21 http://www.bitsavers.org/pdf/emulex/QD2151002-J_QD21_Jun90.pdf Sigma SCD-RQD11-EC (There seems to be multiple versions from different vendors of this same basic board). http://www.bitsavers.org/pdf/sigmaInformationSystems/400740-B_SDC-RQD11-EC_Disk_Ctrl_Man_Jul88.pdf They all support block mode DMA transfers, and command queuing with seek optimization. The Dilog DQ686 and Emulex QD21 are dual wide boards. The Sigma SCD-RQD11-EC is a quad wide board and has 1MB of cache memory (which takes up about a quarter of the board area). The examples I have might only be populated with 512KB of cache memory. I might have had close to a dozen working full height 5.25-inch ESDI drives at one point. Unfortunately most of them have failed while sitting idle over the last few years. Without checking now I don't know if any of them still work. So the dozen or so Q-Bus ESDI controllers don't have any use for me now. (Fortunately I also have more Q-Bus SCSI controllers than backplanes to put them in). I also have a single Andromeda ESDC ESDI controller. Never found a manual for that one. Did eventually figure out how to get into the on-board configuration utility. On Tue, Apr 19, 2022 at 8:56 AM Douglas Taylor via cctech wrote: Once upon a time I used an Emulex QD21, but I sold it because the actual ESDI disks I had were a pain in the butt. Always crashing. I still have a Webster (quad board) SRQD something. I think I had a Dilog board also. It's been a while, probably 20 years. Doug On 4/18/2022 9:12 PM, Chris Zach via cctech wrote: Interesting, what kind of ESDI controllers do you have? They got advanced features like cache, ordered seeks, and burst mode/block mode DMA?
Re: idea for a universal disk interface
I also have multiple ESDI controllers, more than one these flavors: Dilog DQ686 http://www.bitsavers.org/pdf/dilog/2120-0137-1_DQ686_Nov89.pdf Emulex QD21 http://www.bitsavers.org/pdf/emulex/QD2151002-J_QD21_Jun90.pdf Sigma SCD-RQD11-EC (There seems to be multiple versions from different vendors of this same basic board). http://www.bitsavers.org/pdf/sigmaInformationSystems/400740-B_SDC-RQD11-EC_Disk_Ctrl_Man_Jul88.pdf They all support block mode DMA transfers, and command queuing with seek optimization. The Dilog DQ686 and Emulex QD21 are dual wide boards. The Sigma SCD-RQD11-EC is a quad wide board and has 1MB of cache memory (which takes up about a quarter of the board area). The examples I have might only be populated with 512KB of cache memory. I might have had close to a dozen working full height 5.25-inch ESDI drives at one point. Unfortunately most of them have failed while sitting idle over the last few years. Without checking now I don't know if any of them still work. So the dozen or so Q-Bus ESDI controllers don't have any use for me now. (Fortunately I also have more Q-Bus SCSI controllers than backplanes to put them in). I also have a single Andromeda ESDC ESDI controller. Never found a manual for that one. Did eventually figure out how to get into the on-board configuration utility. On Tue, Apr 19, 2022 at 8:56 AM Douglas Taylor via cctech wrote: > > Once upon a time I used an Emulex QD21, but I sold it because the actual > ESDI disks I had were a pain in the butt. Always crashing. > I still have a Webster (quad board) SRQD something. > I think I had a Dilog board also. It's been a while, probably 20 years. > Doug > > On 4/18/2022 9:12 PM, Chris Zach via cctech wrote: > > Interesting, what kind of ESDI controllers do you have? They got > > advanced features like cache, ordered seeks, and burst mode/block mode > > DMA?
Re: idea for a universal disk interface
Once upon a time I used an Emulex QD21, but I sold it because the actual ESDI disks I had were a pain in the butt. Always crashing. I still have a Webster (quad board) SRQD something. I think I had a Dilog board also. It's been a while, probably 20 years. Doug On 4/18/2022 9:12 PM, Chris Zach via cctech wrote: Interesting, what kind of ESDI controllers do you have? They got advanced features like cache, ordered seeks, and burst mode/block mode DMA? C On 4/18/2022 6:09 PM, Douglas Taylor via cctech wrote: Because of this I'm holding on to my DEC Qbus ESDI controllers!!! You never know Doug On 4/17/2022 4:35 PM, Guy Sotomayor via cctech wrote: I chose ESDI and SMD fundamentally because the interface is 100% digital (e.g. the data/clock separator is in the drive itself). So I don't need to do any oversampling. TTFN - Guy On 4/17/22 11:12, Paul Koning via cctalk wrote: On Apr 17, 2022, at 1:28 PM, shad via cctalk wrote: hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. For reading a disk, an attractive approach is to do a high speed analog capture of the waveforms. That way you don't need a priori knowledge of the encoding, and it also allows you to use sophisticated algorithms (DSP, digital filtering, etc.) to recover marginal media. A number of old tape recovery projects have used this approach. For disk you have to go faster if you use an existing drive, but the numbers are perfectly manageable with modern hardware. If you use this technique, you do generate a whole lot more data than the formatted capacity of the drive; 10x to 100x or so. Throw in another order of magnitude if you step across the surface in small increments to avoid having to identify the track centerline in advance -- again, somewhat like the tape recovery machines that use a 36 track head to read 7 or 9 or 10 track tapes. Fred mentioned how life gets hard if you don't have a drive. I'm wondering how difficult it would be to build a useable "spin table", basically an accurate spindle that will accept the pack to be recovered and that will rotate at a modest speed, with a head positioner that can accurately position a read head along the surface. One head would suffice, RAMAC fashion. For slow rotation you'd want an MR head, and perhaps supplied air to float the head off the surface. Perhaps a scheme like this with slow rotation could allow for recovery much of the data on a platter that suffered a head crash, because you could spin it slowly enough that either the head doesn't touch the scratched areas, or touches it slowly enough that no further damage results. paul
Re: idea for a universal disk interface
Interesting, what kind of ESDI controllers do you have? They got advanced features like cache, ordered seeks, and burst mode/block mode DMA? C On 4/18/2022 6:09 PM, Douglas Taylor via cctech wrote: Because of this I'm holding on to my DEC Qbus ESDI controllers!!! You never know Doug On 4/17/2022 4:35 PM, Guy Sotomayor via cctech wrote: I chose ESDI and SMD fundamentally because the interface is 100% digital (e.g. the data/clock separator is in the drive itself). So I don't need to do any oversampling. TTFN - Guy On 4/17/22 11:12, Paul Koning via cctalk wrote: On Apr 17, 2022, at 1:28 PM, shad via cctalk wrote: hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. For reading a disk, an attractive approach is to do a high speed analog capture of the waveforms. That way you don't need a priori knowledge of the encoding, and it also allows you to use sophisticated algorithms (DSP, digital filtering, etc.) to recover marginal media. A number of old tape recovery projects have used this approach. For disk you have to go faster if you use an existing drive, but the numbers are perfectly manageable with modern hardware. If you use this technique, you do generate a whole lot more data than the formatted capacity of the drive; 10x to 100x or so. Throw in another order of magnitude if you step across the surface in small increments to avoid having to identify the track centerline in advance -- again, somewhat like the tape recovery machines that use a 36 track head to read 7 or 9 or 10 track tapes. Fred mentioned how life gets hard if you don't have a drive. I'm wondering how difficult it would be to build a useable "spin table", basically an accurate spindle that will accept the pack to be recovered and that will rotate at a modest speed, with a head positioner that can accurately position a read head along the surface. One head would suffice, RAMAC fashion. For slow rotation you'd want an MR head, and perhaps supplied air to float the head off the surface. Perhaps a scheme like this with slow rotation could allow for recovery much of the data on a platter that suffered a head crash, because you could spin it slowly enough that either the head doesn't touch the scratched areas, or touches it slowly enough that no further damage results. paul
Re: idea for a universal disk interface
Which is more generic. ESDI, SMD or SCSI. In my opinion, SCSI is as close are you are going to get to a universal interface. As for reading raw data from a drive. The newer the drive, the higher the bit density and lower the strength of the magnetic fields and hence the lower the flying height. You have to deal with linear (or horizontal) recording, perpendicular recording, Heat assisted magnetic recording, microwave assisted magnetic recording. The latest technologies are approaching 1TB (yes that's TB) per square inch. If you spin the platters too slowly you will not be able to distinguish individual magnetic fluctuations from noise. What do you propose as your maximum data density (in BPI) and what is the minimum speed you will need to accurately decode it. Also once you get into newer drives the sectoring information may be stored in EEPROM or ROM and not on the disk at all, that would make decoding the data even more complicated as you don't know where a given sector starts. In addition to handling any number of encoding techniques (FM, MFM, GCR, RLL, MAMR, HAMR, PMR, etc.) news drives also do bad sector mapping so you would need to be able to handle that as well. Some kind of programmable data separator (possibly and external DSP) might be able to handle the high aerial densities. The GreaseWeazle does this for floppies. That architecture might be a good starting point to ramp up for hard drives. Well, that's my 2 cents. On 4/18/2022 5:09 PM, Douglas Taylor via cctech wrote: Because of this I'm holding on to my DEC Qbus ESDI controllers!!! You never know Doug On 4/17/2022 4:35 PM, Guy Sotomayor via cctech wrote: I chose ESDI and SMD fundamentally because the interface is 100% digital (e.g. the data/clock separator is in the drive itself). So I don't need to do any oversampling. TTFN - Guy On 4/17/22 11:12, Paul Koning via cctalk wrote: On Apr 17, 2022, at 1:28 PM, shad via cctalk wrote: hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. For reading a disk, an attractive approach is to do a high speed analog capture of the waveforms. That way you don't need a priori knowledge of the encoding, and it also allows you to use sophisticated algorithms (DSP, digital filtering, etc.) to recover marginal media. A number of old tape recovery projects have used this approach. For disk you have to go faster if you use an existing drive, but the numbers are perfectly manageable with modern hardware. If you use this technique, you do generate a whole lot more data than the formatted capacity of the drive; 10x to 100x or so. Throw in another order of magnitude if you step across the surface in small increments to avoid having to identify the track centerline in advance -- again, somewhat like the tape recovery machines that use a 36 track head to read 7 or 9 or 10 track tapes. Fred mentioned how life gets hard if you don't have a drive. I'm wondering how difficult it would be to build a useable "spin table", basically an accurate spindle that will accept the pack to be recovered and that will rotate at a modest speed, with a head positioner that can accurately position a read head along the surface. One head would suffice, RAMAC fashion. For slow rotation you'd want an MR head, and perhaps supplied air to float the head off the surface. Perhaps a scheme like this with slow rotation could allow for recovery much of the data on a platter that suffered a head crash, because you could spin it slowly enough that either the head doesn't touch the scratched areas, or touches it slowly enough that no further damage results. paul
Re: idea for a universal disk interface
Because of this I'm holding on to my DEC Qbus ESDI controllers!!! You never know Doug On 4/17/2022 4:35 PM, Guy Sotomayor via cctech wrote: I chose ESDI and SMD fundamentally because the interface is 100% digital (e.g. the data/clock separator is in the drive itself). So I don't need to do any oversampling. TTFN - Guy On 4/17/22 11:12, Paul Koning via cctalk wrote: On Apr 17, 2022, at 1:28 PM, shad via cctalk wrote: hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. For reading a disk, an attractive approach is to do a high speed analog capture of the waveforms. That way you don't need a priori knowledge of the encoding, and it also allows you to use sophisticated algorithms (DSP, digital filtering, etc.) to recover marginal media. A number of old tape recovery projects have used this approach. For disk you have to go faster if you use an existing drive, but the numbers are perfectly manageable with modern hardware. If you use this technique, you do generate a whole lot more data than the formatted capacity of the drive; 10x to 100x or so. Throw in another order of magnitude if you step across the surface in small increments to avoid having to identify the track centerline in advance -- again, somewhat like the tape recovery machines that use a 36 track head to read 7 or 9 or 10 track tapes. Fred mentioned how life gets hard if you don't have a drive. I'm wondering how difficult it would be to build a useable "spin table", basically an accurate spindle that will accept the pack to be recovered and that will rotate at a modest speed, with a head positioner that can accurately position a read head along the surface. One head would suffice, RAMAC fashion. For slow rotation you'd want an MR head, and perhaps supplied air to float the head off the surface. Perhaps a scheme like this with slow rotation could allow for recovery much of the data on a platter that suffered a head crash, because you could spin it slowly enough that either the head doesn't touch the scratched areas, or touches it slowly enough that no further damage results. paul
Re: idea for a universal disk interface
Guy, I agree on keeping Linux out of the loop, to allow fast access on head location, selection. However, I'm not convinced on the fact that a whole cylinder must be on blockram to achieve this. Given that ram access is fast (on Zynq with PL working at 200MHz and HP port at 64bits I'm running at around 1200MB/s peak), logic can jump across the whole disk without the software intervention, it's just a matter of being able to calculate conversion from CHS to address and read with sufficient buffer. Probably using Xilinx IP cores could be a severe limit, as these are really full of bugs and inefficient implementations... but are free, so you can't argue. On software side, given that you can go also slow, there's no need for very complex driver development, just an user level UIO driver could make do. About language, I know very well VHDL, and it's a little bit at higher level than Verilog, so development with implementation parameters is maybe a little easier. About interfaces which doesn't have separated clock recovery: these need a sort of oversampling, but you don't need to store every sample, just the ones with state change. Leveraging on IOSERDES you can work at a multiple of internal clock. Please keep in consideration that the idea is to develop a single device that can work both as drive and as interface, so implementation should be reversible. Probably this is not very difficult to obtain, as fast data paths for read and write are already in opposite directions. Andrea > I have proceeded as far as full block diagrams (still have to write all > of the verilog) and basic SW architecture.? This is why I've had this > discussion.? I've thought about this *a lot* and have gone through > several iterations of what will or will not work given timing constraints. > > I have all of the components for putting a prototype together but I just > haven't had the time yet to write the verilog, the Linux device driver > and the "personality board".? That is, there is still a lot to do.? ;-) > > Some requirements that I've put on my design: > > * straight forward SW architecture > * SW is *not* time critical (that is I didn't want SW in the critical > path of keeping the data stream) > * Must be able to emulate any SMD/ESDI drive > * Must be able to match performance of the drive (or better) > * Must be able to work with any controller (ESDI or SMD...depending > upon interface) > > With those in mind, that's how I came up with my design. > > I found that the Zynq has sufficient Block RAM to contain a full > cylinder of 512KB.? I'm keeping a full cylinder because that allows > everything to be done in verilog except for seeks (see SW not being > required to be in the critical path).? If I didn't do that, then SW > would have to be involved in some aspects of head switch, etc and those > can have tight (<< 100us) latencies and I just didn't want to try and > get Linux to handle that.? Yes, I could use some form of RTOS (I'm > actually in the middle of writing one...but that's still a ways away) > but I don't see any that are really up to what I need/want to do for > this project. > > BTW, I'm basing my initial implementation on the Zynq 7020 which has 1GB > of DRAM.? However, I'm also planning on a "bigger/better" one based upon > the Zynq Ultrascale+ which has 4GB of DRAM so that I can support > multiple/larger drives. > > The amount required by Linux doesn't have to be large...I plan on having > the KMD just allocate a really big buffer (e.g. sufficient for > containing the entire drive image).? Linux will run happily in > 128MB-256MB since there won't be any GUI.? It could be significantly > less if I were to strip out everything that isn't needed by the kernel > and only have a basic shell for booting/debug.? My plan is to have the > emulated drive data and the configuration file on the SD card...so > there's no real user interaction necessary (and Linux would not be on > the SD card but on the embedded flash on the Zynq module). > > > I chose ESDI and SMD fundamentally because the interface is 100% digital > (e.g. the data/clock separator is in the drive itself). So I don't need > to do any oversampling.
RE: idea for a universal disk interface
Actually I am talking about emulating the bit stream, index to index - RLL encoded and containing gaps, marks, headers, data, CRC, ECC, etc. Exactly as the bit stream would come out of a theoretical disk drive, no bit shift, no write splices, no instantaneous speed variation, no long term speed variation. That means the controller will have a very easy time of it. My point is the ESDI/SME bit stream is 15 Mb/sec or lower and the others are lower still while any modern memory can transfer in the Gb/sec range so the track will arrive at the emulator hardware at much higher rate than the controller can absorb and since the track is coming from memory there is negligible latency. You seem to assume that the transfer out of the emulator can't start until the entire track is in the emulator - that's not what I am saying. To use your example, sure it takes 65us to transfer the entire track out of memory but it takes 16.67 msec to transfer it out of the emulator. I suggest transfer out of the emulator hardware can start with the first word into it. Tom -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Saturday, April 16, 2022 5:14 PM To: t.gard...@computer.org; cctech@classiccmp.org Subject: Re: idea for a universal disk interface I think the issue is that you're thinking of somehow emulating the formatted data. I'm working on just emulating the bit-stream as then it'll work with any controller and sector/track layout so I won't actually know what a sector really is (unless I do "hard sectoring" which some drives did support). At a 15Mhz clock rate, 30 bytes is 1.us. Not a lot of time. And frankly, that's defined by the controller and not the drive (though usually the drives specify some layout but that's only a recommendation). Dealing with drive speed variations doesn't solve anything because it's actually done via the drive itself (e.g. the drive provides the clock to the controller so any variation is already accounted for). The drive really cares about total bits (e.g. bits-per-inch) that the media supports. If we assume 32KB track at 500MB/s DMA transfer rate, that takes 65us. But as I've said, the spec says that the time between a head select and read is 15us or so, you can see that you can't just transfer a track and still meet the minimum timings. I will agree that you can probably take longer but I'm trying to have a design that can meet all of the minimum timings so I can emulate any drive/controller combination with at least the same performance as a real drive (and in many cases I can provide *much* higher performance). By keeping a full cylinder in the FPGA Block RAM I can keep the head select time < 1us (it's basically just selecting the high order address bits going to the block RAM). By keeping the entire disk image in DRAM, I can emulate any drive (that fits in the DRAM) with identical (or faster) performance. If I wanted to do something simpler (not much though) I could have a smaller DRAM (but since the Zynq modules I'm using have 1GB or 4GB of DRAM there isn't much motivation) but then any seek would be limited by access to the backing store. Also remember, in the worst case you have to write the previous track out if it was written to so that will slow things down as well. With the full image maintained in DRAM, any writes can be performed in a lazy manner in the background so that won't impact the performance of the emulated drive. TTFN - Guy On 4/16/22 14:32, Tom Gardner wrote: > -Original Message- > From: Guy Sotomayor [mailto:g...@shiresoft.com] > Sent: Friday, April 15, 2022 3:25 PM > To: t.gard...@computer.org; cctech@classiccmp.org > Subject: Re: idea for a universal disk interface > > I'm looking at what the spec says. ;-) The read command delay from the head > set command is 15us (so I was wrong) but still not a lot of time (that is > after a head set, a read command must be at least 15us later). > > > > - > And after the read command is given there is a gap, usually all zeros, at the > end of which is a sync byte which is then followed by the first good data (or > header) byte. In SMD the gaps can be 20 or 30 bytes long so there is quite > a bit of time until good data. > > Tom > > -- TTFN - Guy
Re: idea for a universal disk interface
I have proceeded as far as full block diagrams (still have to write all of the verilog) and basic SW architecture. This is why I've had this discussion. I've thought about this *a lot* and have gone through several iterations of what will or will not work given timing constraints. I have all of the components for putting a prototype together but I just haven't had the time yet to write the verilog, the Linux device driver and the "personality board". That is, there is still a lot to do. ;-) Some requirements that I've put on my design: * straight forward SW architecture * SW is *not* time critical (that is I didn't want SW in the critical path of keeping the data stream) * Must be able to emulate any SMD/ESDI drive * Must be able to match performance of the drive (or better) * Must be able to work with any controller (ESDI or SMD...depending upon interface) With those in mind, that's how I came up with my design. I found that the Zynq has sufficient Block RAM to contain a full cylinder of 512KB. I'm keeping a full cylinder because that allows everything to be done in verilog except for seeks (see SW not being required to be in the critical path). If I didn't do that, then SW would have to be involved in some aspects of head switch, etc and those can have tight (<< 100us) latencies and I just didn't want to try and get Linux to handle that. Yes, I could use some form of RTOS (I'm actually in the middle of writing one...but that's still a ways away) but I don't see any that are really up to what I need/want to do for this project. BTW, I'm basing my initial implementation on the Zynq 7020 which has 1GB of DRAM. However, I'm also planning on a "bigger/better" one based upon the Zynq Ultrascale+ which has 4GB of DRAM so that I can support multiple/larger drives. The amount required by Linux doesn't have to be large...I plan on having the KMD just allocate a really big buffer (e.g. sufficient for containing the entire drive image). Linux will run happily in 128MB-256MB since there won't be any GUI. It could be significantly less if I were to strip out everything that isn't needed by the kernel and only have a basic shell for booting/debug. My plan is to have the emulated drive data and the configuration file on the SD card...so there's no real user interaction necessary (and Linux would not be on the SD card but on the embedded flash on the Zynq module). TTFN - Guy On 4/17/22 10:28, shad via cctech wrote: hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. About logic implementation, we know that the device must be able to work with one cylinder at a time. Given RAM bandwidth, this doesn't means that it must fit completely in blockram, also it can be produced at output while it is read, so delay time is really the time between first data request and actual read response. In between an elastic FIFO is required to adapt synchronous constant rate transfer of the disk to the burst transfer toward RAM. Guy, you mentioned about development of a similar interface. So you already produced some working hardware? Andrea -- TTFN - Guy
Re: idea for a universal disk interface
hello, there's much discussion about the right method to transfer data in and out. Of course there are several methods, the right one must be carefully chosen after some review of all the disk interfaces that must be supported. The idea of having a copy of the whole disk in RAM is OK, assuming that a maximum size of around 512MB is required, as the RAM is also needed for the OS, and for Zynq maximum is 1GB. About logic implementation, we know that the device must be able to work with one cylinder at a time. Given RAM bandwidth, this doesn't means that it must fit completely in blockram, also it can be produced at output while it is read, so delay time is really the time between first data request and actual read response. In between an elastic FIFO is required to adapt synchronous constant rate transfer of the disk to the burst transfer toward RAM. Guy, you mentioned about development of a similar interface. So you already produced some working hardware? Andrea
Re: idea for a universal disk interface
I think the issue is that you're thinking of somehow emulating the formatted data. I'm working on just emulating the bit-stream as then it'll work with any controller and sector/track layout so I won't actually know what a sector really is (unless I do "hard sectoring" which some drives did support). At a 15Mhz clock rate, 30 bytes is 1.us. Not a lot of time. And frankly, that's defined by the controller and not the drive (though usually the drives specify some layout but that's only a recommendation). Dealing with drive speed variations doesn't solve anything because it's actually done via the drive itself (e.g. the drive provides the clock to the controller so any variation is already accounted for). The drive really cares about total bits (e.g. bits-per-inch) that the media supports. If we assume 32KB track at 500MB/s DMA transfer rate, that takes 65us. But as I've said, the spec says that the time between a head select and read is 15us or so, you can see that you can't just transfer a track and still meet the minimum timings. I will agree that you can probably take longer but I'm trying to have a design that can meet all of the minimum timings so I can emulate any drive/controller combination with at least the same performance as a real drive (and in many cases I can provide *much* higher performance). By keeping a full cylinder in the FPGA Block RAM I can keep the head select time < 1us (it's basically just selecting the high order address bits going to the block RAM). By keeping the entire disk image in DRAM, I can emulate any drive (that fits in the DRAM) with identical (or faster) performance. If I wanted to do something simpler (not much though) I could have a smaller DRAM (but since the Zynq modules I'm using have 1GB or 4GB of DRAM there isn't much motivation) but then any seek would be limited by access to the backing store. Also remember, in the worst case you have to write the previous track out if it was written to so that will slow things down as well. With the full image maintained in DRAM, any writes can be performed in a lazy manner in the background so that won't impact the performance of the emulated drive. TTFN - Guy On 4/16/22 14:32, Tom Gardner wrote: -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Friday, April 15, 2022 3:25 PM To: t.gard...@computer.org; cctech@classiccmp.org Subject: Re: idea for a universal disk interface I'm looking at what the spec says. ;-) The read command delay from the head set command is 15us (so I was wrong) but still not a lot of time (that is after a head set, a read command must be at least 15us later). - And after the read command is given there is a gap, usually all zeros, at the end of which is a sync byte which is then followed by the first good data (or header) byte. In SMD the gaps can be 20 or 30 bytes long so there is quite a bit of time until good data. Tom -- TTFN - Guy
RE: idea for a universal disk interface
-Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Friday, April 15, 2022 3:25 PM To: t.gard...@computer.org; cctech@classiccmp.org Subject: Re: idea for a universal disk interface I'm looking at what the spec says. ;-) The read command delay from the head set command is 15us (so I was wrong) but still not a lot of time (that is after a head set, a read command must be at least 15us later). - And after the read command is given there is a gap, usually all zeros, at the end of which is a sync byte which is then followed by the first good data (or header) byte. In SMD the gaps can be 20 or 30 bytes long so there is quite a bit of time until good data. Tom
Re: idea for a universal disk interface
I'm looking at what the spec says. ;-) The read command delay from the head set command is 15us (so I was wrong) but still not a lot of time (that is after a head set, a read command must be at least 15us later). Since I'm not looking at formatted data rate (just handling the raw bit stream) it doesn't really matter what the formatted rate is...and the formatted data is different between different controllers, so I don't want to even try to do that on the fly (and they might do tricks where different tracks/cylinders have different formats. If some one wants the "formatted" data, then I'd let them post process that off the captured data. As I said, I'm trying to do this with fairly simple logic and low cost storage (as such this isn't going to particularly cheap). I don't want to add another $100+ to the cost just to have a high performance drive when the HW is capable of doing a suitable job with a $10 SD card. In reality an SD card (from a storage perspective) is way overkill. We're talking about emulating drives with capacities < 1GB and good quality SD cards contain 32GB for $10 or so. TTFN - Guy On 4/15/22 12:12, Tom Gardner wrote: I haven't looked it up but I bet the head switch time is a lot longer than 1-2 usec - that's what the leading gap is for and the sync took most of the gap back in those days. The issue is sustained data rate isn't it? The ESMD raw data rate is 24 Mb/s but the formatted data is something like 80% of that or maybe 2.5 MB/sec. A modern HDD in sequential mode can sustain a much higher rate, e.g. Seagate SAS <https://www.seagate.com/files/www-content/solutions/mach-2-multi-actuator-hard-drive/files/tp714-dot-2-2006us-mach-2-technology-paper.pdf> at 520 MB/sec. My understanding is that the sectors are slipped and/or cylinders are horizontal so that head switching doesn't lose any revolutions. Maybe one would run into a problem at the cylinder seek moment so maybe one would have to keep each full emulated cylinder on the modern drive’s cylinder, but with Terabytes of data on a modern drive who cares about some wasted storage Tom -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Friday, April 15, 2022 10:56 AM To: t.gard...@computer.org; cctech@classiccmp.org Subject: Re: idea for a universal disk interface I ran the numbers for Zynq FPGAs. First of all for ESDI and SMD the head switch time is 1-2us (basically the time it takes for the clocks to re-lock on the new data). Two tracks isn't sufficient (which is the "other" track...you will be wrong). So I decided to go and have a full cylinder (I'm allowing for up to 32KB tracks and up to 16 heads) which is 512KB. The Zynq DMA from HW block RAM to DRAM (at 500MB/s) is ~1ms. Given that the previous cylinder could be dirty (e.g. has written data), the worst case seek time is ~2ms. This allows me to emulate any seek latency curve(s) I want. In my design, any dirty data is written back to storage in a lazy manner so the performance of the storage isn't really an issue. I should note that the Zynq 7020 module has 1GB of DRAM on it, so there is no additional cost to just put the entire disk contents in DRAM and I'm using the attached SD Card interface for storage (so you can use a $10 SD Card for storage). Adding a high speed disk interface (e.g. MD.2, PCIe, or other serially attached storage) would add additional cost in terms of having to create the interface as well as a reasonably fast drive and I don't see the advantage. I'm planning on using a Zynq UltraScale+ module to allow for larger disks and multiple disk emulations (it has more block RAM and 4GB of DRAM on the module). TTFN - Guy On 4/14/22 23:34, Tom Gardner wrote: > I suggest if we are talking about an emulator it really isn't necessary to have the entire disk in DRAM, two tracks of DRAM acting as a buffer with a modern HDD holding the emulated drive's data should be fast enough to keep any old iron controller operating without missing any revolutions. The maximum unformatted track length of any old iron drive is well known and therefore one can allocate the number of blocks sufficient to store a full track and then write every track, gaps and all to the modern disk. Given the data rate, track size and sequential seek times of a modern HDD one should be able to fill then next track buffer before the current track buffer is read into the controller. If two track buffers and an HDD isn't fast enough then one could add a track buffer or two or go to SSD's. > > This was the approach IBM used in it's first RAMAC RAID where I think > they had to buffer a whole cylinder but that was many generations ago > > Tom > > -Original Message- > From: Guy Sotomayor [mailto:g...@shiresoft.com <mailto:g...@shiresoft.com>] > Sent: Wednesday, April 13, 2022 10:02 AM > To: cctec
RE: idea for a universal disk interface
I haven't looked it up but I bet the head switch time is a lot longer than 1-2 usec - that's what the leading gap is for and the sync took most of the gap back in those days. The issue is sustained data rate isn't it? The ESMD raw data rate is 24 Mb/s but the formatted data is something like 80% of that or maybe 2.5 MB/sec. A modern HDD in sequential mode can sustain a much higher rate, e.g. Seagate SAS <https://www.seagate.com/files/www-content/solutions/mach-2-multi-actuator-hard-drive/files/tp714-dot-2-2006us-mach-2-technology-paper.pdf> at 520 MB/sec. My understanding is that the sectors are slipped and/or cylinders are horizontal so that head switching doesn't lose any revolutions. Maybe one would run into a problem at the cylinder seek moment so maybe one would have to keep each full emulated cylinder on the modern drive’s cylinder, but with Terabytes of data on a modern drive who cares about some wasted storage Tom -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Friday, April 15, 2022 10:56 AM To: t.gard...@computer.org; cctech@classiccmp.org Subject: Re: idea for a universal disk interface I ran the numbers for Zynq FPGAs. First of all for ESDI and SMD the head switch time is 1-2us (basically the time it takes for the clocks to re-lock on the new data). Two tracks isn't sufficient (which is the "other" track...you will be wrong). So I decided to go and have a full cylinder (I'm allowing for up to 32KB tracks and up to 16 heads) which is 512KB. The Zynq DMA from HW block RAM to DRAM (at 500MB/s) is ~1ms. Given that the previous cylinder could be dirty (e.g. has written data), the worst case seek time is ~2ms. This allows me to emulate any seek latency curve(s) I want. In my design, any dirty data is written back to storage in a lazy manner so the performance of the storage isn't really an issue. I should note that the Zynq 7020 module has 1GB of DRAM on it, so there is no additional cost to just put the entire disk contents in DRAM and I'm using the attached SD Card interface for storage (so you can use a $10 SD Card for storage). Adding a high speed disk interface (e.g. MD.2, PCIe, or other serially attached storage) would add additional cost in terms of having to create the interface as well as a reasonably fast drive and I don't see the advantage. I'm planning on using a Zynq UltraScale+ module to allow for larger disks and multiple disk emulations (it has more block RAM and 4GB of DRAM on the module). TTFN - Guy On 4/14/22 23:34, Tom Gardner wrote: > I suggest if we are talking about an emulator it really isn't necessary to > have the entire disk in DRAM, two tracks of DRAM acting as a buffer with a > modern HDD holding the emulated drive's data should be fast enough to keep > any old iron controller operating without missing any revolutions. The > maximum unformatted track length of any old iron drive is well known and > therefore one can allocate the number of blocks sufficient to store a full > track and then write every track, gaps and all to the modern disk. Given the > data rate, track size and sequential seek times of a modern HDD one should be > able to fill then next track buffer before the current track buffer is read > into the controller. If two track buffers and an HDD isn't fast enough then > one could add a track buffer or two or go to SSD's. > > This was the approach IBM used in it's first RAMAC RAID where I think > they had to buffer a whole cylinder but that was many generations ago > > Tom > > -Original Message- > From: Guy Sotomayor [ <mailto:g...@shiresoft.com> mailto:g...@shiresoft.com] > Sent: Wednesday, April 13, 2022 10:02 AM > To: <mailto:cctech@classiccmp.org> cctech@classiccmp.org > Subject: Re: idea for a universal disk interface > > I've had a similar project in the works for a while (mainly for ESDI and SMD). > > I think the main issue you're going to face is that what you need to do for > something like ESDI or SMD (or any of the bit serial interfaces) is going to > be radically different than something like IDE or SCSI. This is not just the > interface signals but also what's needed in the FPGA as well as the embedded > SW. > > For example, for the ESDI and SMD interface in order to meet the head > switch times (1-2 microseconds) requires that a full cylinder be > cached in HW. Once you do that and look at the timings to move a max > cylinder between the HW cache (that will serialize/de-serialize the > data over the > interface) and storage, you'll see that the only way to have any > reasonable performance (e.g. not have seek times be > 40ms for *any* > seek) is to cache the entire drive image in DRAM and lazily write back dirty > tra
Re: idea for a universal disk interface
I ran the numbers for Zynq FPGAs. First of all for ESDI and SMD the head switch time is 1-2us (basically the time it takes for the clocks to re-lock on the new data). Two tracks isn't sufficient (which is the "other" track...you will be wrong). So I decided to go and have a full cylinder (I'm allowing for up to 32KB tracks and up to 16 heads) which is 512KB. The Zynq DMA from HW block RAM to DRAM (at 500MB/s) is ~1ms. Given that the previous cylinder could be dirty (e.g. has written data), the worst case seek time is ~2ms. This allows me to emulate any seek latency curve(s) I want. In my design, any dirty data is written back to storage in a lazy manner so the performance of the storage isn't really an issue. I should note that the Zynq 7020 module has 1GB of DRAM on it, so there is no additional cost to just put the entire disk contents in DRAM and I'm using the attached SD Card interface for storage (so you can use a $10 SD Card for storage). Adding a high speed disk interface (e.g. MD.2, PCIe, or other serially attached storage) would add additional cost in terms of having to create the interface as well as a reasonably fast drive and I don't see the advantage. I'm planning on using a Zynq UltraScale+ module to allow for larger disks and multiple disk emulations (it has more block RAM and 4GB of DRAM on the module). TTFN - Guy On 4/14/22 23:34, Tom Gardner wrote: I suggest if we are talking about an emulator it really isn't necessary to have the entire disk in DRAM, two tracks of DRAM acting as a buffer with a modern HDD holding the emulated drive's data should be fast enough to keep any old iron controller operating without missing any revolutions. The maximum unformatted track length of any old iron drive is well known and therefore one can allocate the number of blocks sufficient to store a full track and then write every track, gaps and all to the modern disk. Given the data rate, track size and sequential seek times of a modern HDD one should be able to fill then next track buffer before the current track buffer is read into the controller. If two track buffers and an HDD isn't fast enough then one could add a track buffer or two or go to SSD's. This was the approach IBM used in it's first RAMAC RAID where I think they had to buffer a whole cylinder but that was many generations ago Tom -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Wednesday, April 13, 2022 10:02 AM To: cctech@classiccmp.org Subject: Re: idea for a universal disk interface I've had a similar project in the works for a while (mainly for ESDI and SMD). I think the main issue you're going to face is that what you need to do for something like ESDI or SMD (or any of the bit serial interfaces) is going to be radically different than something like IDE or SCSI. This is not just the interface signals but also what's needed in the FPGA as well as the embedded SW. For example, for the ESDI and SMD interface in order to meet the head switch times (1-2 microseconds) requires that a full cylinder be cached in HW. Once you do that and look at the timings to move a max cylinder between the HW cache (that will serialize/de-serialize the data over the interface) and storage, you'll see that the only way to have any reasonable performance (e.g. not have seek times be > 40ms for *any* seek) is to cache the entire drive image in DRAM and lazily write back dirty tracks. I've been looking at the Xylinx Zynq SoCs for this (mainly the Zynq 7020 for single drive emulation and the Zynq Ultrascale+ for up to 4 drives). In my case the HW, FPGA logic and SW will share significant portions but they will not be identical. In my case there is no need for an external PC (just adds complexity) other than something to do basic configuration (e.g. drive parameters such as number of heads, number of cylinders, etc) which will actually be over USB/serial. The actual persistent storage will be an SD card since all reading will be done at "boot time" and writes will be handled in a lazy manner (since the writes will first go to the DRAM based upon time or seek). It may also be sufficient for configuration purposes to have a file (text) on the SD card that defines the configuration so no external interactions would be necessary. I'm still thinking about that one. ;-) TTFN - Guy On 4/12/22 22:35, shad via cctech wrote: Hello, I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. I'm often facing common problems with storage devices, magnetic discs and tapes are a little prone to give headaches after years, and replacement drives/media in case of a severe failure are unobtainable. In some cases, the ability to make a dump of the media, also without a running computer is very important. Whence the idea: realize an universal device, with several input/output interfaces, which could be used both as s
RE: idea for a universal disk interface
I suggest if we are talking about an emulator it really isn't necessary to have the entire disk in DRAM, two tracks of DRAM acting as a buffer with a modern HDD holding the emulated drive's data should be fast enough to keep any old iron controller operating without missing any revolutions. The maximum unformatted track length of any old iron drive is well known and therefore one can allocate the number of blocks sufficient to store a full track and then write every track, gaps and all to the modern disk. Given the data rate, track size and sequential seek times of a modern HDD one should be able to fill then next track buffer before the current track buffer is read into the controller. If two track buffers and an HDD isn't fast enough then one could add a track buffer or two or go to SSD's. This was the approach IBM used in it's first RAMAC RAID where I think they had to buffer a whole cylinder but that was many generations ago Tom -Original Message- From: Guy Sotomayor [mailto:g...@shiresoft.com] Sent: Wednesday, April 13, 2022 10:02 AM To: cctech@classiccmp.org Subject: Re: idea for a universal disk interface I've had a similar project in the works for a while (mainly for ESDI and SMD). I think the main issue you're going to face is that what you need to do for something like ESDI or SMD (or any of the bit serial interfaces) is going to be radically different than something like IDE or SCSI. This is not just the interface signals but also what's needed in the FPGA as well as the embedded SW. For example, for the ESDI and SMD interface in order to meet the head switch times (1-2 microseconds) requires that a full cylinder be cached in HW. Once you do that and look at the timings to move a max cylinder between the HW cache (that will serialize/de-serialize the data over the interface) and storage, you'll see that the only way to have any reasonable performance (e.g. not have seek times be > 40ms for *any* seek) is to cache the entire drive image in DRAM and lazily write back dirty tracks. I've been looking at the Xylinx Zynq SoCs for this (mainly the Zynq 7020 for single drive emulation and the Zynq Ultrascale+ for up to 4 drives). In my case the HW, FPGA logic and SW will share significant portions but they will not be identical. In my case there is no need for an external PC (just adds complexity) other than something to do basic configuration (e.g. drive parameters such as number of heads, number of cylinders, etc) which will actually be over USB/serial. The actual persistent storage will be an SD card since all reading will be done at "boot time" and writes will be handled in a lazy manner (since the writes will first go to the DRAM based upon time or seek). It may also be sufficient for configuration purposes to have a file (text) on the SD card that defines the configuration so no external interactions would be necessary. I'm still thinking about that one. ;-) TTFN - Guy On 4/12/22 22:35, shad via cctech wrote: > Hello, > I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. > I'm often facing common problems with storage devices, magnetic discs and > tapes are a little prone to give headaches after years, and replacement > drives/media in case of a severe failure are unobtainable. > In some cases, the ability to make a dump of the media, also without a > running computer is very important. > > Whence the idea: realize an universal device, with several input/output > interfaces, which could be used both as storage emulator, to run a computer > without real storage, and as controller emulator, to read/write a media > without a running computer. > To reduce costs as much as possible, and to allow the better compatibility, > the main board shall host enough electrical interfaces to support a large > number of disc standard interfaces, ideally by exchanging only a personality > adapter for each specific interface, i.e. connectors and few components. > > There are several orders of problems: > - electrical signals, number and type (most disk employ 5V TTL or 3.3V > TTL, some interfaces use differential mode for some faster signals?) > - logical implementation: several electrical signals are used for a > specific interface. These must be handled with correct timings > - software implementation: the universal device shall be able to > switch between interface modes and be controlled by a remote PC > > I suppose the only way to obtain this is to employ an FPGA for logic > implementation of the interface, and a microprocessor running Linux to handle > software management, data interchange to external (via Ethernet). This means > a Xilinx Zynq module for instance. > I know there are several ready devices based on cheaper microcontrollers, but > I'm sure these can't support fast and tight timing require
RE: idea for a universal disk interface
The IMI 7710 34 pin flat cable interface is a variant on the SMD dumb interface which could be controlled by a UDI (universal disk interface) if someone cared enough to build an adapter and then program the UDI to deal with IMI's specific track format and peculiar command/status protocol. CalComp would be easier, but the question remains would either be worth the effort given their relatively low unit shipments compared to the other interfaces? Tom -Original Message- From: r.stricklin [mailto:b...@typewritten.org] Sent: Wednesday, April 13, 2022 8:12 PM To: t.gard...@computer.org; Tom Gardner; General Discussion: On-Topic Posts Subject: Re: idea for a universal disk interface > On Apr 13, 2022, at 11:51 AM, Tom Gardner via cctech > wrote: > > There are a few others like ANSI and CalComp but they are probably not worth > investigating. > They are if you�re someone who has a machine using one of these interfaces, or e.g. the 40-pin �IMI bus�, or whatever else. ok bear.=
Re: idea for a universal disk interface
> On Apr 13, 2022, at 9:45 PM, Fred Cisin via cctech > wrote: > > On Wed, 13 Apr 2022, Paul Koning wrote: >> Indeed. Though even that is hard for the more exotic formats, if original >> controllers are unavailable. How would you read, for example, an IBM 1620 >> or CDC 6600 disk pack, given that the machine is hard to find and those that >> exists may not have the right controllers? But both are certainly doable >> with a "generic" track extractor engine. Turning track waveforms into data >> then becomes a matter of reverse engineering the encoding and constructing >> the software to do that. This may be hard, or not so hard. For example, if >> you wanted to do this for a CDC 844 disk pack (RP04/06 lookalike but with >> 322 12-bit word sectors) you'd get a lot of help from the fact that the >> source code of the disk controller firmware, and the manual describing it, >> have been preserved. >> >> Then as you said the real goal is to recover files, which means also having >> to reverse engineer the file system. That too may be documented adequately >> (it is in the 6600 case, for example) or not so much (does such >> documentation, or the OS source code, exists for the 1620 disk operating >> system?). > > > Some projects are well beyond the reach of even the most insane of us. > > I don't think that any of us here today have the ability to build a > replacement drive from scratch. Even with full access to the original > construction documents. > > Now, if we had NSA level of facilities, . . . I don't think a 1970s era disk drive replica is quite as hard as you suggest. In my comment I wasn't actually thinking of that, but rather of the possibility that you might have a drive and packs, but no computer to connect the drive to. That said, consider what, say, a 1311 looks like. It's a spindle and a head carriage, each with the levels of precision you would find on a good quality lathe. That suggests the guts of a small CNC lathe, or the building blocks of such a thing, could be put to work for this. One data point: I remember when our RC11 spindle bearings failed (in college, 1974). DEC FS was called in, the tech decided to look for a low cost solution since the machine was not under contract. The normal procedure would have been to replace the spindle motor, of course. Instead, he disassembled the drive, took the motor to Appleton Motor Service Co., which pulled off the failed bearings and pressed on new ones, reinstalled the spindle motor, presto, good as new. He didn't even have to reformat the drive, all the data on it remained intact. So the tolerances on drives of that era are not all that severe, not out of reach for ordinary skilled machinists. paul
Re: idea for a universal disk interface
> On Apr 13, 2022, at 8:12 PM, Fred Cisin via cctech > wrote: > > ... > My mindset is/was still stuck in the disk format conversion realm, of trying > to get information (hopefully information in the form of files, not just data > as track images) from alien media. > And, more often than not, unidirectionally. Indeed. Though even that is hard for the more exotic formats, if original controllers are unavailable. How would you read, for example, an IBM 1620 or CDC 6600 disk pack, given that the machine is hard to find and those that exists may not have the right controllers? But both are certainly doable with a "generic" track extractor engine. Turning track waveforms into data then becomes a matter of reverse engineering the encoding and constructing the software to do that. This may be hard, or not so hard. For example, if you wanted to do this for a CDC 844 disk pack (RP04/06 lookalike but with 322 12-bit word sectors) you'd get a lot of help from the fact that the source code of the disk controller firmware, and the manual describing it, have been preserved. Then as you said the real goal is to recover files, which means also having to reverse engineer the file system. That too may be documented adequately (it is in the 6600 case, for example) or not so much (does such documentation, or the OS source code, exists for the 1620 disk operating system?). paul
Re: idea for a universal disk interface
> On Apr 13, 2022, at 5:27 PM, Fred Cisin via cctech > wrote: > > On Wed, 13 Apr 2022, shad via cctech wrote: >> The main board should include a large enough array of bidirectional >> transceivers, possibly with variable voltage, to support as much interfaces >> as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, >> SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. > > Hmmm. rather than re-inventing the wheel, as we usually do, . . . > > It may be possible to accomplish a subset of those, specifically including > Shugart floppy, ST506/412 MFM, RLL, ESDI, IDE, SASI, SCSI by repurposing > certain commercial hardware. > > You would have a collection of boards, that you would remove/insert into a > connector. > > The main board would have RAM, and a ROM for certain basic (and BASIC?) > functions, but would load software for various modules and output results to > and from one or more interfaces that remain connected. > > I don't doubt that you could design a far better system, but there already > exists a crude version, ready to implement! > > It has a marginal power supply, and it has a poorly designed group of 8 62 > pin connectors for the interfaces, although some of those would need to be > dedicated for other functions of the device, including user interface > hardware. Some software is already available, but some crude tools are > available for creating more. > > It says "IBM", "5160" on the back panel label, although there were plenty of > generic second sources. > The updated "5170" version of it could be more easily set up even for USB. :-) But the main goal that was mentioned is device emulation, which existing products generally don't do. I see the idea as a generalized form of David Gesswein's MFM emulator, which is primarily a device emulator but is also capable of reading and writing devices to capture everything that's on them. The puzzle is how to make it do, say, 2311 emulation suitable to talk to an IBM/360, 1311 emulation for the decimal IBM 1620, or 844 emulation for a CDC 6600 -- to mention just a few of the more exotic cases. paul
Re: idea for a universal disk interface
> On Apr 13, 2022, at 11:51 AM, Tom Gardner via cctech > wrote: > > There are a few others like ANSI and CalComp but they are probably not worth > investigating. > They are if you’re someone who has a machine using one of these interfaces, or e.g. the 40-pin “IMI bus”, or whatever else. ok bear.
Re: idea for a universal disk interface
On Wed, 13 Apr 2022, Paul Koning wrote: Indeed. Though even that is hard for the more exotic formats, if original controllers are unavailable. How would you read, for example, an IBM 1620 or CDC 6600 disk pack, given that the machine is hard to find and those that exists may not have the right controllers? But both are certainly doable with a "generic" track extractor engine. Turning track waveforms into data then becomes a matter of reverse engineering the encoding and constructing the software to do that. This may be hard, or not so hard. For example, if you wanted to do this for a CDC 844 disk pack (RP04/06 lookalike but with 322 12-bit word sectors) you'd get a lot of help from the fact that the source code of the disk controller firmware, and the manual describing it, have been preserved. Then as you said the real goal is to recover files, which means also having to reverse engineer the file system. That too may be documented adequately (it is in the 6600 case, for example) or not so much (does such documentation, or the OS source code, exists for the 1620 disk operating system?). Some projects are well beyond the reach of even the most insane of us. I don't think that any of us here today have the ability to build a replacement drive from scratch. Even with full access to the original construction documents. Now, if we had NSA level of facilities, . . . It certainly seems that it would be THEORETICALLY POSSIBLE, with an extreme budget, to build a high resolution device similar to the 3M Magnetic Tape viewer, . . . https://blog.adafruit.com/2020/03/01/the-magnetic-tape-viewer-see-the-sound-on-a-tape/ . . . and use it to make optical imaging of the magnetic recording, followed by non-trivial analysis to decode that into track images, and then ultimately deciphering the encoding, track structure, and then directory structure, . . . It is certainly not feasible now, but someday, . . . I have a RAMAC platter! It is seriously FAR too damaged to consider restoring it to usable form. I was also told that it was extensively "degaussed" when it was discarded (possibly by Zellerbach Paper). 100 cylinders (with 100 heads in assembled structure), holding 5 million 6 bit characters, or a bit less than 100K per platter. So, I am making a 24" patio table out of it (under 3/8" tempered glass). http://www.ed-thelen.org/RAMAC/RAMAC_Plaque_v40.pdf When Khrshchev was denied access to go to Disneyland, they took him on a tour of the RAMAC factory, "to make up for it". _I_ would rather be at the RAMAC factory in 1959 than at Disneyland, but the Khrushchevs were disappointed. -- Grumpy Ol' Fred ci...@xenosoft.com
Re: idea for a universal disk interface
I'll only mention that there were ICs that could interface to both MFM/ST506 hard drives as well as floppies (System/3 MFM). An example would be the SMC HDC9234, "Universal Disk Controller". Pretty cool chip for the time; has full 24 bit DMA address capability. But different register/controller setup than the stock PC AT. --Chuck
Re: idea for a universal disk interface
On Wed, 13 Apr 2022, shad via cctech wrote: The main board should include a large enough array of bidirectional transceivers, possibly with variable voltage, to support as much interfaces as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. On Apr 13, 2022, at 5:27 PM, Fred Cisin via cctech wrote: Hmmm. rather than re-inventing the wheel, as we usually do, . . . . . . It says "IBM", "5160" on the back panel label, although there were plenty of generic second sources. The updated "5170" version of it could be more easily set up even for USB. On Wed, 13 Apr 2022, Paul Koning wrote: :-) But the main goal that was mentioned is device emulation, which existing products generally don't do. I see the idea as a generalized form of David Gesswein's MFM emulator, which is primarily a device emulator but is also capable of reading and writing devices to capture everything that's on them. The puzzle is how to make it do, say, 2311 emulation suitable to talk to an IBM/360, 1311 emulation for the decimal IBM 1620, or 844 emulation for a CDC 6600 -- to mention just a few of the more exotic cases. You are absolutely right. I apologize for my error. My mindset is/was still stuck in the disk format conversion realm, of trying to get information (hopefully information in the form of files, not just data as track images) from alien media. And, more often than not, unidirectionally. I wasn't even thinking about the hardware emulation aspect, which is, in itself, fascinating. -- Grumpy Ol' Fred ci...@xenosoft.com
Re: idea for a universal disk interface
. . . and, if there is agreement to standardize the connection system for the "personality modules", then some of the other storage systems could be implemented, particularly including some of the tape systemmes. 'course, it would be a lot more fun, instead of the 62 pin card edge, to go with the 100 pin one that George Morrow and Howard Fullmer tried to standardize, . . .
Re: idea for a universal disk interface
On Wed, 13 Apr 2022, shad via cctech wrote: The main board should include a large enough array of bidirectional transceivers, possibly with variable voltage, to support as much interfaces as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. Hmmm. rather than re-inventing the wheel, as we usually do, . . . It may be possible to accomplish a subset of those, specifically including Shugart floppy, ST506/412 MFM, RLL, ESDI, IDE, SASI, SCSI by repurposing certain commercial hardware. You would have a collection of boards, that you would remove/insert into a connector. The main board would have RAM, and a ROM for certain basic (and BASIC?) functions, but would load software for various modules and output results to and from one or more interfaces that remain connected. I don't doubt that you could design a far better system, but there already exists a crude version, ready to implement! It has a marginal power supply, and it has a poorly designed group of 8 62 pin connectors for the interfaces, although some of those would need to be dedicated for other functions of the device, including user interface hardware. Some software is already available, but some crude tools are available for creating more. It says "IBM", "5160" on the back panel label, although there were plenty of generic second sources. The updated "5170" version of it could be more easily set up even for USB.
RE: idea for a universal disk interface
Interesting idea, there are three broad classes of HDD interfaces: 1. Dumb, that is serial data and parallel control 2. Intelligent parallel 3. Intelligent serial IMO if you can do dumb interraces then the others follow and given today’s technology I suspect it is feasible Within the dumb group there are several “standards” with very similar controls but with different data rates and track formats * 2311/2314 including PCM versions * DEC RL0x which is pretty much the same as 2311/2314 except interface voltages and goes up to 200 MB disk pack drives, maybe higher * Diablo which is a simplified control version of RL0x * SMD which is an enhanced control version of RL0x * ST502/412/412 RLL which is a simplified control version of RL0x * ESDI which unfortunately serialized some of the control over the parallel interface otherwise similar to RL0x There are a few others like ANSI and CalComp but they are probably not worth investigating. I don’t know who invented the 1311 interface but IMHO he spawned an industry :) I think the maximum data rate is something on the order of 15 MHz so one ought to be able to read in an entire track at a sufficiently high data rate so as to be able to decode the data using an appropriately programmed DSP. Essentially all the hardware used for serializing/deserializing, formatting/deformatting and ECC in a traditional controller reduced to firmware Likewise there is a maximum number of control pins and only two voltage signaling levels (IBMs and DTL) so a combination of a programmable transceiver with a personality module ought to allow connection to and signaling with any physical device. Communicating control and status then is also a programming exercise Writing all the firmware would be a challenge but in the end you would be dealing with an array of blocks of good data, which would then be the starting point for tackling the intelligent interfaces which purport to start and stop at the same array of blocks of good data. I think u might be able to make this work with all the flavors of SCSI (with maybe a DSP on the personality module to handle the bus states) but good luck with intelligent serial interfaces. Just my 2 cents :) tom -Original Message- From: shad [mailto:shado...@gmail.com] Sent: Tuesday, April 12, 2022 10:36 PM To: cctech@classiccmp.org <mailto:cctech@classiccmp.org> Subject: idea for a universal disk interface Hello, I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. I'm often facing common problems with storage devices, magnetic discs and tapes are a little prone to give headaches after years, and replacement drives/media in case of a severe failure are unobtainable. In some cases, the ability to make a dump of the media, also without a running computer is very important. Whence the idea: realize an universal device, with several input/output interfaces, which could be used both as storage emulator, to run a computer without real storage, and as controller emulator, to read/write a media without a running computer. To reduce costs as much as possible, and to allow the better compatibility, the main board shall host enough electrical interfaces to support a large number of disc standard interfaces, ideally by exchanging only a personality adapter for each specific interface, i.e. connectors and few components. There are several orders of problems: - electrical signals, number and type (most disk employ 5V TTL or 3.3V TTL, some interfaces use differential mode for some faster signals?) - logical implementation: several electrical signals are used for a specific interface. These must be handled with correct timings - software implementation: the universal device shall be able to switch between interface modes and be controlled by a remote PC I suppose the only way to obtain this is to employ an FPGA for logic implementation of the interface, and a microprocessor running Linux to handle software management, data interchange to external (via Ethernet). This means a Xilinx Zynq module for instance. I know there are several ready devices based on cheaper microcontrollers, but I'm sure these can't support fast and tight timing required by hard disk interfaces (SMD-E runs at 24MHz). The main board should include a large enough array of bidirectional transceivers, possibly with variable voltage, to support as much interfaces as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. The common factor determining what kind of disc interface can be support on hardware side is obviously the type of transceiver employed, for instance a SATA would require a differential serial channel, which could not be available. But most old electronic is based on TTL/C
Re: idea for a universal disk interface
I've had a similar project in the works for a while (mainly for ESDI and SMD). I think the main issue you're going to face is that what you need to do for something like ESDI or SMD (or any of the bit serial interfaces) is going to be radically different than something like IDE or SCSI. This is not just the interface signals but also what's needed in the FPGA as well as the embedded SW. For example, for the ESDI and SMD interface in order to meet the head switch times (1-2 microseconds) requires that a full cylinder be cached in HW. Once you do that and look at the timings to move a max cylinder between the HW cache (that will serialize/de-serialize the data over the interface) and storage, you'll see that the only way to have any reasonable performance (e.g. not have seek times be > 40ms for *any* seek) is to cache the entire drive image in DRAM and lazily write back dirty tracks. I've been looking at the Xylinx Zynq SoCs for this (mainly the Zynq 7020 for single drive emulation and the Zynq Ultrascale+ for up to 4 drives). In my case the HW, FPGA logic and SW will share significant portions but they will not be identical. In my case there is no need for an external PC (just adds complexity) other than something to do basic configuration (e.g. drive parameters such as number of heads, number of cylinders, etc) which will actually be over USB/serial. The actual persistent storage will be an SD card since all reading will be done at "boot time" and writes will be handled in a lazy manner (since the writes will first go to the DRAM based upon time or seek). It may also be sufficient for configuration purposes to have a file (text) on the SD card that defines the configuration so no external interactions would be necessary. I'm still thinking about that one. ;-) TTFN - Guy On 4/12/22 22:35, shad via cctech wrote: Hello, I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. I'm often facing common problems with storage devices, magnetic discs and tapes are a little prone to give headaches after years, and replacement drives/media in case of a severe failure are unobtainable. In some cases, the ability to make a dump of the media, also without a running computer is very important. Whence the idea: realize an universal device, with several input/output interfaces, which could be used both as storage emulator, to run a computer without real storage, and as controller emulator, to read/write a media without a running computer. To reduce costs as much as possible, and to allow the better compatibility, the main board shall host enough electrical interfaces to support a large number of disc standard interfaces, ideally by exchanging only a personality adapter for each specific interface, i.e. connectors and few components. There are several orders of problems: - electrical signals, number and type (most disk employ 5V TTL or 3.3V TTL, some interfaces use differential mode for some faster signals?) - logical implementation: several electrical signals are used for a specific interface. These must be handled with correct timings - software implementation: the universal device shall be able to switch between interface modes and be controlled by a remote PC I suppose the only way to obtain this is to employ an FPGA for logic implementation of the interface, and a microprocessor running Linux to handle software management, data interchange to external (via Ethernet). This means a Xilinx Zynq module for instance. I know there are several ready devices based on cheaper microcontrollers, but I'm sure these can't support fast and tight timing required by hard disk interfaces (SMD-E runs at 24MHz). The main board should include a large enough array of bidirectional transceivers, possibly with variable voltage, to support as much interfaces as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. The common factor determining what kind of disc interface can be support on hardware side is obviously the type of transceiver employed, for instance a SATA would require a differential serial channel, which could not be available. But most old electronic is based on TTL/CMOS 5V logic, so a large variety of computer generations should be doable. For the first phase, I would ask you to contribute with a list of interfaces which could be interesting to emulate, specially if these are similar to one from my list. I please submitters to send me by email or by web link when possible, detailed documentation about the interface they propose, so I can check if it could be doable and what kind of electrical signals are needed. Also detailed information about interfaced I listed is appreciated, as could give some detail I'm missing. Thanks Andrea -- TTFN - Guy
Re: idea for a universal disk interface
shad said > Hello, > I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. > I'm often facing common problems with storage devices, magnetic discs and > tapes are a little prone to give headaches after years, and replacement > drives/media in case of a severe failure are unobtainable. > In some cases, the ability to make a dump of the media, also without a > running computer is very important. > > Whence the idea: realize an universal device, with several input/output > interfaces, which could be used both as storage emulator, to run a computer > without real storage, and as controller emulator, to read/write a media > without a running computer. > To reduce costs as much as possible, and to allow the better compatibility, > the main board shall host enough electrical interfaces to support a large > number of disc standard interfaces, ideally by exchanging only a personality > adapter for each specific interface, i.e. connectors and few components.> > There are several orders of problems: > - electrical signals, number and type (most disk employ 5V TTL or 3.3V TTL, > some interfaces use differential mode for some faster signals?) > - logical implementation: several electrical signals are used for a specific > interface. These must be handled with correct timings > - software implementation: the universal device shall be able to switch > between interface modes and be controlled by a remote PC > > I suppose the only way to obtain this is to employ an FPGA for logic > implementation of the interface, and a microprocessor running Linux to handle > software management, data interchange to external (via Ethernet). This means > a Xilinx Zynq module for instance. > I know there are several ready devices based on cheaper microcontrollers, but > I'm sure these can't support fast and tight timing required by hard disk > interfaces (SMD-E runs at 24MHz). > > The main board should include a large enough array of bidirectional > transceivers, possibly with variable voltage, to support as much interfaces > as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, > SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. > The common factor determining what kind of disc interface can be support on > hardware side is obviously the type of transceiver employed, for instance a > SATA would require a differential serial channel, which could not be > available. > But most old electronic is based on TTL/CMOS 5V logic, so a large variety of > computer generations should be doable. > > For the first phase, I would ask you to contribute with a list of interfaces > which could be interesting to emulate, specially if these are similar to one > from my list. > I please submitters to send me by email or by web link when possible, > detailed documentation about the interface they propose, so I can check if it > could be doable and what kind of electrical signals are needed. > Also detailed information about interfaced I listed is appreciated, as could > give some detail I'm missing. The Diablo / Pertec interface was a popular industry standard. Here's a product (no connection) that implements it https://www.arraid.com/data-storage-products/product/aem-5c.html It would be great if there were open source or cheaper devices, maybe there are, I guess the Unibone can do this? (I don't have one yet) (btw I never actually subscribed to cctech but somehow my cctalk ones get echoed over there)
idea for a universal disk interface
Hello, I'm a decent collector of big iron, aka mini computers, mainly DEC and DG. I'm often facing common problems with storage devices, magnetic discs and tapes are a little prone to give headaches after years, and replacement drives/media in case of a severe failure are unobtainable. In some cases, the ability to make a dump of the media, also without a running computer is very important. Whence the idea: realize an universal device, with several input/output interfaces, which could be used both as storage emulator, to run a computer without real storage, and as controller emulator, to read/write a media without a running computer. To reduce costs as much as possible, and to allow the better compatibility, the main board shall host enough electrical interfaces to support a large number of disc standard interfaces, ideally by exchanging only a personality adapter for each specific interface, i.e. connectors and few components. There are several orders of problems: - electrical signals, number and type (most disk employ 5V TTL or 3.3V TTL, some interfaces use differential mode for some faster signals?) - logical implementation: several electrical signals are used for a specific interface. These must be handled with correct timings - software implementation: the universal device shall be able to switch between interface modes and be controlled by a remote PC I suppose the only way to obtain this is to employ an FPGA for logic implementation of the interface, and a microprocessor running Linux to handle software management, data interchange to external (via Ethernet). This means a Xilinx Zynq module for instance. I know there are several ready devices based on cheaper microcontrollers, but I'm sure these can't support fast and tight timing required by hard disk interfaces (SMD-E runs at 24MHz). The main board should include a large enough array of bidirectional transceivers, possibly with variable voltage, to support as much interfaces as possible, namely at least Shugart floppy, ST506 MFM/RLL, ESDI, SMD, IDE, SCSI1, DEC DSSI, DEC RX01/02, DG6030, and so on, to give a starting point. The common factor determining what kind of disc interface can be support on hardware side is obviously the type of transceiver employed, for instance a SATA would require a differential serial channel, which could not be available. But most old electronic is based on TTL/CMOS 5V logic, so a large variety of computer generations should be doable. For the first phase, I would ask you to contribute with a list of interfaces which could be interesting to emulate, specially if these are similar to one from my list. I please submitters to send me by email or by web link when possible, detailed documentation about the interface they propose, so I can check if it could be doable and what kind of electrical signals are needed. Also detailed information about interfaced I listed is appreciated, as could give some detail I'm missing. Thanks Andrea