Re: [Freedos-user] writing a loadable block driver for 4k-sector drive (Questions)
Hi, FreeDOS-devel ! Upon C. Masloch's harsh request\\kind invitation, I'm joining Freedos-devel now and shall continue this question here that started in Freedos-user. Apologies to anybody this change of venue may inconvenience. -- On Mon, 20 Feb 2012 11:15:00 PST BretJ bretjohn@***.com wrote in Freedos-user : Bertho Grandpied wrote: Therefore my first interrogation is, what set of device header attributes - and associated functions, including IOCTL codes - must be present /at a minimum/ for letting DOS access the disk properly ? Bret wrote : FWIW, this is what is implemented in my USBDRIVE: 01h - Media Check 02h - Build BIOS Parameter Block 04h - Read 08h - Write 09h - Write with Verification 0Dh - Device Open 0Eh - Device Close 0Fh - Removable Media 11h - Generic IOCTL DOS 3 13h - Generic IOCTL DOS 4+ 17h - Get Logical Device 18h - Set Logical Device 19h - IOCTL Check DOS 5+ USBDRIVE is installed as a TSR instead of in CONFIG.SYS, so doesn't need or support Function 00h (Initialize). I don't know if all of these are actually needed or not, but they are supported. (list of IOctls scrubbed for brevity, cf. Bret's original msg) Well, OK Bret, but this doesn't tell what the minimum set would be (as you conceded). Are the IOCTLs and functions 17,18,19h required ? Since support for these features has to be marked by corresponding bits #7 6 of the driver attribute, it is conceivable that a driver might not provide it. Unless of course DOS refuses to use (old?) drivers which do not advertise these functions. Someone knows ? Bertho Grandpied wrote: - Using a loadable driver for the block device implies DOS won't use /its/ internal buffers, so I don't have to care about DOS own buffers sizing, right ? Wrong. Of course you're right, I wasn't thinking out well. The reason you're even doing this in the first place is because DOS _will_ use its internal buffers. This wouldn't be necessary otherwise. IOW, you should still check the max sector size in the DOS List of Lists before you install yourself to make sure it's 512 bytes, and should refuse to install yourself if it's already 4096. Uh, here I'm confused, not following any more, please elaborate your point. Assuming we've loaded USBASPI.SYS [which provides ector access to the external disk using SCSI commands], my hypothetical driver wishes to support one or more FAT partitions on that external disk. Whether or not DOS has already increased its buffer size up to 4K before my driver loads from CONFIG, the driver had better be loaded or else DOS won't know to access the partition(s). IOW the role of the driver is not /just/ to inform DOS that it might have to adjust its internal buffer size, as you seem to be writing above; rather the main role of the loaded driveris to augment DOS by allowing it to well, read/write files and clusters and sectors to the disk, a feat DOS by itself would be incapable of (neither USB nor SCSI/ASPI existed when DOS was designed, so I guess we might pardon it). What am I missing of your above point ? The only way around this I know of is to install as an IFS (Installable File System / Network) driver, similar to how MSCDEX (and its clones) work. Bertho Grandpied wrote: - Besides, should I consider using the non IBM format bit in driver attribute ? From whatever docs I saw is unclear what non IBM changes exactly in how DOS uses the driver, None of the devices I've ever seen use the IBM format -- it changes the BPB. Probably a dangerous road to go down. Do you mean to say non-IBM ? UIAM again, the IBM (should be called MS) formats are the usual FAT disks. IOW non-IBM is when attribute bit 13 = 1. See a difference in driver function 2 in Ralf Brown's list : FAT sectors aren't transmitted, does it mean a non IBM disk may be non-FAT ? Again, I'm curious if anybody has experience with this unusual case. WHATEVER... Interestingly Novac's DI1000DD.SYS driver has attribute = 68C2 (or 68C0 for disks under 32 Megs). Yes, it's non IBM whatever that means! Regards -- Czerno -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] writing a loadable block driver for 4k-sector drive (Questions)
Unless of course DOS refuses to use (old?) drivers which do not advertise these functions. Someone knows ? I find that a minimalistic approach almost always leads to problems and compatibility issues. Even if you think a function/feature/API may not be needed, it's still better to put it in. IOW, you should still check the max sector size in the DOS List of Lists before you install yourself to make sure it's 512 bytes, and should refuse to install yourself if it's already 4096. Uh, here I'm confused, not following any more, please elaborate your point. Assuming we've loaded USBASPI.SYS [which provides ector access to the external disk using SCSI commands], my hypothetical driver wishes to support one or more FAT partitions on that external disk. It depends on exactly what your driver does. It sounds like you're intending to essentially make a replacement for DI1000DD.SYS. If so, you're going to need to coordinate your installation with DD1000DD.SYS, in case the user tries to install both at the same time. Whether you abort your installation or not depends on whether DI1000DD.SYS is already installed, whether it supports 4k sectors or not (I have no idea), whether there are any disks with 4k sectors actually plugged in or not, and whether or not the DOS kernel natively supports 4k sectors. You don't want to install if DI1000D.SYS is already installed and can already handle everything. I (and you, apparently) are assuming that USBASPI.SYS supports 4k sectors, but I would actually verify that before I went very far. If, however, you're going to do what Christian (and maybe Eric?) was thinking and essentially install a shim between DI1000DD.SYS and the kernel, things are a little different. Such a shim could work even on non-ASPI disks (such as those provided by Georg's or my USB drivers), but should abort installation if the kernel natively supports 4k sectors. In this scenario, there's also the potential issue of hot-pluggable disks (which are supported by my USB drivers, but I don't think are supported by USBASPI or Georg). Do you mean to say non-IBM ? UIAM again, the IBM (should be called MS) formats are the usual FAT disks. IOW non-IBM is when attribute bit 13 = 1. No, it's the other way around. If bit 13 is set, it means IBM Format, but that doesn't equate to MS. Here's a quote from an old book I have, DOS Programmer's Reference, 2nd Edition, by Que, which gives the practical explanation of what the bit means, specifically regarding function 2, Build BPB: A one sector buffer is passed to this routine. If the non-IBM format bit in the device attribute word is set to zero, the buffer contains the first sector of the FAT and should not be changed. If the bit is set, the buffer can be used as a scratch area in which to build the BPB. It doesn't have anything to do with non-FAT. Bottom line for safety in the driver: ignore the setting of the bit, assume you can't change the buffer, and assume the buffer does not contain the first sector of the FAT. -- Virtualization Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] writing a loadable block driver for 4k-sector drive (Questions)
Bertho Grandpied wrote: Therefore my first interrogation is, what set of device header attributes - and associated functions, including IOCTL codes - must be present /at a minimum/ for letting DOS access the disk properly ? - For a tentative and probably naive self answer, could I get away with the driver attribute being all zeroes - and implement functions 0, 1, 2, 4, 8, 9 (alias for fn 8) /only/ ? Assuming this basic set of functions properly implemented, will the device work ? Do we /need/ 0D,0E (open/close) for instance ? FWIW, this is what is implemented in my USBDRIVE: 01h - Media Check 02h - Build BIOS Parameter Block 04h - Read 08h - Write 09h - Write with Verification 0Dh - Device Open 0Eh - Device Close 0Fh - Removable Media 11h - Generic IOCTL DOS 3 13h - Generic IOCTL DOS 4+ 17h - Get Logical Device 18h - Set Logical Device 19h - IOCTL Check DOS 5+ USBDRIVE is installed as a TSR instead of in CONFIG.SYS, so doesn't need or support Function 00h (Initialize). I don't know if all of these are actually needed or not, but they are supported. Here are the IOCTL functions supported by USBDRIVE: 40h - Set Device Parameters 60h - Get Device Parameters 41h - Write Track 61h - Read Track 42h - Format Track 62h - Verify Track 46h - Set Volume Serial Number 66h - Get Volume Serial Number 47h - Set Access Status 67h - Get Access Status Bertho Grandpied wrote: - Using a loadable driver for the block device implies DOS won't use /its/ internal buffers, so I don't have to care about DOS own buffers sizing, right ? Wrong. The reason you're even doing this in the first place is because DOS _will_ use its internal buffers. This wouldn't be necessary otherwise. IOW, you should still check the max sector size in the DOS List of Lists before you install yourself to make sure it's 512 bytes, and should refuse to install yourself if it's already 4096. The only way around this I know of is to install as an IFS (Installable File System / Network) driver, similar to how MSCDEX (and its clones) work. Bertho Grandpied wrote: - Besides, should I consider using the non IBM format bit in driver attribute ? From whatever docs I saw is unclear what non IBM changes exactly in how DOS uses the driver, nor the (dis)advantages of that approach and the requirements it puts on (removes from) the driver. None of the devices I've ever seen use the IBM format -- it changes the BPB. Probably a dangerous road to go down. -- View this message in context: http://old.nabble.com/writing-a-loadable-block-driver-for-4k-sector-drive-%28Questions%29-tp33355861p33358781.html Sent from the FreeDOS - User mailing list archive at Nabble.com. -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] writing a loadable block driver for 4k-sector drive (Questions)
Dear List... I'm calling back with respect to the 4k-sector USB disk drive. I'm considering writing a loadable DOS 'block' driver for it, as Eric Auer suggested. This doesn't belong on Freedos-user then. I would like to take the simplest approach possible first, even at the expense of performance. My driver would be little more than a 'shell' around USBASPI.SYS. What exactly do you want to accomplish with that? - Using a loadable driver for the block device implies DOS won't use /its/ internal buffers, so I don't have to care about DOS own buffers sizing, right ? This is wrong. Block drivers directly interact with DOS and its buffering scheme. If what you want is to circumvent the restrictions imposed by DOS's buffering scheme (for example, current 512B-/2048B-sector limits of the FDK) you'll have to write your own (FAT) file system driver which interfaces downwards with the block device driver (in your case, USBASPI.SYS) and upwards with DOS's file system redirector interface. (The redirector interface is what file system drivers for non-FAT file systems (eg *CDEX) and networked file systems use to make their files accessible to DOS. Your case, as I understand it, would be a special case to circumvent DOS's FS driver's limits.) regards, C. Masloch -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] writing a loadable block driver for 4k-sector drive (Questions)
Hi Chris, Bertho, Dear List... I'm calling back with respect to the 4k-sector USB disk drive. I'm considering writing a loadable DOS 'block' driver for it, as Eric Auer suggested. This doesn't belong on Freedos-user then. Maybe Czerno / Bertho is not on freedos-devel? Or maybe there is simply too little traffic on freedos-devel, so he preferred to give it a try on freedos-user... I am CCing -devel here, as the kernel discussion is a bit separate: You can either patch that or not, but you can always enjoy a new block device driver ;-) I would like to take the simplest approach possible first You could wait for the big sector size support patch to work in the kernel - but of course I do like the idea of using a block device driver to hide the actual sector size from a DOS kernel, so DOS can I/O in 512 byte units. This is WORSE speed-wise but BETTER kernel-RAM-usage-wise... And of course it is the only chance to use 4k disks on old kernels at all. - Using a loadable driver for the block device implies DOS won't use /its/ internal buffers, so I don't have to care about DOS... If your block device driver shows the DOS kernel a device with 512 byte per sector virtual sectors then the kernel will happily use 512 byte per sector BUFFERS for them... However, your own driver will need one 4096 byte buffer to do any I/O which is not mappable to aligned physical 4 kB sectors. For example imagine the following situation: Real: ... ... (two sectors of 4k each) Virt: ... ,,,X X,,, ... (DOS wants 1k of data) Now you first read all into your buffer, copy the first X to the start of the buffer of the caller, then read all into your buffer, copy the 2nd X to the end of the buffer of the caller. Note how you read 8kB in two parts of 4kB each while your user only wanted 1kB in one part. This is more or less the worst case performance wise, other cases are better. If your driver has a larger buffer, it gets a BIT better... If the user wants to WRITE data, you will first have to READ the whole 4kB sector, then overwrite the part that the user wants to write, then write the whole sector to disk, so you again get a performance loss. If the user wants to read or write MORE data, you will be lucky for the middle area of that data: Imagine a write of 10 kB with a pretty unpleasant mis-alignment: Real: Virt: ..XX XX.. Now you can write sectors b and c as a whole and only have to use your buffer and the read-modify-write style access for the first 1 kB and the last 1 kB. You do not have to read b or c, because they are fully overwritten. So for large I/O transfers, performance loss is smaller. As said, this is separate from what DOS BUFFERS will do and each buffer of the kernel can buffer the usual 512 bytes, which is NICE for your other disks and your RAM. This is wrong. Block drivers directly interact with DOS and its buffering scheme. If what you want is to circumvent the restrictions imposed by DOS's buffering scheme (for example, current 512B-/2048B-sector limits of the FDK) you'll have to write your own (FAT) file system driver which interfaces downwards with the block device driver (in your case, USBASPI.SYS) and upwards with DOS's file system redirector interface. This is also wrong. Because the filesystem still is FAT and the kernel does already understand FAT, your driver can present your 4 kB per sector disk as a BLOCK device or in other words: As a bunch of sectors. However, your driver has to transform what the DOS kernel sees of the boot sector and BPB, because the whole idea of having a 4 kB disk support driver for 512 byte per sector DOSes is that it shows the kernel a bunch of 512 BYTE SECTORS (virtual ones) instead of showing the real 4 kB sectors. This DOES circumvent the limitation of the DOS kernel: The kernel only sees 512 byte per sector blocks. But as explained above, now your driver will have to spend at least 4 kB of buffer space for the transformations. redirector interface is what file system drivers for non-FAT file systems (eg *CDEX) and networked file systems use to make their files accessible That would mean that you duplicate the whole logics of understanding the FAT filesystem into your driver. My suggestion is a TRANSFORMATION of sector SIZES which keeps things simple but SLOW: Everything stays FAT... Of course if you want to support ISO9660, UDF, ExFAT, NTFS, EXT2, EXT3, EXT4 or similar fancy filesystems, there is efficient no way to make them look similar to FAT at all. This is the reason why SHSUCDX, MSCDEX and similar drivers are NOT block device drivers but, as you say, are based on the CD / network redirector API. NOTE: Transformation of sector sizes is easiest in FAT32. Other FAT sizes may take more effort. Also, you can only transform from 4096 byte units to 512 byte units if the filesystem is at most 2 Terabytes and cluster
