Re: [Freedos-user] Re : Support for 4k byte sectors + TDSK
you *could* try USBASPI.SYS /V /W followed by DI1000DD.SYS ( works for me ) .. eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com ..-- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors + TDSK
Mark Brown-27 wrote: you *could* try USBASPI.SYS /V /W followed by DI1000DD.SYS ( works for me ) In my opinion, this is applying a band-aid to a problem that actually requires drastic surgery. USBASPI.SYS + DI1000DD.SYS may in fact work in certain situations. But, it is not a complete, or even a desirable, long-term solution. It is not open source, does not support USB devices other than disks (mice, keyboards, joysticks, printers, network, comm, ...), etc. It may get you by in the meantime while real solutions are developed, though. -- View this message in context: http://old.nabble.com/Re-%3A-Support-for-4k-byte-sectors-%2B-TDSK-tp33277937p33280451.html Sent from the FreeDOS - User mailing list archive at Nabble.com. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors + PLoP
Op 7-2-2012 15:05, Bertho Grandpied schreef: Also of note, PLoP's license has changed it is now free for use including commercial. Donations will be accepted still? Please watch what Elmar had to say about the change (at the above URL). The donation button is quite well hidden, and seems restricted to Paypal only. Vendors like ShareIt(.com) seem to work pretty well if he wants to sell stuff, it even offered the iDeal payment system that's common in the Netherlands (we're not a creditcard culture, sorry). I tend to pay for low-cost quality software if no equal free solution exists. Windows and VMware Workstation have been my most expensive purchases, followed by some collector editions of various games. Bernd -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors + TDSK
On Sat, Feb 4, 2012 at 3:13 PM, Bertho Grandpied y31415926...@yahoo.fr wrote: Hi, Guys! Replying to self, sort of, and Jeremy at the same time. I've been glancing thru the ram disk, TDSK, source. Internal buffer (used for init only) was provisionned for one 4K sector, but for some reason author(s) limited sector size to 2K, as specified on the driver's command line. I boldly hacked the binary so it ignored the limitation and, behold, a first quick and (very) dirty test of a ~8 MBytes FAT12 based RAM disk *with 4096 byte sectors* in MS-DOS 7.1 with a rather fully loaded config SUCCEEDED! I was able to copy several megabyte files to/from the ramdisk (quick tests did include binary compare fo source to dest, and an examination of the RAMdisk with Norton's DISKEDIT revealed no problems). I'm not affirming yet there are no hidden bugs but, clearly, MSDOS CAN support this type of device with no or little problems ! This to me is great news, since it makes it worth developing a simple ASPI to DOS convertor for use with my external USB disk. MSDOS bugs, if any, may be limited to installing large sectored units which are to be managed by IO/MSDOS.SYS internal disk driver. Jeremy wrote : (currently not while testing). ?I have tested 256, 512, 1024, and 2048 byte sectors with tdsk (currently my only way to test). You may try to force TDSK to allow 4096 bps (not more without recompilation) by nullifying the sanity test for command-line size-of-sector, like I did ! Czerno For testing only - warning may corrupt data!!! https://www.fdos.org/kernel/testing/4K/ Included is a test kernel supporting 4KB sectors, note it limits buffers to 2 to avoid memory corruption on boot. Also there is tdsk 3.2 with patch to allow 4KB sectors assembled with jwasm. source can be found on sf fdos svn - hack to limit buffers to 2 not there but available on request. this kernel is there to allow testing with drive that exposes non 512 byte sectors only and will be removed without notice. Thanks for testing, Jeremy -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Improved support for sectors other than 512 bytes has been committed, I am still working on it, default support will still be for 512 bytes (currently not while testing). I have tested 256, 512, 1024, and 2048 byte sectors with tdsk (currently my only way to test). Using max sector size higher than 3KB sizes corrupt memory chain during init (haven't tracked down yet). Constructive comments welcome. Jeremy -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Jeremy, Bret, Georg, others, Improved support for sectors other than 512 bytes has been committed, I am still working on it, default support will still be for 512 bytes (currently not while testing). I have tested 256, 512, 1024, and 2048 byte sectors with tdsk (currently my only way to test). Using max sector size higher than 3KB sizes corrupt memory chain during init (haven't tracked down yet). Constructive comments welcome. Actually I literally constructed something related :-) I made a FAT32 filesystem with 4096 bytes per sector, then looked at 38 bytes in it, ultimately changing 24 (two copies of 12) to make it a filesystem with 512 bytes per sector :-) I copied a small file into both versions and dosfsck is happy with both. Also, both versions still only differ in boot sectors and timestamps... This is hopefully good for two things: 1. as a proof that you could mount 4096 byte per sector filesystems as if they were 512 byte per sector block devices with a small driver which shows a slightly modified representation of the BPB but gives a simple 1:1 access to everything else, for both reading and writing :-) 2. as test filesystem to evaluate the 4096 byte per sector support of operating systems of your choice, including FreeDOS, and their drivers. I used 4 kB per sector based on the original thread, although somebody mentioned that 4 kB / sec harddisks are seen as a temp solution by the industry until GPT partitioning becomes more widespread. GPT *is* nice. While the filesystems are 2 GB in size, my zip containing the bzip2-ed versions of both the 512 and the 4096 byte per sector variants of them, as well as some additional text and binary files to document my tweak, is only 17 kilobytes :-) Please contact me if you want a copy by email from me - I try to avoid sending files directly to the mailing list. I hope some people have interest in my test and proof of concept data! Regards, Eric :-) PS: I could also upload the file, if you know a nice location... dosfsck 3.0.9 (31 Jan 2010) dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN Checking we can access the last sector of the filesystem Boot sector contents: System ID mkdosfs Media byte 0xf8 (hard disk) 4096 bytes per logical sector 32768 bytes per cluster 32 reserved sectors First FAT starts at byte 131072 (sector 32) 2 FATs, 32 bit entries 294912 bytes per FAT (= 72 sectors) Root directory start at cluster 2 (arbitrary size) Data area starts at byte 720896 (sector 176) 66538 data clusters (2180317184 bytes) 32 sectors/track, 64 heads 0 hidden sectors 532480 sectors total Checking for unused clusters. Checking free cluster summary. diskimage-4096.bin: 1 files, 2/66538 clusters dosfsck 3.0.9 (31 Jan 2010) dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN Checking we can access the last sector of the filesystem Boot sector contents: System ID mkdosfs Media byte 0xf8 (hard disk) 512 bytes per logical sector 32768 bytes per cluster 256 reserved sectors First FAT starts at byte 131072 (sector 256) 2 FATs, 32 bit entries 294912 bytes per FAT (= 576 sectors) Root directory start at cluster 2 (arbitrary size) Data area starts at byte 720896 (sector 1408) 66538 data clusters (2180317184 bytes) 256 sectors/track, 64 heads 0 hidden sectors 4259840 sectors total Checking for unused clusters. Checking free cluster summary. diskimage-512.bin: 1 files, 2/66538 clusters PS: Note that going from 32x64x??? 4096 byte sectors in one track to 512 byte sectors gives 256 sectors/track which is no realistic CHS geometry. Please use LBA here. Loadable DOS block devices have linear geom. anyway :-) Archive: fat32-sector-size-demo.zip Zip file size: 16532 bytes, number of entries: 8 drwxr-x--- 3.0 unx0 bx stor 12-Feb-04 13:00 fat32-sector-size-demo/ -rw-r- 3.0 unx 3695 tx defN 12-Jan-17 01:46 fat32-sector-size-demo/diskimage-4k-empty.txt -rw-r- 3.0 unx 2301 bx defN 12-Feb-04 12:39 fat32-sector-size-demo/diskimage-512-header.bin.gz -rw-r- 3.0 unx 3146 tx defN 12-Feb-04 12:43 fat32-sector-size-demo/diskimage-512-edit.txt -rw-r- 3.0 unx 117918 bx defN 12-Feb-04 12:41 fat32-sector-size-demo/diskimage-512.bin.bz2 -rw-r- 3.0 unx 2307 bx defN 12-Feb-04 12:59 fat32-sector-size-demo/diskimage-4096-header.bin.gz -rw-r- 3.0 unx 117926 bx defN 12-Feb-04 12:50 fat32-sector-size-demo/diskimage-4096.bin.bz2 -rw-r- 3.0 unx19905 tx defN 12-Feb-04 12:57 fat32-sector-size-demo/diskimage-4k-with-file.txt 8 files, 267198 bytes uncompressed, 14782 bytes compressed: 94.5% -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps,
Re: [Freedos-user] Re : Support for 4k byte sectors
Links please? On 1/25/12 1:46 PM, Bertho Grandpied wrote: Just a note, Folks, /who/ said advanced format disks (presenting 512 byte sectors) are with us for ten years - or more, so we should be little concerned about having to support true 4K sector disks ? But I stumbled upon a couple pages that say otherwise : the industry has agreed to sell AF disks only *until the end of 2014*! This if true is way shorter than 10 years, and would IMO justify real work done on updating the kernel. I've not kept the links, ooops! but Google is our friend (is it?) By procrastinating one would be doing the same kind of costly mistake than, say, for IPv6 support (lack of it). Regards -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
I had a look at Bret's open source USB drivers, unfortunately they only support Intel/Via (UHCI) controllers yet. True. Working on that. I also think they have hard coded 512 bytes per sector. No. USBDRIVE reads the maximum buffer size from the DOS List of Lists (as discussed some earlier in the thread), and uses that as its maximum sector size. If a USB disk has 4k sectors, but DOS can only handle 512 byte sectors, USBDRIVE won't load the disk. If DOS can handle (buffer) 4k sectors OK, USBDRIVE will load it and automatically mount any FAT partitions it finds. Also as stated earlier, I don't personally have any disks with sector sizes other than 512, so this has never been tested (at least by me) on a real system. That's why you need to modify the kernel to handle 4k sectors, also as discussed earlier (at least with my drivers). Based on what Eric says, though, that doesn't work with the FreeDOS kernel. Because USBDRIVE provides an INT 13h interface, you can also use external drivers to mount/access the non-FAT partitions that may be on the USB disks (NTFS, EXTx, exFAT, ...). USBDRIVE won't mount non-FAT drives automatically, though. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
The advantage of a write-delay cache is that that the writing can be done when the system is idle (a simple form of multi-tasking). That counts as advanced cache with a lot of code and can go as far as a sort of ramdisk which syncs back to the harddisk slowly but steadily when the harddisk has time, in big cache. And it is not what I would suggest for DOS... That's basically what SMARTDRV and its equivalents do, though in a limited sense (they don't use oodles of memory and cache/RAMDisk the entire hard drive). Are you saying that you don't think programs like SMARTDRV belong in DOS? -- View this message in context: http://old.nabble.com/Re-%3A-Support-for-4k-byte-sectors-tp33144850p33162629.html Sent from the FreeDOS - User mailing list archive at Nabble.com. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
is this too simplistic or what (?): you could reformat ntfs and use a freeware reader, or reformat the whole hard drive and then use that... or i've had excellent luck with USBASPI.SYS 2.27 + DI1000DD.SYS (links below). http://panasonic.jp/com/support/drive/other/f2h_usb.html http://www.hiren.info/download/dos-files/di1000dd.sys eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com eufdp...@yahoo.com From: Bertho Grandpied y31415926...@yahoo.fr To: freedos-user@lists.sourceforge.net Sent: Wednesday, January 18, 2012 1:46 PM Subject: [Freedos-user] Re : Support for 4k byte sectors In reply to: Bret Johnson bretjohn@ju... only support Intel/Via (UHCI) controllers yet. True. Working on that. Great :=) I also think they have hard coded 512 bytes per sector. No. USBDRIVE reads the maximum buffer size from the DOS List of Lists (as discussed some earlier in the thread), and uses that as its maximum sector size. If a USB disk has 4k sectors, but DOS can only handle 512 byte sectors, USBDRIVE won't load the disk. If DOS can handle (buffer) 4k sectors OK, USBDRIVE will load it and automatically mount any FAT partitions it finds. TY for the heads-up. Also as stated earlier, I don't personally have any disks with sector sizes other than 512, so this has never been tested (at least by me) on a real system. It would be a good thing if someone on the list having access to such a device /and/ Intel-based USB would experiment and report their findings... That's why you need to modify the kernel to handle 4k sectors, also as discussed earlier (at least with my drivers). Based on what Eric says, though, that doesn't work with the FreeDOS kernel. As I noted earlier, I'm sure the default disk driver MS DOS kernels can handle bigger sectors, /but/ there are problems to be fixed - as pointed to me by R. Loew, and I verified it too : MSDOS init module (patition scanner) discards partitions whose boot sectors indicate any sector size other than 512. Disks operated through user drivers (config.sys) should not be so limited. While trying a free fix for MSDOS could be attempted - but is the effort worth it ? - I would vote for FreeDOS to be enhanced. Does someone here know if DR/Open DOS recognises 512k sectors, either or both with its built-in disk driver and user drivers ? Because USBDRIVE provides an INT 13h interface, you can also use external drivers to mount/access the non-FAT partitions that may be on the USB disks (NTFS, EXTx, exFAT, ...). USBDRIVE won't mount non-FAT drives automatically, though. Understood. Regards -- Czerno -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 18-1-2012 20:15, Mark Brown schreef: is this too simplistic or what (?): you could reformat ntfs and use a freeware reader, or reformat the whole hard drive and then use that... or i've had excellent luck with USBASPI.SYS 2.27 + DI1000DD.SYS (links below). How would your suggestion be a solution to the problem at hand? What is desired is a solution to both read and write sectors of 4K size each, due to some silly hardware USB bridge using it. Formatting something as NTFS doesn't guarantee a proper working on such a USB bridge. Besides, using a reader implies no writing. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 18-1-2012 17:11, Bret Johnson schreef: I had a look at Bret's open source USB drivers, unfortunately they only support Intel/Via (UHCI) controllers yet. True. Working on that. That's promising, even to see if your drivers can really make FDISK work. I probably should prepare a bootdisk image for you that hides all physical/int13 drives, then loads your USB driver, then tries to execute FDISK. My only way of trying UHCI driver is by running inside VMware, and then your driver doesn't seem to succeed due to some IRQ issue. Might be a VM bug for that matter. QEMU also has USB support but can only connect devices if you run QEMU under Linux, which means I still can't test much as I run QEMU under Windows. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
On Wed, Jan 18, 2012 at 2:23 PM, Bernd Blaauw bbla...@home.nl wrote: Formatting something as NTFS doesn't guarantee a proper working on such a USB bridge. Besides, using a reader implies no writing. I use NTFS under Windows. Mark Russinovitch offered a freeware NTFS *reader* for DOS through his old Sysinternals site, and a payware driver that could also *write* to NTFS from DOS through the sister Winternals site. (It was intended for rescue operations on NTFS filesystems from DOS.) I'm not sure how thrilled I'd be at trying to use NTFS as the native file system on a DOS machine (aside from the philosophical questions about whether it's still a DOS system if you do...). I wouldn't mind a driver that would let me read ext2/3/4 file systems under Linux from DOS, since FreeDOS is installed on a partition on a box that has Linux, too, but I'd hardly expect FreeDOS to run on top of a Linux FS. __ Dennis -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
That's promising, even to see if your drivers can really make FDISK work. Trust me, it works. I've partitioned many USB disks from DOS, though I normally use Ranish Partition Manager instead of FDISK (it's MUCH easier to use, and will also format the partitions). I know MS FDISK will crash in some cases with USBDRIVE, but that's actually a bug in MS FDISK that doesn't understand removable hard drives. For example, if you have USBDRIVE configured for 3 INT 13h Disk Numbers (controlled with the /Disks:# option), but only have 1 USB disk actually plugged in, MS FDISK will crash. If you have USBDRIVE configured for the same number of disks that are actually plugged in, MS FDISK works OK. I don't think FreeDOS FDISK has this problem, and I know Ranish works OK. ... My only way of trying UHCI driver is by running inside VMware, and then your driver doesn't seem to succeed due to some IRQ issue. Might be a VM bug for that matter. Probably. USB under VM's is generally flaky. Also, USB support under VM's (at least correctly designed VM's) shouldn't really be necessary, anyway. The important USB devices (mice, keyboards, disks, network cards) should be emulated by the VM at some other level already (mice and keyboards should be emulated as PS2 devices, disks should appear as some sort of network drive or somehow mounted directly, network devices should be emulated as some sort of generic network card like an NE2000). At least in theory, USB joysticks should be emulated as standard game port joysticks as well, though I don't know if any VM's actually emulate joysticks. From a practical perspective, DOS USB support is more applicable to real hardware than virtualized. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
I use NTFS under Windows. Mark Russinovitch offered a freeware NTFS *reader* for DOS through his old Sysinternals site, and a payware driver that could also *write* to NTFS from DOS through the sister Winternals site. (It was intended for rescue operations on NTFS filesystems from DOS.) There are a few of different NTFS utilities for DOS, including the Winternals ones. I've only had limited success with the ones I've tried, and wouldn't recommend any of them very highly. It certainly doesn't help that NTFS is 100% proprietary, or that there are also several different revisions of NTFS. I'm not sure how thrilled I'd be at trying to use NTFS as the native file system on a DOS machine (aside from the philosophical questions about whether it's still a DOS system if you do...). I'm not sure anybody's even considering booting DOS from anything other than FAT. But, it's really nice to be able to access from DOS the non-boot partitions on a hard drive, or external drives like USB, that are formatted with NTFS or EXTx or exFAT or whatever. I wouldn't mind a driver that would let me read ext2/3/4 file systems under Linux from DOS, since FreeDOS is installed on a partition on a box that has Linux, too, but I'd hardly expect FreeDOS to run on top of a Linux FS. Is anybody even working on a EXTx driver for DOS? I know I've heard of a few people experimenting with exFAT, but haven't heard of anything actually being released. -- View this message in context: http://old.nabble.com/Re-%3A-Support-for-4k-byte-sectors-tp33163231p33164647.html Sent from the FreeDOS - User mailing list archive at Nabble.com. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi, On Wed, Jan 18, 2012 at 4:31 PM, BretJ bretj...@juno.com wrote: I use NTFS under Windows. Mark Russinovitch offered a freeware NTFS *reader* for DOS through his old Sysinternals site, and a payware driver that could also *write* to NTFS from DOS through the sister Winternals site. (It was intended for rescue operations on NTFS filesystems from DOS.) There are a few of different NTFS utilities for DOS, including the Winternals ones. I've only had limited success with the ones I've tried, and wouldn't recommend any of them very highly. I only tried one or two very very briefly years ago. IIRC, one of them was ridiculously slow, and the other was very buggy. It's probably a stretch to expect NTFS to work well in FreeDOS. Most people would say, Just use Linux. (Dumb but admittedly decent workaround.) However, NTFS is fairly ubiquitous, similarly to ext2. It certainly doesn't help that NTFS is 100% proprietary, or that there are also several different revisions of NTFS. In fairness, I think a lot of security features would be compromised if they totally opened it up. Not saying I agree, of course, but NTFS is admittedly meant to do a lot more than FAT. This also may be why they have shifted away from booting from FAT. (rant excised) I wouldn't mind a driver that would let me read ext2/3/4 file systems under Linux from DOS, since FreeDOS is installed on a partition on a box that has Linux, too, but I'd hardly expect FreeDOS to run on top of a Linux FS. Is anybody even working on a EXTx driver for DOS? I know I've heard of a few people experimenting with exFAT, but haven't heard of anything actually being released. exFAT is patented, so is VFAT (sadly). MS knows this and milks it to death. I wouldn't recommend such tech for that reason alone. However, ext2 from Linux would be interesting for us, but no, I don't think anybody is working on it. We'd probably honestly have to hire someone a la Kickstarter. Benjamin David Lunt's hobby OS (FYSOS) seems to have a billion file systems, so quite honestly, he would be a good candidate (heavy dreaming here). Or perhaps the TestDisk dude, Christophe Grenier. http://www.fysnet.net/fysos.htm http://www.cgsecurity.org/wiki/TestDisk How would you even do it? It's been (briefly) discussed before. 32-bit TSR? JLM? Network redirector interface (similar to ISO9660)?? (Dunno!) Typical workaround (besides Just use Linux, ugh) is LTOOLS, but that didn't work for me at all last I tried, and that's not exactly a native, bootable solution anyways. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
On Wed, Jan 18, 2012 at 5:31 PM, BretJ bretj...@juno.com wrote: I use NTFS under Windows. Mark Russinovitch offered a freeware NTFS *reader* for DOS through his old Sysinternals site, and a payware driver that could also *write* to NTFS from DOS through the sister Winternals site. (It was intended for rescue operations on NTFS filesystems from DOS.) There are a few of different NTFS utilities for DOS, including the Winternals ones. I've only had limited success with the ones I've tried, and wouldn't recommend any of them very highly. It certainly doesn't help that NTFS is 100% proprietary, or that there are also several different revisions of NTFS. Yeah, Linux support has been laboriously reverse engineered. And the revisions are problematic. Vista/Win7, for example, support Honest-to-God symlinks as well as the hard links supported under 2K/XP. I could actually use those in Windows, but symlinks alone aren't sufficient to justify an upgrade. I'm not sure how thrilled I'd be at trying to use NTFS as the native file system on a DOS machine (aside from the philosophical questions about whether it's still a DOS system if you do...). I'm not sure anybody's even considering booting DOS from anything other than FAT. But, it's really nice to be able to access from DOS the non-boot partitions on a hard drive, or external drives like USB, that are formatted with NTFS or EXTx or exFAT or whatever. It would indeed, but I'm not exactly holding my breath waiting. I have Win2K on the box FreeDOS is on, and found an open source extFS driver that reads/writes the ext4 filesystems I use on the Linux instances, and 2K has native FAT32 support, so I'm covered there. I wouldn't mind a driver that would let me read ext2/3/4 file systems under Linux from DOS, since FreeDOS is installed on a partition on a box that has Linux, too, but I'd hardly expect FreeDOS to run on top of a Linux FS. Is anybody even working on a EXTx driver for DOS? I know I've heard of a few people experimenting with exFAT, but haven't heard of anything actually being released. I don't know of anyone doing so, but I haven't actually looked. If I'm in FreeDOS, my need to actually *read* an FS other than FAT is small, and I can live without it. If I need to get to the NTFS or Ext4 slices, I'll boot into an OS that can. __ Dennis -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bret, The advantage of a write-delay cache is that that the writing can be done when the system is idle (a simple form of multi-tasking). That counts as advanced cache with a lot of code and can go as far as a sort of ramdisk which syncs back to the harddisk slowly but steadily when the harddisk has time, in big cache. And it is not what I would suggest for DOS... That's basically what SMARTDRV and its equivalents do, though in a limited sense (they don't use oodles of memory and cache/RAMDisk the entire hard drive). Are you saying that you don't think programs like SMARTDRV belong in DOS? I am saying that for gaining speed on modern disks, in particular flash disk ands large sector disks, you should already make a big difference with a small pooling cache and a short delay, which is both simpler and less risky than large, long delay caches where a lot of hooks try to find out when a good moment to write is and a last safe moment to write, e.g. before reboot, before crash? or before you return to the prompt, as many users assume it is safe to switch off a DOS PC while it is idle at the prompt. My suggestion would be similar to the DR DOS / NWCACHE write pool function, with a maximum write delay of for example 1/4 to 1/2 of a second. The maximum allowed in NWCACHE is 5000 msec, granularity are timer ticks of 55 msec and a document about NWCACHE says that delayed writes are off by default because they have compatibility risks :-p The same NWCACHE document also explains an effect called deferred writes with write combining: If you write the same sector several times in a short time, or quickly write consecutive sectors, the writes result in only one, in the consecutive case larger, write instead of multiple small ones or multiple same-sector writes :-) This document also implies that write-combining of consecutive or same-sector writes happens inside the same buffer which is also used for read-ahead, called the lookahead buffer which can be max 16 kB and defaults to 9 kB. Probably for reading a whole diskette track whenever you read any sector from that track. The size 9 kB does not ring a bell for me in harddisk terms. This thing differs from another option where you can use the WHOLE cache memory of a few megabytes for writes that are delayed up to at most 5 seconds. So... If anybody has NWCACHE (ask me if needed) they could try if NWCACHE with 555 msec delay and 16 kb lookahead / write-combining buffer and delayed/deferred/combined/pooled/whatever writes on is already making a quite nice difference in write speed, eg. on USB and when working with many small files :-) Regards, Eric -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
I am saying that for gaining speed on modern disks, in particular flash disk ands large sector disks, you should already make a big difference with a small pooling cache and a short delay, That's true -- but I don't think either LBACACHE or UIDE actually do that. I could be wrong, but I think they are very simple write-through caches that don't delay or try to pool/combine any of the writes at all. If they did, and also added support for non-INT 13h disks, it could indeed make a HUGE difference. which is both simpler and less risky than large, long delay caches where a lot of hooks try to find out when a good moment to write is and a last safe moment to write, e.g. before reboot, before crash? or before you return to the prompt, as many users assume it is safe to switch off a DOS PC while it is idle at the prompt. Less risky, at least if the system is unstable. Actually only a little simpler, though, since any kind of delay at all, no matter how small, requires monitoring reboots and prompts. In all cases, it should be OK to shut off the system at a prompt. All caches, including SMARTDRV, must commit their write-delay caches before the DOS prompt returns. I personally have an OFF.BAT file to shut down my computers instead of using the power switch, anyway, and it makes sure everything is flushed and committed. Handling crashes is another story -- no way to handle those in any OS unless you use a journaling file system. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
... No cache will ever compete with a RAM disk. RAM is always faster than disks with their seek/rotational latencies and their much slower transfer rates. I knew this would provoke a comment from you, Jack. The purpose of a cache is to put as much data in RAM as it can, so that the disk is accessed as little as possible. It's true that the cached data does eventually get written to disk, and that part is slow. From a speed perspective, though, a well-designed cache can be competitive with a RAM disk. The advantage of a write-delay cache is that that the writing can be done when the system is idle (a simple form of multi-tasking). The end result is that even though disk writes actually take the same amount of time they always did, the SYSTEM actually runs much faster. In my experience and opinion (and it is only an opinion), write-through caches don't provide enough speed benefit to be very helpful, and I don't use them. ... There is a REASON why Write Back caches are all so large -- they demand MANY hooks of a non-generic type into a DOS system, to handle Ctrl/Alt/DEL and other events that require a flush of sectors not-yet written to disk. Yes, write-delay caches are MUCH more complicated than write-through caches. But, they also provide MUCH more practical benefit, IMO. Even with that being said, I don't use SMARTDRV all the time. I only use it in certain situations when it provides noticeable benefit, and in those particular situations a write-through cache doesn't help. Also, FWIW, you can disable write-delay caching in SMARTDRV if you want, in which case it works sort of like UIDE or LBACACHE (except that it will also _natively_ work with non-INT 13h disks like USB and SCSI), but at the expense of requiring more memory. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
I knew this would provoke a comment from you, Jack. Yes, you always were a provoker, weren't you, Bret? The purpose of a cache is to put as much data in RAM as it can, so that the disk is accessed as little as possible. It's true that the cached data does eventually get written to disk, and that part is slow. From a speed perspective, though, a well-designed cache can be competitive with a RAM disk. Exactly why I designed UIDE to have all the features I noted before, along with using only 944 bytes of memory (and some HMA which nothing but MS-DOS HIMEM ever used before) and using XMS for its cache tables and cache data. Try to find any Write Back caches that do so much, for so little memory! The advantage of a write-delay cache is that that the writing can be done when the system is idle (a simple form of multi-tasking). The end result is that even though disk writes actually take the same amount of time they always did, the SYSTEM actually runs much faster. In my experience and opinion (and it is only an opinion), write-through caches don't provide enough speed benefit to be very helpful, and I don't use them. Well, we remain on different sides of a fence! I say Write Through caches provide a LOT of benefit, especially UIDE which can cache up to 4-GB of data! Assuming only 2.5 or 3-GB is assigned to UIDE, one can have 500-MB+ for a nice RAM disk like my RDISK offers, and so one gets The best of both worlds: A (big!) RAM disk for fast files, plus a (big!) cache for ordinary disk files. UIDE/RDISK handle GIGABYTES, not KB or MB like too many other never-upgraded RAM disk and Write- Back cache programs! You can KEEP all those old guys, and I shall continue to do the BEST possible in UIDE/RDISK, for a LOT less memory! There is a REASON why Write Back caches are all so large -- they demand MANY hooks of a non-generic type into a DOS system ... Yes, write-delay caches are MUCH more complicated than write-through caches. But, they also provide MUCH more practical benefit, IMO ... How I just DETEST Internet abbreviations, which are always such LAZY and/or BAD English! But FYB, IMO it is rather IMPRACTICAL to use so much system memory in SMARTDRV/NCACHE2/etc. DOS is in fact memory limited, and if I can have 90% the benefit of most Write Back caches for 90% LESS memory by using UIDE, THAT seems a little more PRACTICAL! Even with that being said, I don't use SMARTDRV all the time. I only use it in certain situations when it provides noticeable benefit, and in those particular situations a write-through cache doesn't help. Why not just use UIDE all the time? You would get UltraDMA I-O rather than whatever your BIOS currently does, and I bet your NET system speed would be greater, from having SOME type of cache active at all times! Also, FWIW, you can disable write-delay caching in SMARTDRV if you want, in which case it works sort of like UIDE or LBACACHE (except that it will also _natively_ work with non-INT 13h disks like USB and SCSI), but at the expense of requiring more memory. SCSI disks are rarely seen on PCs, due to their high disk and controller cost. USB disks are also rarely seen, since they are not-yet reliable, nor in many cases are they fast-enough to replace hard disks. SATA/IDE own the hard-disk market and probably will for a LONG time. So, I am not-at-all bothered by UIDE handling only Int 13h disks, especially as it still CAN be called by other drivers to cache THEIR disks, as well! -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 17-1-2012 18:48, Jack schreef: Also, FWIW, you can disable write-delay caching in SMARTDRV if you want, in which case it works sort of like UIDE or LBACACHE (except that it will also _natively_ work with non-INT 13h disks like USB and SCSI), but at the expense of requiring more memory. SCSI disks are rarely seen on PCs, due to their high disk and controller cost. USB disks are also rarely seen, since they are not-yet reliable, nor in many cases are they fast-enough to replace hard disks. SATA/IDE own the hard-disk market and probably will for a LONG time. So, I am not-at-all bothered by UIDE handling only Int 13h disks, especially as it still CAN be called by other drivers to cache THEIR disks, as well! I wonder if the caching can be implemented in the El-Torito DOS driver. This driver is used for anything looking like a bootable CD, be it a physical bootable CD on IDE/SATA/SCSI/USB-connected controller, or a MEMDISK-loaded (thus in13 supporting) ISO file. I'll do some benchmarking anyway. Copying from a virtual CD located in system memory, to a SHSURDRV-generated ramdisk. I'll try UIDE, CDRCACHE and LBACACHE. Eric suspected all of them won't have that much use. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 17-1-2012 18:48, Jack schreef: SCSI disks are rarely seen on PCs, due to their high disk and controller cost. USB disks are also rarely seen, since they are not-yet reliable, nor in many cases are they fast-enough to replace hard disks. SATA/IDE own the hard-disk market and probably will for a LONG time. So, I am not-at-all bothered by UIDE handling only Int 13h disks, especially as it still CAN be called by other drivers to cache THEIR disks, as well! Helps if I actually paste the link to the code for that eltorito driver [ http://git.kernel.org/?p=boot/syslinux/syslinux.git;a=blob;f=dosutil/eltorito.asm;h=d6b6b50e228474df6e21af2297b7cad637359d0f;hb=refs/heads/master ]. My apologies for the extra mail. Bernd -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bertho, a DRIVER could interface with any disk with any sector size and then just provide an int13 or int25/26 interface with 512 byte sector size for data transfer to DOS. I'm not opposed to this method, which I see as a workaround rather than a fully satisfying answer however. On one hand, it is in the way of formatting and is less fast. But on the plus side, it saves RAM for the DOS kernel... :-) Also, the driver can parse GPT partition tables so it helps kernels which only understand classic MBR partition tables. Of course to have 2 TB in ONE drive letter, you need the support for big sectors and/or (?) GPT in the kernel itself. You insist on FAT32 compatibility , but what about FAT16 FAT32 is more flexible in some aspects, so it is easier to tell DOS that a FAT32 partition on a 4k-disk is a slightly odd FAT32 partition on a (virtually) 512-disk than to do the same for FAT16 or FAT12. Also, what would you do with such a tiny partition on such a large harddisk? ;-) physical Hitachi disk has 512 K sectors, the SATAUSB bridge already does its own 512/4096 conversion (including internal buffering and, I'm not sure but possibly, delaying write back)... your proposed driver would in effect dutifully cancel the packing/unpacking done by the appliance's firmware ! Yeah talk about strange design decisions... The USB box makes you believe that a 512 byte disk is 4k based and my suggested DOS driver makes DOS believe that a 4k one consists of 512 byte units. But unless you change the USB firmware, there is no way to transfer only those 512 byte sectors to your 512 byte disk in the USB box that you actually wanted to update, I guess... This means you cannot make the RAW DISK visible to DOS that way, but you ONLY have to show DOS a modified boot sector to make the rest of an otherwise unchanged PARTITION work from native 4k sectors into show DOS 512 byte fake sec size. Kind of crippling a device if you ask me. You probably lose some speed, but this is a general DOS problem already: Modern disks are below their peak speed if you access single sectors (even native 4k sectors). DOS would have to do more pooling and maybe read-ahead. the real answer is for the DOS kernel to be able to support native 4k sectors To have a real decent performance, especially (!) on a large disk, you must not use FAT. Or if you really do have to use FAT, at least cache the FAT in RAM as Win9x did. For DOS, that means XMS or similar, a few MB of it. Or of course implement a decent filesystem like ext2, as others like NTFS or ext4 will take even more RAM to run. 4k buffers aren't /that/ expensive You should have a few per drive letter and buffers should be in your HMA, of which 40k of 64k are already filled by the kernel code. So you want a few times N sectors in 24k and you suggest to make each buffer 4k instead of 0.5k... Regards, Eric -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Jack, Try to find any Write Back caches that do so much, for so little memory! Sure, it takes more memory. If it is not just local pooling within a few kB and with tiny timeout, it will take even more memory, for logics and extra security logics for writeback. But larger writes really help, in particular with flash / SSD. The advantage of a write-delay cache is that that the writing can be done when the system is idle (a simple form of multi-tasking). That counts as advanced cache with a lot of code and can go as far as a sort of ramdisk which syncs back to the harddisk slowly but steadily when the harddisk has time, in big cache. And it is not what I would suggest for DOS... Why not just use UIDE all the time? Or combine with SMARTDRV / NWCACHE for the write pooling... in which case it works sort of like UIDE or LBACACHE (except that it will also _natively_ work with non-INT 13h disks like USB and SCSI) Actually ancient SMARTDRV (dot sys) versions were int13 based. SCSI disks are rarely seen on PCs SATA or USB could also offer SCSI interfaces next to int 13... But I think for the moment, USB is the most useful non-int13 thing to cache. Because USB storage can be a lot of things: A floppy drive, CD / DVD / BD burner, harddisk, flash stick... Eric -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Re: 4K sector sizes, I realized today that UIDE, UIDE2, and UIDEJR likely will NOT be affected at all -- 1) DOS has a 64K-byte limit for read/write requests, in fact 127 sectors of 512 bytes (the UIDE drivers do accept 128). Since 4K-byte sectors fit into this limit, no physical-level driver changes are needed. 2) All 3 UIDE drivers do ONLY physical block I-O and know nothing about directories, file systems, etc. The drivers remain DOS independent, i.e. they just read/write sectors at the orders of the DOS system and user programs. MUCH simpler and a LOT smaller! 3) UIDE/UIDE2 require, and UIDEJR can set, a 64K buffer in XMS memory for misaligned or other I-O unsuited to UltraDMA. Since DOS cannot do more than 64K I-O at a time, no change to the UIDE drivers' UltraDMA buffering is needed. 4) UIDE/UIDE2 set cache blocks of varying sizes, 8K for a 5-MB cache, up thru 64K for an 80-MB+ cache. So, the drivers have enough cache blocks in all cases to be effective, in handling both directory sectors and data files. 4K disk sectors fit evenly, into any UIDE/UIDE2 cache-block size same as 512-byte sectors do. So UIDE/UIDE2 will need NO changes for caching the larger disk sectors! About all that MAY be needed in the 3 UIDE drivers is some init logic to select using 4K-byte sectors, if a hard-disk demands this (FOOLISH if so, in my opinion!). Assuming boot or FDISK/FORMAT problems are dealt with as required, my comment about 4K-byte sectors is Bring 'em on! The 3 UIDE drivers will run fine with them and so regular DOS I-O should not be any problem! [In the U.S.A., we have an old engineering joke known as the K.I.S.S. Principle, whose letters denote Keep it SIMPLE, stupid!. Less and less of a JOKE, as time goes by and computers become unjustifiably COMPLEX! I and the UIDE drivers still OBEY the KISS Principle, insofar as is possible!]. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bertho, trying to reiterate / re-explain my plan / idea: By the way - a DRIVER could interface with any disk with any sector size and then just provide an int13 or int25/26 interface with 512 byte sector size for data transfer to DOS. As explained in a longer mail this week, it actually SHOULD work: Only a few values in the boot sector would differ from a native 4k sector FAT filesystem compared to a filesystem where things work in groups of 8 sectors of 512 byte each, which is exactly what you would get when you make a 4k disk (containing a native 4k based FAT partition) visible as if it would have 512 byte sized sectors instead. This means you cannot make the RAW DISK visible to DOS that way, but you ONLY have to show DOS a modified boot sector to make the rest of an otherwise unchanged PARTITION work from native 4k sectors into show DOS 512 byte fake sec size. Because DOS only checks partition tables at boot, it never looks at a whole RAW DISK through block device drivers, not counting those built into the kernel for your boot drives. This means that it already is part of the normal job of any loadable block device driver to process partition tables and show sectors and boot sector properties to DOS. Software has routinely modified boot sector properties on the fly to show DOS tuned values for many years, e.g. for ramdisks etc. Note, however, that the OTHER way round is NOT compatible with other operating systems: You cannot FORMAT native 4k sector disks in a way that uses areas of 512 byte size as independent sectors and then show that disk to another OS as 4k sector disk. The disk must be already formatted with groups of 4k size in mind. Again, you CAN safely use 4k disks with 4k FAT partitions in a DOS with 512 byte sector size, as long as your block device driver converts the boot sector BPB properties on the fly into using 512 byte units. And as explained in my earlier mail, this ALWAYS works on FAT32 partitions with at most 2 TB size and at most 64 kB cluster size as limit. A block device driver for accessing native 4k disks with a fixed 512 byte per sector DOS kernel can easily show only those partitions which are not more than 2 TB, 64k-cluster. I mean my 15 Jan 2012 23:33 CET mail :-) This scheme won't work if the disk has to be usable aslo in other OSes like Windows XP, Linux, etc. that recognise the 4k sectors natively ! See above - disks partitioned and formatted in 4k compatible way or simply using the native 4k sector size can be used in all OSes in a compatible way, even in a 512 byte sector size with the help of a converted view on the boot sector block device driver in DOS which shows DOS each 4k sector as a row of 8 sectors of 512. They also STAY compatible that way, so DOS does not break anything by doing access in 512 byte units on the block device driver size. It only is a bit SLOWER, as the block device driver will still talk in 4k sectors when talking to that native 4k sector size disk. Someone, maybe not Eric, asked what I have been using for accessing USB mass storage in DOS. Answer : a version of Panasonic's USBASPI.SYS. This allows access to 4k sectors using SCSI commands. Interesting, but would that be easier than using Bret's or Georg's USB storage drivers? As long as somebody explains me how to write and read 4k sectors with those drivers, I should be able to show how to show 512 sectors and a transformed BPB on the DOS side, in that way making FAT32 partitions on that disk useable by unpatched FreeDOS with a simple loadable block device driver in a safe* way. *I cannot stop you from breaking the wonder by reformatting that partition in a non-4k-compatible way, apart from write-protecting the boot sector. You cannot repartition disks through block device drivers anyway, so THAT part is safe. Also, DOSFSCK will be happy (will not notice anything strange) and CHKDSK skips FAT32 anyway. Regards, Eric -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bret, yes you would see a problematic mismatch if you were to talk raw SCSI or CHS to a disk while being inconsistent about whether you use 512 byte or rather 4096 bytes per sector... However, when DOS mounts a partition with help of a loadable block device driver, nothing would access the raw view of the disk. In any case the kernel will not, it will only talk to the block device driver. Anything beyond that is of course evil, but so is dosfsck-ing a partition in Linux while it is mounted at the same time. Can you recommend any free int 13 or block device based delayed / pooled write caching software? As far as I can remember, all modern (LBA compatible, given disk sizes on current PCs) implementations of this are commercial. ... This is why I suggest to first grow such a file to the final size, putting all FAT writes into ONE access and then writing the file's contents in large N kB chunks. Excellent idea, especially with USB. Thanks, but can you benchmark it? ;-) So you actually do the virtual trick the other way around? ... if a buffer is 2048 and the sectors are only 512, the last 1536 bytes of the buffer are simply never used. Ah, I had understood that you were showing a group of 4 native 512 byte sectors as if they were a 2k sector. Yes. But for reasons already discussed, this should not be done by virtualizing the sector size -- that is asking for problems, IMHO. Only if you can look beyond the virtualization in a bad way... For example the same issue existed with ontrack ezdrive drivers installed in the master boot record: When you booted a native-LBA-enabled operating system, the disk contents would appear shifted compared to the virtual LBA as seen while the driver is loaded... Only those versions of such drivers which tried to hide the space taken up by the driver suffered from the problem. In the suggested virtual 512 byte sector scenario, no shifting happens, BUT DOS is allowed to write in units of 512 bytes. Therefore, the virtual 512 byte sector block device driver must NOT allow DOS to re-format the partition because that would allow DOS to place things which have to be whole native 4k sectors at fractional locations. Also, DOS must NOT be allowed to repartition the disk, but that is easily done by not giving int 13 access to it. Block device drivers do not provide int13 anyway, so that one is easy? As long as DOS is not able to change partitioning or boot sector, the whole set of FAT32 structures will stay nicely aligned to 4k units. Apart from that, I still believe: Convert the BPB and boot sector on the fly, and DOS will be totally fine with looking at (and writing to!) a native 4k sector partition through looks like 512 glasses... :-) And of course it is nice that the magic values are just before the end of the first 512 bytes even when the native sector size is larger, makes things easy. Eric -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
yes you would see a problematic mismatch if you were to talk raw SCSI or CHS to a disk while being inconsistent about whether you use 512 byte or rather 4096 bytes per sector... That's precisely the problem. Depending on which DOS programs you use, some simply call DOS, some may use INT 25h/26h, some may use INT 13h, others may use raw SCSI/ASPI, and there are also other possibilities. If you could force all calls through the DOS device driver, you could eliminate any potential problems with DOS. But DOS doesn't force programs to do that, and there are certain kinds of programs (FDISK, FORMAT, SYS, CHKDSK, SCANDISK, ...) that require low-level access and can't go through the DOS Device Driver. And that doesn't even address the potential issues of using that same device on another OS or BIOS. The potential compatibility problems are just too great for me to even want to attempt this. FYI, my driver provides an INT 13h interface (and I think it's the only one that does). That allows you to use standard DOS utilities to partition and format and create bootable disks, without needing to resort (at least in some cases) to the special Windows and Linux utilities. However, when DOS mounts a partition with help of a loadable block device driver, nothing would access the raw view of the disk. In any case the kernel will not, it will only talk to the block device driver. Anything beyond that is of course evil, but so is dosfsck-ing a partition in Linux while it is mounted at the same time. It actually isn't evil in DOS to do low-level maintenance on a mounted partition. You just need to make sure you flush any buffers and caches, and re-mount if anything that DOS may care about (sector size, number of sectors, cluster size, ...) is changed. In the case of removable media (like USB and floppies), that shouldn't even require a reboot. Can you recommend any free int 13 or block device based delayed / pooled write caching software? As far as I can remember, all modern (LBA compatible, given disk sizes on current PCs) implementations of this are commercial. I don't know of any, but there's definitely a need for one. I normally don't use caches at all, but when I need one it needs to be a write-delay cache and I use MS SMARTDRV. I actually don't like SMARTDRV very much (it uses too much memory and requires a reboot to uninstall), but I have it and it works OK. I don't find write-through caches like LBACACHE and UIDE to provide enough speed benefit to be very helpful (though they might increase disk life to some degree). In addition, LBACACHE and UIDE are INT 13h caches instead of INT 21h caches (like SMARTDRV is), so they don't work with all disks (including USB and SCSI). When I really need to increase disk access speed (a big problem in some VM's), I copy my commonly-used utilities and batch files to a RAM disk. Thanks, but can you benchmark it? ;-) I've already done a simple benchmark, though I don't think it's exactly what you're looking for. It's discussed on pages 158 and 159 of USBINTRO.DOC (in the Sector Transfer Size discussion of the USBDRIVE section). Basically, the speed increases when you pool large numbers of contiguous sectors and send them at the same time. A true, valid benchmark would require all sorts of external parameter variability elimination, which I probably won't ever do. Only if you can look beyond the virtualization in a bad way... For example the same issue existed with ontrack ezdrive drivers installed in the master boot record: ... There are lots of examples of this in the past. I also remember when there were some programs that used incorrect formulas in the CHS - LBA translations that created all sorts of compatibility problems. That's why I think it's dangerous to do anything unique that no other OS or BIOS does. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
To set the record straight on caches and on UIDE -- Can you recommend any free int 13 or block device based delayed/ pooled write caching software? As far as I can remember, all modern (LBA compatible, given disk sizes on current PCs) implementations of this are commercial. I don't know of any, but there's definitely a need for one. I normally don't use caches at all, but when I need one it needs to be a write-delay cache and I use MS SMARTDRV. I actually don't like SMARTDRV very much (it uses too much memory and requires a reboot to uninstall), but I have it and it works OK. I agree with the above -- Write Back (delayed-write) caches usually are commercial, as they require a LOT of work and must be sold for a price. And I too absolutely HATED SMARTDRV -- Takes the most memory but caches the LEAST amount of data for it! I used Norton NCACHE2 for years, which is also a memory hog, but not as bad for the amount of data it handles. I don't find write-through caches like LBACACHE and UIDE to provide enough speed benefit to be very helpful (though they might increase disk life to some degree). In addition, LBACACHE and UIDE are INT 13h caches instead of INT 21h caches (like SMARTDRV is), so they don't work with all disks (including USB and SCSI). When I really need to increase disk access speed (a big problem in some VM's), I copy my commonly-used utilities and batch files to a RAM disk. No cache will ever compete with a RAM disk. RAM is always faster than disks with their seek/rotational latencies and their much slower transfer rates. But I dispute your comment about Write Through caches offering not-enough speed benefit, and I bet all VERY happy! users of UIDE might argue with you, as well. There is a REASON why Write Back caches are all so large -- they demand MANY hooks of a non-generic type into a DOS system, to handle Ctrl/Alt/ DEL and other events that require a flush of sectors not-yet written to disk. UIDE only needs to hook the Int 13h vector to do its functions. Also, UIDE is not just an Int 13h cache -- It CAN still be called by user drivers which desire its caching, though I am sadly aware that no one has ever done this. DOS driver development, or enhancement, isn't done that much any more. I did everything possible to make UIDE a FAST Write Through cache. It integrates caching with UltraDMA I-O for disks/CDs/DVDs; it can do direct UltraDMA to/from cache buffers to save time; it uses a binary-search when looking for cached data blocks (NOT hashing, which breaks-DOWN in speed at about a 35%-40% full search table); and it is only 5.2K (UIDE) or 4.5K (UIDE2) of assembly-language, as I REFUSE to have God-forsaken C in any SYSTEM driver as important as UIDE is!! But, if you wish to continue giving-up 40K+ for SMARTDRV, 80K+ for Norton NCACHE2 or however-many K for other Write Back schemes, feel free to do just that! I and others will continue using the 944-byte UIDE/UIDE2 and will be VERY happy! with the speed increase we DO get, from doing so! [And if there are still those who do not understand how UIDE/UIDE2 manage to require only 944 bytes of upper/DOS memory, just call it Sorcery!!]. -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Eric Auer e.a...@jpberlin.de wrote : Interestingly, even 3 TB disks are still sold with 512 byte sectors. Conversely, even 1 TB USB disks are already sold with 2048 byte sectors ;=) (...snipping much...) By the way - a DRIVER could interface with any disk with any sector size and then just provide an int13 or int25/26 interface with 512 byte sector size for data transfer to DOS. This scheme won't work if the disk has to be usable aslo in other OSes like Windows XP, Linux, etc. that recognise the 4k sectors natively ! Someone, maybe not Eric, asked what I have been using for accessing USB mass storage in DOS. Answer : a version of Panasonic's USBASPI.SYS. This allows access to 4k sectors using SCSI commands. A DOS driver similar to DI1000DD.SYS would be needed in order to access and mount the partitions. /Of course/ the standard DI1000DD has /hard coded/ 512k sector size for hard disks - but even it it was hacked/rewritten so it would transfer 4k sectors to/from the USBASPI interface, it won't work untill the DOS kernel work buffer and structures are adapted. I'm eager to see this prioritised, even though I am afraid I can't help - never learned C :( Regards -- Czerno -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 14-1-2012 23:23, Eric Auer schreef: Talking about new ways of booting, it would be very interesting to have a CD / DVD / BD boot loader for DOS. However, after you loaded the kernel, you still need some initrd from which you can load a CD/... driver pair like UIDE + SHSUCDX or ...CDROM + MSCDEX. As we use MEMDISK (a bootable RAMDISK which can be loaded by any bootmenu which can load Linux) for that now anyway, the only gain would be a little bit of memory saving: Instead of booting a ramdisk, the boot loader itself would somehow give access to a FAT diskimage on CD/... How about this? : http://www.syslinux.org/wiki/index.php/MEMDISK#ISOHYBRID_images Unfortunately I've got no idea how it technically works, or how it would apply to DOS. Perhaps a bootsector with hardcoded sector address for the FreeDOS kernel, just like Syslinux uses for its LDLINUX.SYS I wouldn't mind the extra flexibility, as some people like harddisk-style emulation instead of CD. label ramhdd # load hybrid as virtual partitioned HDD, contents is read-only FAT? linux memdisk initrd hybrid.iso # load hybrid ISO as virtual CD, contents is read-only ISO9660 label ramiso linux memdisk initrd hybrid.iso append iso Now how exactly did we end up on the users list with all this technical mumbo jumbo? -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Bertho, I spoke from the point of view of the device (the hard drive) - if the hardware that the device is attached to chooses not to expose all of the options that the device supports, there is little the device can do about that. In the case of your external storage somebody made a design decision that 4KB blocks were more desirable, and in a lot of cases that is true. But I'm quite sure that the underlying devices (hard drives) will support 512 byte sector sizes in emulation for a very long time. To be able to share data on a hard drive directly between two different operating systems the format of the device and the layout of the filesystem on that device have to be understood by both operating systems. Linux and Windows have a clear advantage; there are lots of people working on them and they have the supporting code and operating system resources to do lots of transforms on the data. We are more limited in DOS - DOS expects FAT or an installable filesystem (network redirector interface). If you give DOS a block device (whether through BIOS or device driver) DOS expects 512 byte blocks. (The network redirector interface is a higher level file interface, so it does not have such problems.) If you want to write a block device driver for DOS to use with FAT you can accommodate any physical sector size you want - as long as the device driver hides that and gives DOS 512 byte blocks, you will be fine. The network redirector interface can do magic - it is often used to take the 2048 byte blocks of a CD-ROM and the ISO filesystem on it and present it to DOS in a way that can be used by DOS. The important part is that it doesn't try to present a CD-ROM and the foreign filesystem on it (ISO) as a block device to DOS; it is a higher level installable filesystem interface. Given enough code, you can have an IFS driver to read EXT2, EXT3, NTFS, etc. So the bottom line is that DOS will probably work just fine when natively attached to storage devices, and that will work for a long time. Appliance storage devices are going to break that if they can't emulate 512 byte sectors. I'm not entirely sure that Linux or Windows will let you create a 512 byte FAT style filesystem on storage that is using larger sectors. If they can do that and you want to share data with them on your DOS system by directly reading the storage, then it's time to start writing some device drivers. ;-0 Mike -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
FWIW: In my USB disk driver (USBDRIVE), here's what I've done. USBDRIVE does not try to virtualize the sector sizes as others are suggesting here as a possibility -- I figure doing that has the potential to cause as many problems as the alternative (using defective utilities/programs that are hard-coded for 512-byte sectors). USBDRIVE will only assign drive letters to disks/partitions that it believes DOS can handle properly. E.g., USBDRIVE will ignore partitions that aren't formatted properly (NTFS, HPFS, FAT32 in certain cases, etc.). Likewise, it will ignore disks with sector sizes larger than what DOS says it can handle (this particular detail is part of the easily accessible DOS List of Lists). In the source code for USBDRIVE (starting at line 289 in the latest official release of USBDRIVE.A36), I have a comment that shows you how you can modify MS-DOS and IBM-DOS to handle sector sizes larger than 512 bytes. It involves using DEBUG (or something similar) to modify the DOS kernel (MSDOS.SYS or IBMDOS.COM). I don't know if the same method will work with FreeDOS or not. With the modification, DOS itself can supposedly handle sector sizes up to 8192 bytes, which means you can read/write to the disk using standard DOS internal calls. Programs and utilities that do not let DOS do the work for them may have problems. Booting from such a device is another level of complexity, and I'm not sure which versions of DOS (if any) can boot from a disk with sector sizes larger than 512 bytes. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bret, USBDRIVE does not try to virtualize the sector sizes as others are suggesting here as a possibility -- I figure doing that has the potential to cause as many problems as the alternative... Maybe you could make that configurable, so people can experiment with virtual 512 byte sectors at their own risk? Might be useful for people who want to make their disk oriented tools 4k-wrapped- to-512-sane :-) Likewise, it will ignore disks with sector sizes larger than what DOS says it can handle... comment that shows you how you can modify MS-DOS and IBM-DOS to handle sector sizes larger than 512 bytes. It involves using DEBUG (or something similar) to modify the DOS kernel I was not aware that it was THAT easy... I mean for example the low disk transfer buffer in SDA... If you make it larger, other things will change their offset, which may confuse other apps / drivers... But then, I am no expert for *MS* DOS kernels :-) Can you send the URL of that MS DOS patching instruction? I am wondering how much it could tell us about how to patch FreeDOS but of course FreeDOS already is known to have a bit more hard coded 512 byte ish ness. On the other hand, FreeDOS sources are easily available and easy to read... :-) (MSDOS.SYS or IBMDOS.COM). I don't know if the same method will work with FreeDOS or not. With the modification, DOS itself can I do want to know :-) supposedly handle sector sizes up to 8192 bytes Interesting. Any known side effects? Of course it will not help tools like FORMAT or CHKDSK unless they already are sector-size flexible. In that sense, I still think it will be pretty useful if the KERNEL can handle FAT file systems on virtual-512-byte-sector drives with 4k sector size, even if e.g. FORMAT would not. Eric PS: I think for virtual-512 you only need to recalculate the BPB values in the boot sector before you show them to DOS (only works for FAT anyway). For the rest, it SHOULD not matter much whether you R/W a 8*512 or a 1*4096 chunk. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
In response to : Bret Johnson Subj : MSDOS - increasing max sector size ... Likewise, it will ignore disks with sector sizes larger than what DOS says it can handle (this particular detail is part of the easily accessible DOS List of Lists). In the source code for USBDRIVE (starting at line 289 in the latest official release of USBDRIVE.A36), I have a comment that shows you how you can modify MS-DOS and IBM-DOS to handle sector sizes larger than 512 bytes. MSDOS technicalities may be slightly off-topic here :=), anyway... According to Rudolf Loew, increasing maximum sector size in LoL of an unpatched MSDOS will work up to 2048 byte sectors, not 4096 :( I have not verified it for a fact. (Incidentally R. Loew *sells* patches that bring the limit up to 32 kbytes in MSDOS 7.10, claims the author). In your own writing Bret, I think you say you did /not/ try 4k sectors. Too bad! -- Czerno -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
USBDRIVE does not try to virtualize the sector sizes as others are suggesting here as a possibility -- I figure doing that has the potential to cause as many problems as the alternative... Maybe you could make that configurable, so people can experiment with virtual 512 byte sectors at their own risk? Might be useful for people who want to make their disk oriented tools 4k-wrapped- to-512-sane :-) Possible, but a REALLY BAD IDEA if you ask me. The most obvious problem is that the disk will instantly become non-portable, and can't be used with other OS's or even other DOS drivers that don't virtualize the sectors. Internally, there are lots of technical issues with things like DMA and buffering that will probably cause at least as many, if not more, problems than trying to use hard-coded-512-byte programs with disks that aren't 512-bytes. There is also a significant speed and memory usage problem -- all disk transfers will need to be double-buffered (the driver will need to internally store the entire large sector and then transfer only the part that was requested). Again, possible, but NOT a good idea, IMHO. Can you send the URL of that MS DOS patching instruction? Go to my web site: http://bretjohnson.us and download the source code for the USB drivers (USBSOURC.ZIP -- the link is near the bottom of the first section). Inside the zip file is USBDRIVE.A36, and the comment that shows how to do it is around line 300. I am wondering how much it could tell us about how to patch FreeDOS but of course FreeDOS already is known to have a bit more hard coded 512 byte ish ness. On the other hand, FreeDOS sources are easily available and easy to read... :-) From what I understand, in MS-DOS and IBM-DOS, the maximum sector size is stored in only one place (part of the List of Lists). This value is adjusted while booting to match the largest sector size of any disk that the kernel finds and assigns drive letters to. In the kernel code, it always uses the value stored there in its various calculations, and never uses any hard-coded values. That's the way FreeDOS _should_ be as well, though I don't know if it actually is. Interesting. Any known side effects? Not that I know of. I've been using a 2048-byte sector size in MS-DOS for a long time now (since USBDRIVE came out), and haven't had any problems. I should point out that I don't actually have any disks with other than 512-byte sectors, so my testing is not very complete. But this trick has been around for a LONG time, and I've never seen any mentions of problems. Of course it will not help tools like FORMAT or CHKDSK unless they already are sector-size flexible. In my opinion, there's no legitimate reason for programs like FORMAT and CHKDSK to disallow non-512-byte sectors. Other sector sizes have been around since before MS-DOS was created, and MS has always stated that programs should never assume 512 bytes. I can't say this for sure, but I don't think any MS utilities ever had problems with other sector sizes. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Bret, maybe virtual 512 byte sectors are actually not that evil: Imagine a NORMAL 4096 byte sector based FAT32 filesystem. Each cluster and each FAT will be a multiple of 4096 byte in size, as will be the boot and fsinfo sectors. In FAT32 the root directory is just any directory, so like all the directories, it will consist of 1 or more normal clusters. So what are the differences between that FAT32 system and a system with 512 byte sectors where all structures only HAPPEN to be multiples of 4096 bytes in size and offset? BPB bytes per sector differs (512 vs 4096) BPB sectors per cluster differs (? vs ?/8) BPB total sectors differs (? vs ?/8) BPB sectors per fat differs (? vs ?/8) BPB CHS geometry differs - but does a disk with 4096 byte sectors allow CHS based access at all? I hope it does not. BPB hidden sectors differs, as in offset of partition *but* a DOS kernel which uses a disk via an external block device driver will not have access to the raw disk at all because the accessible area starts at the partition then. BPB FAT32 root dir position is the same (a cluster number) BPB FAT32 fsinfo sector and backup boot sector positions differ and are ? versus ?/8 again. Common values are NOT multiples of 8, but DOS of course does allow them... :-) BPB reserved sectors differs, as in how many sectors are reserved for boot, fsinfo, backup boot and really reserved areas. Common values are multiples of 16 for a 512 byte per sector filesystem, not sure about 4096 byte filesystems. As long as you can express the size of the reserved area of your 4096 byte per sector disk in units of 512 bytes, things work - which is always true. In short: To mount a 4096 byte per sector disk with a driver that allows you to access numbered 512 byte areas AS IF the sector size would be 512 bytes per sector, you have to multiply some BPB values by 8 because N sectors of 4096 bytes each equal 8 times N areas of 512 bytes... You also have to change the sector size from 4096 to 512 to tell DOS that access is in units of 512 bytes. No other steps should be needed. Also, because 4096 is just a multiple of 512, this will always (*) work. Only going the OTHER way round is hard: You cannot install a simple diskimage of a 512 byte per sector FAT filesystem on a disk with 4096 bytes per sector, because you cannot know that all structures align to multiples of 4096 bytes and because you cannot have fractional locations of sizes. * Note that copying a diskimage of a 4096 byte per sector disk to a 512 byte per sector disk, or using a driver to access a native 4096 byte per sector disk in VIRTUAL 512 byte per chunk sectors can SOMETIMES overflow values: 1. Large sector disks can have cluster sizes above 64 kB Such filesystems cannot be transformed to 512 byte/sec 2. Large sector disks can hold FAT32 filesystems 2 TB. Such filesystems cannot be transformed to 512 byte/sec 3. Large sector disks can have reserved area sizes above 32 MB, which could not be expressed as uint16 count of sectors after the transform to 512 bytes per sector but in practice, reserved areas are always below 1 MB. In short: Expressing a 4096 byte per sector disk with a FAT32 filesystem on it in terms of a virtual 512 byte per sector FAT32 system ALWAYS works, simply by converting a few values in the BPB of that disk, UNLESS the filesystem or cluster size are larger than would be possible on 512 byte per sector classic harddisks in the first place :-) Internally, there are lots of technical issues with things like DMA and buffering that will probably cause at least as many, if not more, problems than trying to use hard-coded-512-byte programs with disks that aren't 512-bytes. There is also a significant speed and memory usage problem -- all disk transfers will need to be double-buffered While it is true that performance is lost if writing a virtual 512 byte sector means reading-modifying-writing a real 4096 byte sector, this loss in performance is a general DOS problem: Other operating systems would gain performance by pooling writes to MULTIPLE sectors which makes a very noticeable difference on flash based disks such as SSD or USB flash sticks. As for the DMA and memory usage, you have to suffer the real 4096 byte sectors at SOME place anyway... If it is not in the allow access to virtual 512 byte per sector disk representation block device driver, you will have to suffer larger sector buffers in the DOS kernel's SDA and BUFFERS areas instead for native 4096 byte sectors. So you arguably even suffer more with the 4k/sec kernel. (the driver will need to internally store the entire large sector and then transfer only the part that was requested). Again, possible, but NOT a good idea, IMHO. Yet that is exactly what many caches including LBACACHE and SMARTDRV do: To reduce the size of bookkeeping data they store data in multi sector bites of memory. Still transfers need not be rounded up to full bites. For the native
Re: [Freedos-user] Re : Support for 4k byte sectors
maybe virtual 512 byte sectors are actually not that evil: Imagine a NORMAL 4096 byte sector based FAT32 filesystem. ... Actually they are, or at least potentially are, at least from a compatibility perspective. In the case of USB, the SCSI protocol is normally used. The sector size is not simply stored on the disk (in the BPB's). It is also stored outside the disk in the SCSI overhead that is part of the initialization process, and those can't be virtualized (there's more than one of them). The sector size that the SCSI overhead tells you and what's stored on the disk itself need to be exactly the same, or you'll likely to run into problems somewhere that will either prevent a disk from being recognized, or contaminate the data. BPB CHS geometry differs - but does a disk with 4096 byte sectors allow CHS based access at all? I hope it does not. Already answered by Bertho -- of course it does. CHS addressing and 512-byte sectors aren't synonymous. ... Other operating systems would gain performance by pooling writes to MULTIPLE sectors which makes a very noticeable difference on flash based disks such as SSD or USB flash sticks. Some DOS programs already do that, though usually only the ones that use low-level INT 13h access. I don't think DOS itself ever does. ... This is why I suggest to first grow such a file to the final size, putting all FAT writes into ONE access and then writing the file's contents in large N kB chunks. Excellent idea, especially with USB. Then I expect the side effect to be that the kernel inflates all sector-sized areas to 4096 bytes, to be prepared to handle 4096 byte sized sectors in SOME disks. For all old disks with classic 512 byte sectors, this means a lot of wasted space in RAM for each such buffer in memory. Unless I misunderstood the details, MS DOS would not adjust sizes on a per-buffer basis. That is correct -- the buffers are all the same size, big enough for the largest sector of any disk. It does indeed waste memory, especially if you set BUFFERS to a large number. If somebody was ambitious enough, they might be able to implement dynamic buffer sizes, I suppose (I don't know what the required data structures for DOS Buffers, if there are any, are supposed to look like). ... You mean you know any sort of USB storage device which has native 2048 byte sectors? I thought that for flash and large disk formats, only 4096 was popular and 512? No, I personally don't have anything except 512-byte sectors. I just did 2048 as an experiment to see if there were problems, while still keeping the size small enough that I didn't waste too much memory. So you actually do the virtual trick the other way around? As mentioned above, that only works when all relevant structures on the actually-512-byte-per-sec disk are aligned to and sized in 2 kB boundaries, not counting the global offset of the partition, though. Unless I'm misunderstanding what you're saying, it's not the other way around. If a buffer is 2048 and the sectors are only 512, the last 1536 bytes of the buffer are simply never used. There aren't any cluster alignment issues. ... if we can mount a 4096 byte per sector disk in DOS, no matter what tool and operating system formatted / fdisked it then we can already enjoy using those disks :-) Yes. But for reasons already discussed, this should not be done by virtualizing the sector size -- that is asking for problems, IMHO. ... By the way, is the 55aa boot sector thing and similar flags for fsinfo and 2nd stage boot sectors on FAT32 always at the end of the sector or rather just at the end of the first 512 bytes of the sector? Also already answered by Bertho. It's always at offset 510 in the sector, not necessarily the end of the sector (the MBR and VBR's still need to fit in 512 bytes even if the sector is larger). If the sector size is larger than 512, you can use the rest of the sector to store other things if you want, though. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Op 14-1-2012 19:48, Bertho Grandpied schreef: It's been a few days and I'm surprised my first mail hasn't been acknowledged in any way, let alone answered; strange, I've been part of various lists before, usually 'newbies' are greeted rather than ignored altogether. So I'll reiterate and articulate the above question just in case it was not clear : have I done something wrong ? should I try posting to the kernel or developers lists instead ? Truth is we're not sure, this 4K sector size thing comes in several versions: 1) emulation with aligned 512byte sector emulation 2) emulation with non-aligned 512byte sector emulation 3) native 4K (especially USB bridges apparently) In other words, this is asking what the plan is for the FreeDOS kernel to be able to mount mass storage devices having 4 kilobytes per sector ? If the disk has traditional BIOS partitioning layout (MBR, primary/logical partitions with FAT16/FAT32 filesystems) then it might be possible for the kernel to work with this as long as it is a data disk. Booting from a partition without 512byte sector-size is probably more challenging, let alone guarantee file(-system) integrity and disk manipulating (defragmentation programs, filesystem checkers like CHKDSK and DOSFSCHK). GPT partitioning scheme isn't supported at all (nor is EFI/UEFI without BIOS emulation). As said in previous message, I for one am ready to test development kernels against my USB disk appliance (1 TiB Iomega Prestige). freedos-kernel would be most appropriate list as the developers on there have most expertise. However they're also the ones who are rarely present due to other interests or obligations, so getting answers can take a while. You could also try freedos-devel for this specific technical question but answers might take as long as getting answers on the users list. People usually don't answer if they don't have the correct answer, thus things stay quiet for a while. -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
The list membership is not that large. You can assume that people are busy or don't know the answer. As far as 4K blocks go, I wouldn't worry about it too much. 512 byte sectors will be supported either natively or by emulation in the drive itself for a long time to come - at least 5 to 10 years. Too many existing systems depend on a 512 byte sector and manufacturers are more likely to demand reasonable 512 byte emulation from the hard drive makers than to do anything themselves. Mike -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Mike, The list membership is not that large. You can assume that people are busy or don't know the answer. I was grepping though a backup of our FAQ (which is down, is it worth the effort to replay a September 2008 backup? Apparently SourceForge server directory structure changed...) but did not find the topic discussed. As far as 4K blocks go, I wouldn't worry about it too much. 512 byte sectors will be supported either natively or by emulation in the drive itself for a long time to come - at least 5 to 10 years. Too many existing systems depend on a 512 byte sector and manufacturers are more likely to demand reasonable 512 byte emulation from the hard drive makers than to do anything themselves. Interestingly, even 3 TB disks are still sold with 512 byte sectors. Disks with 4096 byte sectors are not always supported for booting, but that probably depends on how new your BIOS is. Using small 512 byte sectors with multi-TB disks means that you can no longer use a MFT based partition table. Instead, GPT partitions have to be used: (quoting some September 2009 notes about GPT partition tables) Luckily EFI does not seem to be complicated - EFI partitions ignore all CHS values and have a partition ID of 0xef, and classic partitions can coexist with them... Header, padded to suitable power of 2 size: qword 0x5452415020494645 / EFI PART dword 0x0001 version dword header size dword header crc32 dword 0 qword lba address of THIS sector qword lba address of alternate header(?) qword first data block of partition? qword last data block of partition? 16 bytes GUID to uniquely identify disk qword lba address of GUID partition entry array dword array size dword array crc32 padding Partition entry, 128 bytes each: 16 bytes partition type (0 if unused) 16 bytes GUID to very uniquely identify partition qword lba address of partition start qword lba address of partition end qword attribute bits (only low bit required?) (high word reserved, middle 47 bits undefined?) 72 bytes UTF-8 (correct?) partition name string (end quote) We also had a long thread on freedos-user in April 2011 about the topic Large drives with 4k sectors presenting as 512b? Note that the BIOS does not let you query the sector size (e.g. eltorito.sys has to detect by trying: either 512 bytes disk style or 2048 bytes raw CD/DVD/BD) and that many apps just assume 512 bytes... Also depending on your BIOS, you could have a limit of at most 2^28, 2^32 or 2^48 sectors per disk. There also is a DOS UIDE cache-and-driver. For example LBACACHE is one of the tools which assume 512 bytes per sector and uses only the first 2^32 sectors which is 2 TB for 512 byte/sector. Even for disks with large native sectors, the BIOS or the firmware of the harddisk may still support access to 512 byte units. I guess actual geometry of modern disks does not have much to do with classic CHS or LBA-512-byte etc anyway. Flash / SSD disks can be much faster when access is done in aligned multiples of 4kB or even much larger. Of course it is also simply INTERESTING whether we could support 4 kB sector size. After all, eastern versions of MS DOS also supported floppies with 1 kB sector size and SMARTDRV and ramdisks also tend to support sectors of not only 512 bytes size. This is also useful for caching CDs. Note that CDRCACHE simply has a fixed 2 kB sector size... By the way - a DRIVER could interface with any disk with any sector size and then just provide an int13 or int25/26 interface with 512 byte sector size for data transfer to DOS. For transfers which do not access (aligned) blocks of 4 kB size, performance will be worse than native, and you cannot boot from such a disk, but at least this would WORK and you could even have a driver as part of the boot chain before DOS is loaded, just as Ontrack / Ezdrive MBR-loaded drivers which added LBA support to PCs without a LBA BIOS, decades ago. Making the KERNEL sector size of DOS 4 kB has the side effect of BUFFERS also being 4 kB each - wasting 3.5 kB in each buffer when you access a 512 byte / sector disk with such a kernel. Regards, Eric -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
At this time there are no plans to explicitly support 4K block sectors directly in the kernel. The kernel itself does not support any block devices directly, it relies on the BIOS interface, though additional drives or support may be added through device drivers. I would like to review the FreeDOS utilities to ensure we are optimally using 4K sectored drives when emulation is being used (i.e. 4K aligned read/writes where feasible), but can not provide any specific timelines as I still need to figure out how 4K sectors effect the various utilities and how best to handle them. In your specific case, given that USB support is not directly in the kernel I would need more information about which USB stack you are using to access the drives. Assuming the driver is providing BIOS LBA based extended read/write functions and get device parameters reports sector size of 4K then it may be feasible to update the kernel to support such a device - at least via a kernel provided translation (512 byte emulation) - for data purposes. The boot code is currently limited to 512 byte sectored devices, but actually 4K sized sectors could be supported with a different boot sector if the computer's BIOS would support booting such a device. I would need to research the code and other issues before providing a good response. Its been a while since I read the cluster to sector code, I believe since DOS works at cluster level primarily, accessing 4KB drives is feasible without breaking too much compatibility assuming minimal of 4KB cluster size; though for many DOS programs 512 byte access is expected and would either need to be provided via some emulation or simply indicate such programs are not supported with 4KB sectored drives. Note: I am not sure any current drives/BIOS combinations will actually provide correct sector size of drive. Sorry it has taken a while to respond, I follow the list fairly often but not on a computer and I find it tedious to to reply on a tiny touch screen. I also get automatically emailed for most sourceforge bug issues though I only respond when I have something useful to add. Jeremy -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Eric, I expect that in the next few years we'll see very large hard drives and they will continue to support 512 byte sector sizes - that is what the system manufacturers demand. The actual sector size of the drive might be 4KB but the drive will allow the host to choose a 512 byte or 4KB sector format and adjust accordingly. If 512 byte sectors are chosen and the drive is truly using 4KB it will do the proper read/modify/write sequence for writing random 512 byte blocks. For reads and sequential writes the performance impact will be negligible. Random writes present the biggest challenge - better drives will be able to minimize the performance impact. Most of us run with write caching enabled which also helps the performance problem that random writes cause. From an operating system point of view things are more complicated. The classic partitoning scheme doesn't work well on giant drives. You are better versed in the newer partition table schemes than I am. We might just have to live within the limits of a 2TB device if we don't want to fix the problem. I think I can live in 2TB for my DOS systems ... In the early days of DOS device drivers often had to deal with sector sizes that were not 512 bytes. I have an Iomega Bernoulli Box A220 (2 8 cartridges holding 20MB each) that uses 256 byte sectors, and the device driver is responsible for dealing with that. I think even the later Adaptec ASPI drivers know how to work with SCSI devices that report 256 byte sectors. If we ever have to deal with this issue in the kernel I would be inclined to continue to use a 512 byte sector size within the OS and hide any differences at the lowest levels. This might not be as efficient as supporting different sector sizes natively but that probably becomes too complicated and error-prone. There is a lot of code that assumes 512 byte sector sizes and it is not worth disturbing it. I'm going to get flamed for this but I'm not too worried about the performance hit that translation layers cause. FreeDOS is *insanely* mismatched for modern hardware, and there is plenty of performance overhead available to dip into. Anybody who really needs the speed should step up to a more modern operating system. We're not making use of the hardware we have already and we're probably a decade or two late in trying to keep up with advances in hardware. Mike -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Jeremy, At this time there are no plans to explicitly support 4K block sectors directly in the kernel. The kernel itself does not support any block devices directly, it relies on the BIOS interface, though additional drives or support may be added through device drivers. Unfortunately, FreeDOS does not support device drivers with other sector sizes than 512 bytes either. There were discussions about this a few years ago, mainly because a RAMDISK can in some cases be more efficient with other sector sizes, in particular smaller. As BUFFERS and the transfer buffer in low memory (below 640k) and possibly other structures (in SDA?) all have to be at least as large as the biggest supported sector size, I think it would not be very nice to support 4 kB sector size in a default FreeDOS kernel... I would like to review the FreeDOS utilities to ensure we are optimally using 4K sectored drives when emulation is being used (i.e. 4K aligned read/writes where feasible), but can not provide... FORMAT, in the FAT32 case, has a command line option to align the data clusters to multiples of 4 kB. I think that got added because Windows also introduced the feature and it felt useful for SSDs. Not sure about FDISK and similar tools. In general, I recommend to use the more comfortable and also free open source GPARTED from a bootable CD / DVD or USB stick for partitioning, not classic DOS. I have a strong intuition that neither CHKDSK nor DOSFSCK for DOS support other sector sizes than 512 bytes. Basically all software which acts only on files does not care for your sector size anyway, only low level tools are an issue here. (512 byte emulation) - for data purposes. The boot code is currently limited to 512 byte sectored devices, but actually 4K sized sectors could be supported with a different boot sector if the computer's BIOS would support booting such a device. I would need to research... Maybe somebody else can report experiences about booting ANY system from disks with 4 kB sector size: I guess only newest BIOSes work. Talking about new ways of booting, it would be very interesting to have a CD / DVD / BD boot loader for DOS. However, after you loaded the kernel, you still need some initrd from which you can load a CD/... driver pair like UIDE + SHSUCDX or ...CDROM + MSCDEX. As we use MEMDISK (a bootable RAMDISK which can be loaded by any bootmenu which can load Linux) for that now anyway, the only gain would be a little bit of memory saving: Instead of booting a ramdisk, the boot loader itself would somehow give access to a FAT diskimage on CD/... while since I read the cluster to sector code, I believe since DOS works at cluster level primarily, accessing 4KB drives is feasible without breaking too much compatibility assuming minimal of 4KB cluster size; though for many DOS programs 512 byte access is expected and would either need to be provided via some emulation or... Luckily most programs do not work with sectors or clusters at all, they only care for files and directories. But you are right that tools like FORMAT / CHKDSK could break if they fail to check size. Interestingly, even MS DOS DRIVPARM and DRIVER.sys do not let you change sector sizes and MS UNFORMAT only goes up to 2 kB sizes, but MS RAMDRIVE and free ramdisks (TDSK) do support other sizes. In RAMDRIVE, 128 to 1024 bytes were possible, in TDSK 32 to 2048. To reply to another mail: 1) emulation with aligned 512byte sector emulation 2) emulation with non-aligned 512byte sector emulation 3) native 4K (especially USB bridges apparently) Depending on your BIOS, you may need a driver for USB disks in DOS anyway, so that driver can easily be extended to give you a 512 byte sectors view of the disk. The easily comes from the interesting situation that we have 2 free drivers from DOS fans but none from mainboard manufacturers. Self-driven DOS :-) According to an ancient (2002) post by Matthias Paul (hi :-)), MS DOS 7.1+ and DR DOS 6.0+ and DR DOS 3.31+ support up to 2 kB sector sizes (and min 32 bytes, or for DR DOS 3-5 min 128) and Win 95/98/ME have a minimum of 512 bytes but still max 2 kB. I would guess this is somehow related to using FAT on CD-RW/WORM? Some versions of DR DOS might support up to 16 kB sector size. Apparently pre-2003 FreeDOS DISKCOMP tried to support all sizes. The Microsoft fatgen103.pdf specs of the FAT32 filesystem say that sector sizes must be 512, 1024, 2048 or 4096 bytes, while some other aspects of pre-Win9x DOS suggest that small sectors of 32 bytes and more and maybe large up to 8 kB (maybe 16/32?) were also supported at least for e.g. ramdisks... Note that you cannot have more than 2^32 sectors in one drive and max 2^28 clusters in FAT32. So you need some drive letters. In a 2003 FreeDOS 1.0/after thread, Aitor considered COUNTRY- support, MS DOS style config.sys menu system, FASTOPEN, more than 1 UMB area, variable sector size, DRIVPARM, DOS for ROM. Also in 2003, I mentioned
Re: [Freedos-user] Re : Support for 4k byte sectors
Replaying a 12 May 2005 mail :-) [Freedos-kernel] Analysis: Support for sector sizes != 512 bytes Hi, I have browsed the CVS kernel a bit, and got the impression that it would not be too hard to support sector sizes below 512 bytes (32, 64, 128 and 256 bytes should be possible). For sector sizes above 512 bytes (e.g. 1024 and 2048), each element in BUFFERS and the DiskTransferBuffer would have to be bigger. Are there DOS versions which support big sectors? How do they solve the buffering problem? I think sector sizes below 64 bytes cannot contain enough of a BPB to be useful. Sector sizes below 32 bytes are definitely not useful, not even a single dirent fits in them. Sector sizes above 512 bytes are at most useful for the 1024 byte (far-east 1.2 MB diskette format / drives which support full 64 MB of a single XMS2 handle as ramdisk without having to support the DOS3.x+ 32bit sector number calls) and 2048 byte (raw sector size on CD-ROM and maybe on magneto-optical drives) cases. Comments about useful sector sizes would be welcome: Smaller (64, 128, 256), normal 512, Bigger (1024, 2048, others). Next part: I believe that the BIOS / kernel drives (diskette, harddisk) only have to handle 512 bytes per sector. Otherwise, you would have to adjust: CalculateFATData, DosDefinePartitions, the kernel driver IOCTLS (e.g. format track), LBA_Transfer, max_transfer (the DMA wrap checker) and maybe floppy_bpb (and -_bpbs). In case you have wondered, directory handling already reads the current sector size from the BPB, so it is flexible. If you want drives with bigger sectors to be detected at boot time, you not only have to get BUFFERs and DiskTransferBuffer bigger and adjust _maxbksize (in the global SDA data structure, which is, by the way, not yet read by FreeDOS itself, as only 512 bytes per sector are supported anyway) but you also have to make InitDiskTransferBuffer bigger as well. For boot purposes you just have to make SYS able to pad unused parts of the boot sector to allow bigger sector sizes - BUT you also have to make SYS able to adjust the boot sector code to that, I remember that not all our boot sectors actually used the BPB bytes-per-sector value (there was not enough space to process it). Smaller sector sizes are simply too small to be bootable anyway. I wonder if you should be able to boot from nonstandard sector sizes at all - for example a CD- ROM does not even have a FAT filesystem, so why would you boot from a (raw!) CD-ROM then? So I suggest to support nonstandard sector sizes ONLY for drives which are connected to separate drivers (e.g. ramdisks, USB drives, ZIP...). To make that possible, BUFFERS and DiskTransferBuffer handling would have to get better (even for smaller sectors - for bigger sectors, the BUFFERS and DiskTransferBuffers themselves, being only _maxbksize big, will be the problem). In addition, getbpb and dskxfer (THE central send/receive a sector to/ from a device driver function) have to be made independent from the hardcoded SEC_SIZE. Instead, they have to use the sector size from the BPB of the drive. Hint: getbpb assumes that the boot sector magic word is at offsets 1fe and 1ff. This IMPLIES a sector size of 512 bytes, too. The GOOD part is that FreeDOS hardly uses anything else than getbpb and dskxfer to communicate disk contents (for which it uses BUFFERS and DiskTransferBuffer as intermediate storage places). rqblockio does call block device drivers directly, too, but not to access sectors, for example, and the other functions all use dskxfer or getbpb directly or indirectly. I hope I have not overlooked things in this review... Summary: - To support bigger sectors, raise _maxbksize and make each element of BUFFERS and DiskTransferBuffer bigger - To support nonstandard sector sizes at boot time, CalculateFATData, InitDiskTransferBuffer and DosDefinePartitions have to be modified - To support nonstandard sector sizes for harddisk / diskette (kernel built-in drivers), you have to adjust the IOCTLs (format track...), LBA_Transfer and max_transfer - To support nonstandard sector sizes at all, you have to modify dskxfer and getbpb and the BUFFERS and DiskTransferBuffer handling has to accept sectors in other sizes than SEC_SIZE, including e.g. sectors which are smaller than _maxbksize... - SYS and boot sectors can only support bigger, not smaller, sector sizes, and SYS might have to do extra patching for those of the boot sectors which do not actually use the boot sector BPB sector size info We should probably start by using _maxbksize in BUFFERS and either only DiskTransferBuffer or both DiskTransferBuffer and InitDiskTransferBuffer to get things a bit clearer... Then we should make SYS and boot sectors able to support bigger sector sizes (not because it is really useful but because this part is more self-contained / easier to maintain than the rest of the kernel). Then the first WORKING support for nonstandard sector sizes can be added by
Re: [Freedos-user] Re : Support for 4k byte sectors
For what it is worth, my take on 4K-byte (or other-than-512 byte) sectors for DOS systems is very-much the same as Eric's and I shall reply to some comments from one of his posts -- Also depending on your BIOS, you could have a limit of at most 2^28, 2^32 or 2^48 sectors per disk ... Not a problem for a driver, as drivers usually assume the programs doing I-O will not try to access PAST the end of a disk! The disk directory controls how much data shall get read/written, and the disk driver takes its orders from a DOS system. Absolute disk size is only a problem when specific HARDWARE commands are needed with different sizes. In the SATA/IDE world, there is only 28-bit or 48-bit LBA addressing, and a 48-bit drive can accept 28-bit commands. Thus, in my UIDE driver, I issue 28-bit commands for up to 28-bit addresses, while I issue 48-bitters for larger addresses. Runs fine! ... There also is a DOS UIDE cache-and-driver. For example LBACACHE is one of the tools which assumes 512 bytes/sector and uses only the first 2^32 sectors which is 2 TB for 512 bytes/sector. As noted above, UIDE (also UIDE2 and UIDEJR) handles 48-bit LBA addressing, if any DOS system were ever to give it an Int 13h AH=42h/43h LBA block with addresses so large. I depend on the DOS system to determine if this is necessary, as UIDE does physical I-O only, not logical directory I-O where the 32-bit directory limit is of concern.Only the 48-bit LBA address limit matters to me and my drivers. By the way - a DRIVER could interface with any disk with any sector size and then just provide an int13 or int25/26 interface with 512 byte sector size for data transfer to DOS. For transfers which do not access (aligned) blocks of 4 kB size, performance will be worse than native, and you cannot boot from such a disk, but at least this would WORK and you could even have a driver as part of the boot chain before DOS is loaded, just as Ontrack / Ezdrive MBR-loaded drivers which added LBA support to PCs without a LBA BIOS, decades ago. Making the KERNEL sector size of DOS 4 kB has the side effect of BUFFERS also being 4 kB each - wasting 3.5 kB in each buffer when you access a 512 byte / sector disk with such a kernel. I agree with much of the above. Changing DOS, and all user apps, for any new sector size at THIS late date would be a Cast-Iron BITCH, in my opinion! [U.S. expression for BAD wives/girlfriends, who are usually made of softer items!]. For the moment, UIDE supports only 512-byte sectors, same as LBACACHE and almost every other DOS hard-disk driver. But, if new 4K-byte sector schemes provide a RELIABLE means for a drive to tell its driver what sector size is in use, I can change UIDE to detect that size! Not-much help for booting or with independent programs like FDISK, etc.But, Eric is correct: Changing only the DOS disk driver to accomodate large sector sizes, then letting the driver interface with the existing DOS system at 512 bytes/sector, would be a HUGE simplification of this entire issue! -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
Re: [Freedos-user] Re : Support for 4k byte sectors
Hi Jack, driver, I issue 28-bit commands for up to 28-bit addresses, while I issue 48-bitters for larger addresses. Runs fine! I meant if the BIOS only sees the first 2^N sectors and the partition table describes more than that, UIDE will probably not modify the BIOS-reported disk size: So on one side, UIDE could help DOS to access more of a disk than the BIOS, but on the other, that might be unsafe... Eric PS: LBACACHE does treat access beyond 2^32 sectors as a flush event and will not cache the additional sectors. Of course MBR based partitions cannot go 2^32 either. PPS: What do you think about using Ontrack/Ezdrive style load-before-kernel drivers to map from 4k to 512 sectors? -- RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 ___ Freedos-user mailing list Freedos-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-user
