Re: Issues with lseek(2) on a block device
On Mon, 26 Feb 2024 09:20:21 -0500 Thor Lancelot Simon wrote: > On Sun, Feb 25, 2024 at 04:41:37PM +, Sad Clouds wrote: > > > > The whole idea of Unix was "everything is a file" and the sooner people > > get rid of magic ioctls the better. > > But everything is *not*, actually, a file, and forcing things that > are not files into file-like shapes is not always the right design > choice. It is not about forcing things into file-like shapes. For example sockets are represented by file descriptors, but also have their own UDP, TCP, SCTP, etc protocol options and API extensions. There is a small set of core system calls which should work uniformly across all objects represented by file descriptors - open, close, read, write, stat, lseek, etc. The fact that you can use most of such system calls with files, pipes, sockets, block devices, character devices, etc. is a pretty good design in my humble opinion. For example System V shared memory API is just horrid, compared to Posix file-like shm_open, shm_close and mmap. Thanks.
Re: Issues with lseek(2) on a block device
On Sun, Feb 25, 2024 at 04:16:03AM -, Michael van Elst wrote: > t...@panix.com (Thor Lancelot Simon) writes: > > >Probably not a good idea to start with lseek() because if you _do_ > >encounter a tape device, seeking to SEEK_END could take you an extremely > >long time. > > lseek() doesn't move the tape. But the open() or close() may trigger > tape operations like loading or winding. Special files are special. Ah, yes, that's the MTEOM ioctl -- approximately. I have it stuck in my mind that SEEK_END did this, too, back when I was a literal tape jockey for 9 track tapes on an 11/750 and a collection of TK and QIC-150 drives. But this could be an invention of memory. -- Thor Lancelot Simont...@panix.com "The inconsistency is startling, though admittedly, if consistency is to be abandoned or transcended, there is no problem." - Noam Chomsky
Re: Issues with lseek(2) on a block device
On Sun, Feb 25, 2024 at 04:41:37PM +, Sad Clouds wrote: > > The whole idea of Unix was "everything is a file" and the sooner people > get rid of magic ioctls the better. But everything is *not*, actually, a file, and forcing things that are not files into file-like shapes is not always the right design choice. As simple as possible, but no simpler. -- Thor Lancelot Simont...@panix.com "The inconsistency is startling, though admittedly, if consistency is to be abandoned or transcended, there is no problem." - Noam Chomsky
Re: Issues with lseek(2) on a block device
On Sun, 25 Feb 2024 07:21:07 -0300 Crystal Kolipe wrote: > In most cases, unless we are talking about a low-level disk utility, if > userland code is trying to find out the size of a raw block device then it > seems like a design error. There are many valid reason why applications may want to work with block or raw devices directly. A design error is to place artificial barriers which stop people from doing useful things with those devices. The whole idea of Unix was "everything is a file" and the sooner people get rid of magic ioctls the better.
Re: Issues with lseek(2) on a block device
> On Feb 25, 2024, at 12:29 AM, Robert Elz wrote: > > ps: as a semi-related anecdote, the first system I ever personally > installed any unix version onto (way back in the mid 70's) booted from > a reel to reel (1/2" I think) tape - and by booted, I mean ran with its > filesystems (or filesystem, everything was in one) on the tape (read only, > naturally). HP-UX install media on QIC-24 back in the HP-UX 7/8/9 days (earliest I have experience with) was also mounted as the file system. It’s quite a trip to see (well, hear, anyway) a tape drive get randomly-accessed like that. And, yah, it’s not fast. -- thorpej
Re: Issues with lseek(2) on a block device
On Sat, Feb 24, 2024 at 10:54:56PM +, Taylor R Campbell wrote: > > Date: Sat, 24 Feb 2024 16:21:42 -0500 > > From: Thor Lancelot Simon > > > > On Wed, Feb 21, 2024 at 09:20:55PM +, Taylor R Campbell wrote: > > > I think this is a bug, and it would be great if stat(2) just returned > > > the physical medium's size in st_size -- currently doing this reliably > > > takes at least three different ioctls to handle all storage media, if > > > I counted correctly in sbin/fsck/partutil.c's getdiskinfo. > > > > I am not sure this can be done for all block devices. Tapes have block > > devices, and open-reel tape drives do not really know the length of the > > loaded media, while on any other tape drive supporting compression, there > > may really be no such size. > > Sure, it's fine if it doesn't give an answer (or, returns st_size=0) > for devices where there's no reasonable answer. > > But for a medium which does have a definite size that is known up > front, it should just be returned by stat(2) in st_size instead of > requiring a dance of multiple different NetBSD-specific ioctls to > guess at which one will work. We need to be careful with the definition of "definite size". In simple terms, we can think of examples such as: A physical HDD has a fixed size(*), so stat should report that, whereas a tape drive with removable media can potentially present different sizes so stat should return zero or indicate that situation in some other way. But in a world of virtual machines with block devices that are dynamically re-sizable on the host, it becomes more difficult. Consider such a device which _could_ be resized but in practice almost never is: If stat returns what it thinks it the current or stable size of the device and userland software retrieves and stores that value for future use rather than using it immediately, then bad things will likely happen if the real value changes later on. On the other hand, if stat returns zero for the device because it could possibly change size, (even though it almost certainly won't), then the userland code will not function in conjunction with that device even though it otherwise could. (*) Ignoring the fact that the drive firmware can often be configured to clip the size and report a smaller disk than is physically present. In most cases, unless we are talking about a low-level disk utility, if userland code is trying to find out the size of a raw block device then it seems like a design error.
Re: Issues with lseek(2) on a block device
On Sat, 24 Feb 2024 16:28:28 -0500 Thor Lancelot Simon wrote: > On Thu, Feb 22, 2024 at 12:08:14PM +, Sad Clouds wrote: > > Hello, thanks to everyone who responded with their suggestions. Using > > various non-portable ioctls I can device size on most platforms, for > > both block and raw devices. > > > > This is more convoluted than a single lseek() call, but it is what it > > is. If anyone wants to do something similar, then the following example > > can give you a good start. > > Probably not a good idea to start with lseek() because if you _do_ > encounter a tape device, seeking to SEEK_END could take you an extremely > long time. > > Thor Hello, good point, however the software I'm developing is only meant to work with storage devices like disks, capable of random I/O. If someone decides to point it at the tape device, they get what they deserve :-)
Re: Issues with lseek(2) on a block device
Date:Sun, 25 Feb 2024 04:16:03 - (UTC) From:mlel...@serpens.de (Michael van Elst) Message-ID: | lseek() doesn't move the tape. But the open() or close() may trigger | tape operations like loading or winding. Special files are special. The issue is more the return value from lseek() - POSIX requires that lseek(fd, 0, SEEK_END) return the "size of the file" on any device which supports seeking (ie: not ttys, pipes, etc). A tape that supports seeking would be one of those, so somehow, lseek() as it is worded now, would be required to return "the size of the file" whatever that actually means for a tape. I have submitted a bug report (just earlier today) to get that fixed, but it will be some time before anything happens (it is too late now for the next version of the standard, which is currently in its final stages). kre ps: as a semi-related anecdote, the first system I ever personally installed any unix version onto (way back in the mid 70's) booted from a reel to reel (1/2" I think) tape - and by booted, I mean ran with its filesystems (or filesystem, everything was in one) on the tape (read only, naturally). It wasn't fast, but it worked...It worked even better once the filesystem was copied to a disc! (The issue was that, at the time, there was no available unix code that allowed booting from a disc, or not one of the kind that we had - whereas the hardware made booting from a tape absolutely trivial. All in the days before any kind of intelligent firmware console was available - booting required entering the boot program to ram via a front panel - "put X in location Y" ...)
Re: Issues with lseek(2) on a block device
On Sat, Feb 24, 2024 at 04:28:28PM -0500, Thor Lancelot Simon wrote: > > Hello, thanks to everyone who responded with their suggestions. Using > > various non-portable ioctls I can device size on most platforms, for > > both block and raw devices. > > > > This is more convoluted than a single lseek() call, but it is what it > > is. If anyone wants to do something similar, then the following example > > can give you a good start. > > Probably not a good idea to start with lseek() because if you _do_ > encounter a tape device, seeking to SEEK_END could take you an extremely > long time. Also there at least used to be tapes where if you spun them all the way to the end you'd unthread the tape onto the take-up spindle and have to take it out and fix it. -- David A. Holland dholl...@netbsd.org
Re: Issues with lseek(2) on a block device
t...@panix.com (Thor Lancelot Simon) writes: >Probably not a good idea to start with lseek() because if you _do_ >encounter a tape device, seeking to SEEK_END could take you an extremely >long time. lseek() doesn't move the tape. But the open() or close() may trigger tape operations like loading or winding. Special files are special.
Re: Issues with lseek(2) on a block device
> Date: Sat, 24 Feb 2024 16:21:42 -0500 > From: Thor Lancelot Simon > > On Wed, Feb 21, 2024 at 09:20:55PM +, Taylor R Campbell wrote: > > I think this is a bug, and it would be great if stat(2) just returned > > the physical medium's size in st_size -- currently doing this reliably > > takes at least three different ioctls to handle all storage media, if > > I counted correctly in sbin/fsck/partutil.c's getdiskinfo. > > I am not sure this can be done for all block devices. Tapes have block > devices, and open-reel tape drives do not really know the length of the > loaded media, while on any other tape drive supporting compression, there > may really be no such size. Sure, it's fine if it doesn't give an answer (or, returns st_size=0) for devices where there's no reasonable answer. But for a medium which does have a definite size that is known up front, it should just be returned by stat(2) in st_size instead of requiring a dance of multiple different NetBSD-specific ioctls to guess at which one will work.
Re: Issues with lseek(2) on a block device
On Thu, Feb 22, 2024 at 12:08:14PM +, Sad Clouds wrote: > Hello, thanks to everyone who responded with their suggestions. Using > various non-portable ioctls I can device size on most platforms, for > both block and raw devices. > > This is more convoluted than a single lseek() call, but it is what it > is. If anyone wants to do something similar, then the following example > can give you a good start. Probably not a good idea to start with lseek() because if you _do_ encounter a tape device, seeking to SEEK_END could take you an extremely long time. Thor
Re: Issues with lseek(2) on a block device
On Wed, Feb 21, 2024 at 09:20:55PM +, Taylor R Campbell wrote: > > Date: Wed, 21 Feb 2024 10:52:55 + > > From: Sad Clouds > > > > Hello, for most operating systems determining the size of a block > > device can be done with: > > > > lseek(fd, 0, SEEK_END); > > > > However, on NetBSD this does not seem to work. > > Internally, this is happens for more or less the same reason that > stat(2) doesn't return the size of a block device in st_size. > > Each file system on which device nodes can reside implements its own > VOP_GETATTR (used by both lseek(2) and stat(2)), and most of them use > the _inode_ size (more or less) rather than querying the block device > that the device node represents for its physical size. > > I think this is a bug, and it would be great if stat(2) just returned > the physical medium's size in st_size -- currently doing this reliably > takes at least three different ioctls to handle all storage media, if > I counted correctly in sbin/fsck/partutil.c's getdiskinfo. I am not sure this can be done for all block devices. Tapes have block devices, and open-reel tape drives do not really know the length of the loaded media, while on any other tape drive supporting compression, there may really be no such size.
Re: Issues with lseek(2) on a block device
On Thu, Feb 22, 2024 at 05:47:55AM +, Sad Clouds wrote: > On Wed, 21 Feb 2024 12:48:34 -0800 > Jason Thorpe wrote: > > > > > > On Feb 21, 2024, at 2:52???AM, Sad Clouds > > > wrote: > > > > > > Hello, for most operating systems determining the size of a block > > > device can be done with: > > > > > > lseek(fd, 0, SEEK_END); > > > > On what operating systems does this do what you claim? > > > > > However, on NetBSD this does not seem to work. > > > > It doesn???t work on macOS, either: > > > > thorpej-mbp:thorpej 541$ sudo ./lseek /dev/disk4 > > Password: > > lseek(fd, 4096, SEEK_SET) = 4096 bytes > > lseek(fd, 0, SEEK_END)= 0 bytes > > thorpej-mbp:thorpej 542$ > > > > -- thorpej > > > > Hello, it works on Linux, FreeBSD, Solaris and it also works on NetBSD I don't think that's correct for all devices on Linux - generally, on Linux systems I have to use the BLKGETSIZE64 ioctl to reliable determine the size of a block device.
Re: Issues with lseek(2) on a block device
>> It looks to me like "we didn't bother making it do anything in >> particular, so you get whatever it happens to give you". > "bug" ultimately means "failure to conform to expectations". Well...maybe. Depends on whose expectations. If I expect, say, that typing a tab on a command line puts a tab into the command, and someone else expects it to do filename completion, which one is the bug? Usually, I use "bug" to mean something close to what you said, where the answer to "whose" is "the author's". > We could debate what expectations might be for stat and block device > sizes, but it is definitely against expectations that something so > simple as retrieving the size of a storage device has such a messy > interface. Again, whose expectations? My expectations, formed from decades of experience, are that it _is_ a messy thing. At least one person clearly expected that lseek would do the trick. (Interestingly, I would not have even considered lseek; if I'd had to pick a stock call that I would expect to return the size of a disk, it would be stat() or one of its variants - lstat, fstat, etc.) >> My own method of finding disk device size is [...]. I'd expect it >> to be highly portable. The only cases I'd expect it to fail in are >> disks over 4G (or perhaps 2G) on systems with only 32 bits for >> off_t. > ...or tapes. Or ttys. Tapes don't really have a size in that sense to obtain. (Most tapes. Some DEC tapes do look a lot disks with sec/trk and trk/cyl both 1 and what for disks would be _extremely_ long seek times.) /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Issues with lseek(2) on a block device
On Thu, Feb 22, 2024 at 08:13:28AM -0500, Mouse wrote: > I wouldn't call it buggy, not unless there is a spec that it's supposed > to conform to that says otherwise (even if the "spec" is just an > author's description of intent), which is something I so far haven't > seen reason to think exists. It looks to me like "we didn't bother > making it do anything in particular, so you get whatever it happens to > give you". "bug" ultimately means "failure to conform to expectations". We could debate what expectations might be for stat and block device sizes, but it is definitely against expectations that something so simple as retrieving the size of a storage device has such a messy interface. > My own method of finding disk device size is to lseek to offset N and > try to read one sector, for various N: initially N = sector size, then > double N until I get EOF-or-error, then do binary search between the > last working value and the first failing value. I think that's worked > on everything I've tried it on; admittedly, that's only a few OSes, but > I'd expect it to be highly portable. The only cases I'd expect it to > fail in are disks over 4G (or perhaps 2G) on systems with only 32 bits > for off_t. ...or tapes. -- David A. Holland dholl...@netbsd.org
Re: Issues with lseek(2) on a block device
On Thu, Feb 22, 2024 at 08:13:28AM -0500, Mouse wrote: > >>> lseek(fd, 0, SEEK_END); > [on a disk device] > > >> [...] > > [...] > > This is such a buggy behaviour that [...] > > I wouldn't call it buggy, not unless there is a spec that it's supposed > to conform to that says otherwise (even if the "spec" is just an > author's description of intent), which is something I so far haven't > seen reason to think exists. It looks to me like "we didn't bother > making it do anything in particular, so you get whatever it happens to > give you". In general: If a file descriptor references anything other than a regular file, then the assumptions that portable code can make about it are constrained.
Re: Issues with lseek(2) on a block device
>>> lseek(fd, 0, SEEK_END); [on a disk device] >> [...] > [...] > This is such a buggy behaviour that [...] I wouldn't call it buggy, not unless there is a spec that it's supposed to conform to that says otherwise (even if the "spec" is just an author's description of intent), which is something I so far haven't seen reason to think exists. It looks to me like "we didn't bother making it do anything in particular, so you get whatever it happens to give you". > I stopped using stat for finding out block device size a long time > ago, and not just on NetBSD. That sounds sensible to me. I think your first mail on the subject was the first time I'd even _considered_ that stat() applied to a disk device might return drive/partition size. My own method of finding disk device size is to lseek to offset N and try to read one sector, for various N: initially N = sector size, then double N until I get EOF-or-error, then do binary search between the last working value and the first failing value. I think that's worked on everything I've tried it on; admittedly, that's only a few OSes, but I'd expect it to be highly portable. The only cases I'd expect it to fail in are disks over 4G (or perhaps 2G) on systems with only 32 bits for off_t. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Issues with lseek(2) on a block device
Hello, thanks to everyone who responded with their suggestions. Using
various non-portable ioctls I can device size on most platforms, for
both block and raw devices.
This is more convoluted than a single lseek() call, but it is what it
is. If anyone wants to do something similar, then the following example
can give you a good start.
/*
* devsize.c
* gcc -DNETBSD -Wall -Wextra -pedantic -o devsize devsize.c
*/
#include
#include
#include
#include
#include
#include
#include
#if defined(NETBSD)
#include
#endif
#if defined(FREEBSD)
#include
#endif
#if defined(MACOS)
#include
#endif
#if defined(LINUX)
#include
#endif
/*
* Use lseek() to get device size in bytes.
* This should be portable, but on some platforms returns 0 bytes,
* in which case nonportable ioctl should be used.
*/
static void print_devsize_lseek(int fd)
{
off_t size;
size = lseek(fd, 0, SEEK_END);
assert(size >= 0);
printf("lseek SEEK_END = %ju bytes\n", (uintmax_t)size);
}
/* NetBSD, FreeBSD */
#if defined(DIOCGMEDIASIZE)
static void print_devsize_DIOCGMEDIASIZE(int fd)
{
int int_val;
off_t size;
/* This sets size to total number of bytes */
int_val = ioctl(fd, DIOCGMEDIASIZE, );
assert(int_val >= 0);
printf("ioctl DIOCGMEDIASIZE = %ju bytes\n", (uintmax_t)size);
}
#endif
/* MacOS */
#if defined(DKIOCGETBLOCKCOUNT) && defined(DKIOCGETBLOCKSIZE)
static void print_devsize_DKIOCGETBLOCKCOUNT(int fd)
{
int int_val
uint64_t block_count;
uint32_t block_size;
/* Total block count */
int_val = ioctl(fd, DKIOCGETBLOCKCOUNT, _count);
assert(int_val >= 0);
/* Block size */
int_val = ioctl(fd, DKIOCGETBLOCKSIZE, _size);
assert(int_val >= 0);
printf("ioctl DKIOCGETBLOCKCOUNT = %ju bytes\n",
(uintmax_t)block_count * (uintmax_t)block_size);
}
#endif
/* Linux */
#if defined(BLKGETSIZE)
static void print_devsize_BLKGETSIZE(int fd)
{
int int_val;
unsigned long size;
/* This sets size to number of 512-byte blocks */
int_val = ioctl(fd, BLKGETSIZE, );
assert(int_val >= 0);
printf("ioctl BLKGETSIZE = %ju bytes\n", (uintmax_t)size * 512);
}
#endif
/* Linux */
#if defined(BLKGETSIZE64)
static void print_devsize_BLKGETSIZE64(int fd)
{
int int_val;
uint64_t size;
/* This sets size to total number of bytes */
int_val = ioctl(fd, BLKGETSIZE64, );
assert(int_val >= 0);
printf("ioctl BLKGETSIZE = %ju bytes\n", (uintmax_t)size);
}
#endif
int main(int argc, char *argv[])
{
int fd;
if (argc != 2)
{
fprintf(stderr, "Usage: devsize \n");
exit(EXIT_FAILURE);
}
/* Open device */
fd = open(argv[1], O_RDONLY);
assert(fd >= 0);
print_devsize_lseek(fd);
#if defined(DIOCGMEDIASIZE)
print_devsize_DIOCGMEDIASIZE(fd);
#endif
#if defined(DKIOCGETBLOCKCOUNT) && defined(DKIOCGETBLOCKSIZE)
print_devsize_DKIOCGETBLOCKCOUNT(fd);
#endif
#if defined(BLKGETSIZE)
print_devsize_BLKGETSIZE(fd);
#endif
#if defined(BLKGETSIZE64)
print_devsize_BLKGETSIZE64(fd);
#endif
}
Re: Issues with lseek(2) on a block device
mlel...@serpens.de (Michael van Elst) writes:
>But it also does not work for wedges or device mapper volumes
>(zfs vol probably fail too) as these don't implement the
>used internal ioctl for disk devices. At least that part
>would be easy to fix, but of questionable value.
Like this:
Index: sys/miscfs/specfs/spec_vnops.c
===
RCS file: /cvsroot/src/sys/miscfs/specfs/spec_vnops.c,v
retrieving revision 1.218
diff -p -u -r1.218 spec_vnops.c
--- sys/miscfs/specfs/spec_vnops.c 22 Apr 2023 15:32:49 - 1.218
+++ sys/miscfs/specfs/spec_vnops.c 22 Feb 2024 08:54:16 -
@@ -727,11 +727,11 @@ spec_open(void *v)
enum kauth_device_req req;
specnode_t *sn, *sn1;
specdev_t *sd;
+ int dtype;
spec_ioctl_t ioctl;
u_int gen = 0;
const char *name = NULL;
bool needclose = false;
- struct partinfo pi;
KASSERT(VOP_ISLOCKED(vp) == LK_EXCLUSIVE);
KASSERTMSG(vp->v_type == VBLK || vp->v_type == VCHR, "type=%d",
@@ -1038,11 +1038,23 @@ spec_open(void *v)
* forbidden to devsw_detach until closed). So it is safe to
* query cdev_type unconditionally here.
*/
- if (cdev_type(dev) == D_DISK) {
- ioctl = vp->v_type == VCHR ? cdev_ioctl : bdev_ioctl;
- if ((*ioctl)(dev, DIOCGPARTINFO, , FREAD, curlwp) == 0)
- uvm_vnp_setsize(vp,
- (voff_t)pi.pi_secsize * pi.pi_size);
+ switch (vp->v_type) {
+ case VCHR:
+ ioctl = bdev_ioctl;
+ dtype = cdev_type(dev);
+ break;
+ default:
+ ioctl = bdev_ioctl;
+ dtype = bdev_type(dev);
+ break;
+ }
+ if (dtype == D_DISK) {
+ off_t count;
+ u_int sz;
+
+ if ((*ioctl)(dev, DIOCGMEDIASIZE, , FREAD, curlwp) == 0
+ && (*ioctl)(dev, DIOCGSECTORSIZE, , FREAD, curlwp) == 0)
+ uvm_vnp_setsize(vp, (voff_t)count * sz);
}
/* Success! */
Re: Issues with lseek(2) on a block device
cryintotheblue...@gmail.com (Sad Clouds) writes:
>Hello, for most operating systems determining the size of a block
>device can be done with:
>lseek(fd, 0, SEEK_END);
>However, on NetBSD this does not seem to work.
The disk size is only retrieved at open time and stored in the
cached vnode. stat() therefore only sees the information as
long as the vnode is in cache. Before open, the vnode stores
the bits from e.g. the UFS inode, for UFS that would be zero.
It also does not work for block devices as these aren't really
opened when you call open(). Character devices ("raw disk")
are better, the lseek() or fstat() methods do work for raw
disk devices.
But it also does not work for wedges or device mapper volumes
(zfs vol probably fail too) as these don't implement the
used internal ioctl for disk devices. At least that part
would be easy to fix, but of questionable value.
Currently the only reliable way is to use ioctls. You can use
DIOCGDISKINFO (with proplib) or you can use the two FreeBSD ioctls
DIOCGMEDIASIZE and DIOCGSECTORSIZE to retrieve block count and
size individually as numbers.
For compatibility with netbsd < 8 you may need to fall back to
various older ioctls. That's what is still being implemented
for tools like fsck.
Re: Issues with lseek(2) on a block device
On Wed, 21 Feb 2024 12:48:34 -0800 Jason Thorpe wrote: > > > On Feb 21, 2024, at 2:52 AM, Sad Clouds wrote: > > > > Hello, for most operating systems determining the size of a block > > device can be done with: > > > > lseek(fd, 0, SEEK_END); > > On what operating systems does this do what you claim? > > > However, on NetBSD this does not seem to work. > > It doesn’t work on macOS, either: > > thorpej-mbp:thorpej 541$ sudo ./lseek /dev/disk4 > Password: > lseek(fd, 4096, SEEK_SET) = 4096 bytes > lseek(fd, 0, SEEK_END)= 0 bytes > thorpej-mbp:thorpej 542$ > > -- thorpej > Hello, it works on Linux, FreeBSD, Solaris and it also works on NetBSD but only with ld(4) driver, with SCSI or SATA drives, lseek returns bogus size of 0 bytes, even though the kernel knows the size of block device, otherwise partition formatting would be impossible. Specifically for NetBSD: # uname -mps NetBSD evbarm earmv7hf # dmesg | grep ld0 [ 1.768438] ld0 at sdmmc0: <0x03:0x5344:SE16G:0x80:0x08834ec0:0x0d8> [ 1.768438] ld0: 15193 MB, 7717 cyl, 64 head, 63 sec, 512 bytes/sect x 31116288 sectors [ 1.798199] ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz [ 2.338199] boot device: ld0 [ 2.338199] root on ld0a dumps on ld0b Initially stat on /dev/ld0 reports size as 0 bytes: # stat -x /dev/ld0 File: "/dev/ld0" Size: 0Blocks: 0IO Block: 2048 Block Device Device: 92,0 Inode: 1573Links: 1 Mode: (0640/brw-r-) Uid: (0/root) Gid: (5/operator) Access: 2024-02-21 13:30:44.19198 + Modify: 2021-10-17 17:58:03.0 +0100 Change: 2021-10-17 17:58:03.0 +0100 Birth: 1970-01-01 01:00:00.0 +0100 So stat is no good here, let's try lseek: # gcc -Wall -Wpedantic lseek_test.c && ./a.out lseek(fd, 4096, SEEK_SET) = 4096 bytes lseek(fd, 0, SEEK_END)= 15931539456 bytes But now, after lseek, stat reports the correct size: # stat -x /dev/ld0 File: "/dev/ld0" Size: 15931539456 Blocks: 0IO Block: 2048 Block Device Device: 92,0 Inode: 1573Links: 1 Mode: (0640/brw-r-) Uid: (0/root) Gid: (5/operator) Access: 2024-02-21 13:30:44.19198 + Modify: 2021-10-17 17:58:03.0 +0100 Change: 2021-10-17 17:58:03.0 +0100 Birth: 1970-01-01 01:00:00.0 +0100 This is such a buggy behaviour that I stopped using stat for finding out block device size a long time ago, and not just on NetBSD.
Re: Issues with lseek(2) on a block device
On Wed, Feb 21, 2024 at 09:20:55PM +, Taylor R Campbell wrote: > But it's a little annoying to make that happen because it requires > going through all the file systems that have device nodes and > convincing their VOP_GETATTR implementations to do something different > for block devices (and ideally also character devices, if they > represent fixed-size media too). If I ever find time to follow through on my threat to reorganize the way device vnodes work, that will get rid of this problem :-) (and provide everyone a pony, too) (if anyone's curious, not sure I've posted about it before, the idea is to flip things so that the device logic sits on top of the fs-level special file vnode; currently it's the other way around, so the fs gets involved with all kinds of things it doesn't actually need to care about) -- David A. Holland dholl...@netbsd.org
Re: Issues with lseek(2) on a block device
(This should really be on tech-kern…) > On Feb 21, 2024, at 1:20 PM, Taylor R Campbell > wrote: > >> Date: Wed, 21 Feb 2024 10:52:55 + >> From: Sad Clouds >> >> Hello, for most operating systems determining the size of a block >> device can be done with: >> >> lseek(fd, 0, SEEK_END); >> >> However, on NetBSD this does not seem to work. > > Internally, this is happens for more or less the same reason that > stat(2) doesn't return the size of a block device in st_size. > > Each file system on which device nodes can reside implements its own > VOP_GETATTR (used by both lseek(2) and stat(2)), and most of them use > the _inode_ size (more or less) rather than querying the block device > that the device node represents for its physical size. Add a d_stat devsw entry point and implement spec_getattr()? Except you don’t want ALL of the results from spec_getattr(), just some of them - rather, specfs can’t possibly provide all of the info, just some of it. So the file system that owns the vnode (UFS, in this case) would need to call spec_getattr() to get the stuff that specfs can provide, then do its own stuff to overlay the usual UFS stuff over it. -- thorpej
Re: Issues with lseek(2) on a block device
> Date: Wed, 21 Feb 2024 10:52:55 + > From: Sad Clouds > > Hello, for most operating systems determining the size of a block > device can be done with: > > lseek(fd, 0, SEEK_END); > > However, on NetBSD this does not seem to work. Internally, this is happens for more or less the same reason that stat(2) doesn't return the size of a block device in st_size. Each file system on which device nodes can reside implements its own VOP_GETATTR (used by both lseek(2) and stat(2)), and most of them use the _inode_ size (more or less) rather than querying the block device that the device node represents for its physical size. I think this is a bug, and it would be great if stat(2) just returned the physical medium's size in st_size -- currently doing this reliably takes at least three different ioctls to handle all storage media, if I counted correctly in sbin/fsck/partutil.c's getdiskinfo. But it's a little annoying to make that happen because it requires going through all the file systems that have device nodes and convincing their VOP_GETATTR implementations to do something different for block devices (and ideally also character devices, if they represent fixed-size media too).
Re: Issues with lseek(2) on a block device
> On Feb 21, 2024, at 2:52 AM, Sad Clouds wrote: > > Hello, for most operating systems determining the size of a block > device can be done with: > > lseek(fd, 0, SEEK_END); On what operating systems does this do what you claim? > However, on NetBSD this does not seem to work. It doesn’t work on macOS, either: thorpej-mbp:thorpej 541$ sudo ./lseek /dev/disk4 Password: lseek(fd, 4096, SEEK_SET) = 4096 bytes lseek(fd, 0, SEEK_END)= 0 bytes thorpej-mbp:thorpej 542$ -- thorpej
