Re: Large filesystem woes
The resolution to this problem turned out to be enabling the 3Ware 9500S auto carving option. This splits all partitions into 2TB chunks, which are then presented to the OS as separate LUNs. FreeBSD 5.4-RELEASE's driver for the 3Ware card does not support multiple LUNs, but the new driver (Common Layer) in 5.4-STABLE does. So if you're running in to this same problem, here's a quick run down of what you need to do: 1) Enable auto-carving in either the 3ware BIOS or in the 3dm web control panel. 2) Delete and then re-create any RAID that is larger than 2TB, that you want accessable from the OS. If you have sufficient drives available, you can migrate the data to a new RAID, instead of deleting. 3) Install 5.4-RELEASE. sysinstall will show you a single 2TB device. Feel free to use that device however you wish -- you can fill it to its boundaries. 4) cvsup to 5.4-STABLE, make buildworld buildkernel installkernel, reboot. 5) You'll now see da1 and perhaps da2 and beyond, in dmesg or camcontrol devlist. If you should ever have to reboot into the 5.4-RELEASE GENERIC kernel, the da1..n partitions will be unavailable. da0, however, will operate normally. As such it is safe to use as a boot device. (For sanity's sake it is probably best to create a much smaller slice or partition for system files, but that's your decision). ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On Thu, 21 Jul 2005, Giorgos Keramidas wrote: Are you trying to start from scratch, or just making changes to the existing table? If it's the second, then try the -u option: # fdisk -u /dev/da0 This should give you a chance to interactively change the partition table of the disk. If this fails too, please show us the exact command line you used and the exact error messages. - Giorgos Unfortunately I can't run fdisk -u, if that command will wipe out the partition table, since / is also on the array, just in a smaller partition. I found this page: http://www.freebsd.org/projects/bigdisk/ It was last updated in January. Has there been more progress on this front? I wonder if this is all a known problem, and I just overlooked it because of my understanding that UFS2 is supposed to handle large partitions. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On 2005-07-28 15:52, dpk [EMAIL PROTECTED] wrote: On Thu, 21 Jul 2005, Giorgos Keramidas wrote: Are you trying to start from scratch, or just making changes to the existing table? If it's the second, then try the -u option: # fdisk -u /dev/da0 This should give you a chance to interactively change the partition table of the disk. If this fails too, please show us the exact command line you used and the exact error messages. Unfortunately I can't run fdisk -u, if that command will wipe out the partition table, since / is also on the array, just in a smaller partition. It won't wipe away the table. It will just let you edit the existing table interactively, through a series of questions like: - Do you want to edit partition 1? - Do you want to edit partition 2? - Do you want to edit partition 3? - Do you want to edit partition 4? - Do you want to change the active partition? - Do you want to save your changes to the disk? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On Fri, 29 Jul 2005, Giorgos Keramidas wrote: It won't wipe away the table. It will just let you edit the existing table interactively, through a series of questions like: - Do you want to edit partition 1? - Do you want to edit partition 2? - Do you want to edit partition 3? - Do you want to edit partition 4? - Do you want to change the active partition? - Do you want to save your changes to the disk? # df -k Filesystem 1K-blocksUsedAvail Capacity Mounted on /dev/da0s1a 35082074 1147642 31127868 4%/ devfs 1 10 100%/dev procfs 4 40 100%/proc # fdisk -u fdisk: cannot open disk /dev/da0: No such file or directory # ls -ald /dev/da0* crw-r- 1 root operator4, 12 Jul 28 15:56 /dev/da0 crw-r- 1 root operator4, 13 Jul 28 15:56 /dev/da0s1 crw-r- 1 root operator4, 18 Jul 28 08:56 /dev/da0s1a crw-r- 1 root operator4, 19 Jul 28 15:56 /dev/da0s1b crw-r- 1 root operator4, 20 Jul 28 15:56 /dev/da0s1c truss indicates that fdisk may be getting the error from somewhere else: stat(/dev/da0,0xbfbfeb30) = 0 (0x0) open(/dev/da0,0x2,00) ERR#1 'Operation not permitted' open(/dev/da0,0x0,027757765630)= 6 (0x6) open(/dev/da0s1,0x2,01001210100) ERR#1 'Operation not permitted' open(/dev/da0s2,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s3,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s4,0x2,01001210100) ERR#2 'No such file or directory' Because it is using devfs, I'm not able to create these missing slices in /dev. Most unfortunately, it appears it uses devfs in single user mode as well, so I can't test the theory. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On 2005-07-28 16:04, dpk [EMAIL PROTECTED] wrote: # df -k Filesystem 1K-blocksUsedAvail Capacity Mounted on /dev/da0s1a 35082074 1147642 31127868 4%/ devfs 1 10 100%/dev procfs 4 40 100%/proc # fdisk -u fdisk: cannot open disk /dev/da0: No such file or directory # ls -ald /dev/da0* crw-r- 1 root operator4, 12 Jul 28 15:56 /dev/da0 crw-r- 1 root operator4, 13 Jul 28 15:56 /dev/da0s1 crw-r- 1 root operator4, 18 Jul 28 08:56 /dev/da0s1a crw-r- 1 root operator4, 19 Jul 28 15:56 /dev/da0s1b crw-r- 1 root operator4, 20 Jul 28 15:56 /dev/da0s1c truss indicates that fdisk may be getting the error from somewhere else: stat(/dev/da0,0xbfbfeb30) = 0 (0x0) open(/dev/da0,0x2,00) ERR#1 'Operation not permitted' open(/dev/da0,0x0,027757765630)= 6 (0x6) open(/dev/da0s1,0x2,01001210100) ERR#1 'Operation not permitted' open(/dev/da0s2,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s3,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s4,0x2,01001210100) ERR#2 'No such file or directory' Because it is using devfs, I'm not able to create these missing slices in /dev. Most unfortunately, it appears it uses devfs in single user mode as well, so I can't test the theory. Hmmm, in multiuser mode, your root filesystem is mounted as read-write and it resides in da0, so GEOM will forbid opening the disk device in read-write mode for editing the partition table. In single user mode, devfs is still used, but your root filesystem should be mounted read-only (unless you manually mount it as read-write), so fdisk -u should work. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On Thu, 28 Jul 2005, dpk wrote: truss indicates that fdisk may be getting the error from somewhere else: stat(/dev/da0,0xbfbfeb30) = 0 (0x0) open(/dev/da0,0x2,00) ERR#1 'Operation not permitted' open(/dev/da0,0x0,027757765630)= 6 (0x6) open(/dev/da0s1,0x2,01001210100) ERR#1 'Operation not permitted' open(/dev/da0s2,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s3,0x2,01001210100) ERR#2 'No such file or directory' open(/dev/da0s4,0x2,01001210100) ERR#2 'No such file or directory' Because it is using devfs, I'm not able to create these missing slices in /dev. Most unfortunately, it appears it uses devfs in single user mode as well, so I can't test the theory. Under a chrooted shell, I created the s1-4 /dev entries. fdisk -u now reports 'Device not configured', which supports the theory that fdisk is just using whatever error it last sees and gives up. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On Fri, 29 Jul 2005, Giorgos Keramidas wrote:
Hmmm, in multiuser mode, your root filesystem is mounted as read-write
and it resides in da0, so GEOM will forbid opening the disk device in
read-write mode for editing the partition table.
In single user mode, devfs is still used, but your root filesystem
should be mounted read-only (unless you manually mount it as
read-write), so fdisk -u should work.
I've remounted the disk readonly, and was still seeing the same errors.
I've looked at the fdisk source and compared it to the truss output.
... /* rwmode is O_RDWR due to -u */
fd = open(disk, rwmode);
... /* errno is EPERM here, from truss */
if (fd == -1 errno == ENXIO)
return -2;
if (fd == -1 errno == EPERM rwmode == O_RDWR) {
... /* this is successful: */
fd = open(disk, O_RDONLY);
... /* the following opens get device not configured, or no such file or
directory under normal operation, from truss */
for (p = 1; p 5; p++) {
asprintf(s, %ss%d, disk, p);
fdw = open(s, O_RDONLY);
free(s);
if (fdw == -1)
continue;
break;
}
... /* ah ha! open_disk is returning -4 because the last slice had some
error */
if (fdw == -1)
return -4;
This change was introduced with version 1.67 of the fdisk.c file.
Commenting out if (fdw == -1) return -4; allows fdisk -u to function. Here
are the results, not changing anything:
# ./fdisk -u
*** Working on device /dev/da0 ***
parameters extracted from in-core disklabel are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Do you want to change our idea of what BIOS thinks ? [n]
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 75489372 (36860 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
Do you want to change it? [n]
The data for partition 2 is:
UNUSED
Do you want to change it? [n]
The data for partition 3 is:
UNUSED
Do you want to change it? [n]
The data for partition 4 is:
UNUSED
Do you want to change it? [n]
Partition 1 is marked active
Do you want to change the active partition? [n]
Here are the results of ./fdisk -u when trying to add the second
partition:
# ./fdisk -u
*** Working on device /dev/da0 ***
parameters extracted from in-core disklabel are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 75489372 (36860 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
Do you want to change it? [n] n
The data for partition 2 is:
UNUSED
Do you want to change it? [n] y
Supply a decimal value for sysid (165=FreeBSD) [0] 165
Supply a decimal value for start [0]
Supply a decimal value for size [0]
fdisk: ERROR: size of partition is zero
fdisk: ERROR: failed to adjust; setting sysid to 0
Explicitly specify beg/end address ? [n] y
Supply a decimal value for beginning cylinder [0] 1024
Supply a decimal value for beginning head [0]
Supply a decimal value for beginning sector [0]
Supply a decimal value for ending cylinder [0] 534921
Supply a decimal value for ending head [0] 254
Supply a decimal value for ending sector [0] 63
sysid 0 (),(unused)
start 0, size 0 (0 Meg), flag 0
beg: cyl 0/ head 0/ sector 0;
end: cyl 393/ head 254/ sector 63
Writing it fails, but I didn't really expect it to succeed with the above
values. There's some overflowing going on.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
Nice catch! I think you should report this to phk@ or ru@ who have been
the most active in the fdisk area :-)
- Giorgos
On 2005-07-28 16:40, dpk [EMAIL PROTECTED] wrote:
I've remounted the disk readonly, and was still seeing the same errors.
I've looked at the fdisk source and compared it to the truss output.
... /* rwmode is O_RDWR due to -u */
fd = open(disk, rwmode);
... /* errno is EPERM here, from truss */
if (fd == -1 errno == ENXIO)
return -2;
if (fd == -1 errno == EPERM rwmode == O_RDWR) {
... /* this is successful: */
fd = open(disk, O_RDONLY);
... /* the following opens get device not configured, or no such file or
directory under normal operation, from truss */
for (p = 1; p 5; p++) {
asprintf(s, %ss%d, disk, p);
fdw = open(s, O_RDONLY);
free(s);
if (fdw == -1)
continue;
break;
}
... /* ah ha! open_disk is returning -4 because the last slice had some
error */
if (fdw == -1)
return -4;
This change was introduced with version 1.67 of the fdisk.c file.
Commenting out if (fdw == -1) return -4; allows fdisk -u to function. Here
are the results, not changing anything:
# ./fdisk -u
*** Working on device /dev/da0 ***
parameters extracted from in-core disklabel are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Do you want to change our idea of what BIOS thinks ? [n]
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 75489372 (36860 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
Do you want to change it? [n]
The data for partition 2 is:
UNUSED
Do you want to change it? [n]
The data for partition 3 is:
UNUSED
Do you want to change it? [n]
The data for partition 4 is:
UNUSED
Do you want to change it? [n]
Partition 1 is marked active
Do you want to change the active partition? [n]
Here are the results of ./fdisk -u when trying to add the second
partition:
# ./fdisk -u
*** Working on device /dev/da0 ***
parameters extracted from in-core disklabel are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=534921 heads=255 sectors/track=63 (16065 blks/cyl)
Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 75489372 (36860 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
Do you want to change it? [n] n
The data for partition 2 is:
UNUSED
Do you want to change it? [n] y
Supply a decimal value for sysid (165=FreeBSD) [0] 165
Supply a decimal value for start [0]
Supply a decimal value for size [0]
fdisk: ERROR: size of partition is zero
fdisk: ERROR: failed to adjust; setting sysid to 0
Explicitly specify beg/end address ? [n] y
Supply a decimal value for beginning cylinder [0] 1024
Supply a decimal value for beginning head [0]
Supply a decimal value for beginning sector [0]
Supply a decimal value for ending cylinder [0] 534921
Supply a decimal value for ending head [0] 254
Supply a decimal value for ending sector [0] 63
sysid 0 (),(unused)
start 0, size 0 (0 Meg), flag 0
beg: cyl 0/ head 0/ sector 0;
end: cyl 393/ head 254/ sector 63
Writing it fails, but I didn't really expect it to succeed with the above
values. There's some overflowing going on.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On 2005-07-20 06:47, dpk [EMAIL PROTECTED] wrote: On Wed, 20 Jul 2005, Lowell Gilbert wrote: dpk [EMAIL PROTECTED] writes: Has anyone here had any luck whatsoever getting FreeBSD 5.4-RELEASE to use 4TB of space from a 4TB RAID array? It is apparent that there is a 1TB/slice limit, and sysinstall doesn't seem to be able to handle creating more than 2 slices. Since you're using slices, I would try working with fdisk directly... fdisk gives a No such file or directory error when you run fdisk -i /dev/da0, but fdisk /dev/da0 shows the partition table. Are you trying to start from scratch, or just making changes to the existing table? If it's the second, then try the -u option: # fdisk -u /dev/da0 This should give you a chance to interactively change the partition table of the disk. If this fails too, please show us the exact command line you used and the exact error messages. - Giorgos ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
dpk [EMAIL PROTECTED] writes: Has anyone here had any luck whatsoever getting FreeBSD 5.4-RELEASE to use 4TB of space from a 4TB RAID array? It is apparent that there is a 1TB/slice limit, and sysinstall doesn't seem to be able to handle creating more than 2 slices. Since you're using slices, I would try working with fdisk directly... -- Lowell Gilbert, embedded/networking software engineer, Boston area http://be-well.ilk.org/~lowell/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Large filesystem woes
On Wed, 20 Jul 2005, Lowell Gilbert wrote: dpk [EMAIL PROTECTED] writes: Has anyone here had any luck whatsoever getting FreeBSD 5.4-RELEASE to use 4TB of space from a 4TB RAID array? It is apparent that there is a 1TB/slice limit, and sysinstall doesn't seem to be able to handle creating more than 2 slices. Since you're using slices, I would try working with fdisk directly... fdisk gives a No such file or directory error when you run fdisk -i /dev/da0, but fdisk /dev/da0 shows the partition table. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Large filesystem woes
Has anyone here had any luck whatsoever getting FreeBSD 5.4-RELEASE to use 4TB of space from a 4TB RAID array? It is apparent that there is a 1TB/slice limit, and sysinstall doesn't seem to be able to handle creating more than 2 slices. I read somewhere that people recommend using gpt to manage large filesystems. Unfortunately, this tool is not to be found on the fixit floppy or the mfsroot during installation. If the server is up, gpt complains about Operation not permitted because the disk is in use. I managed to get gpt on a floppy -- along with a hand written cp/mkdir all-in-one binary because, when you use the Fixit option it does not chroot you to the fixit floppy, meaning you'll need /libexec/ld-elf.so.1 and /lib/libc.so.5, because for some reason critical system utilities are built dynamically now but I am straying from the topic -- and ran gpt migrate as recommended in its man page. Now gpt no longer recognizes the the disk at all. gpt show fails entirely. gpt -vvv show lists some entries, but the sizes are all very wrong, as if they were truncated. Anyone know the specific tricks involved in getting all 4TB used, even if it has to be in 4 different slices, or am I stuck with using just 2TB? We're gonna have to go with Linux on this particular server since we need it up today but we'll be getting another one soon and I can try things there. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
